A user account is required in order to edit this wiki, but we've had to disable public user registrations due to spam.

To request an account, ask an autoconfirmed user on IRC (such as one of these permanent autoconfirmed members).

Difference between revisions of "Storage"

From WHATWG Wiki
Jump to: navigation, search
(API proposal)
 
(14 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Existing APIs ==
+
{{obsolete|spec=[http://storage.spec.whatwg.org/ Storage Standard]}}
 
 
* localStorage
 
* sessionStorage
 
* Indexed DB
 
* Cookies
 
  
== Storage types ==
+
== Goals ==
  
* Best effort: default for non-temporary APIs, can be upgraded to persistent through the user.
+
* Provided a unified architecture for storage on the web.
* Persistent: resources that need to be available in order for the application to run offline. A distinction can be made between app layer (e.g. anything required to play music) and data layer (offline-available music).
+
* Enable persistent storage with the same "unlimited" quota as native.
* Temporary: resources that would be useful to cache to reduce latency/bandwidth but are not strictly required when offline. A priority system can be useful so the user agent can determine what to preserve from the temporary storage when running out of the global quota.
+
* Enable applications to more effectively manage their storage.
  
== Stories ==
+
== Use cases ==  
  
 
=== Music ===
 
=== Music ===
Line 30: Line 25:
 
A social network wants to temporarily cache the latest activities locally.
 
A social network wants to temporarily cache the latest activities locally.
  
== Requirements ==
+
=== Maps ===
 +
 
 +
Have the directions to your destination abroad locally available so you can avoid roaming.
 +
 
 +
=== News & books ===
 +
 
 +
Have news articles or books available offline to read on the London Tube.
 +
 
 +
== Infrastructure ==
 +
 
 +
A user agent has zero or more '''site storage units'''. A site storage unit's scope is eTLD+1. A site storage unit consists of a 'default box' and zero or more '''origin storage units'''.
 +
 
 +
An origin storage unit's cope is origin. An origin unit consists of a ''default box'' and zero or more ''boxes''.
 +
 
 +
HTTP cookies are stored in a site storage unit's ''default box''.
 +
 
 +
localStorage, sessionStorage, IDB, Service Workers, Cache API, History API, and Notifications API are stored in an origin storage unit's ''default box''.
 +
 
 +
=== Boxes ===
 +
 
 +
A '''box''' is where the bytes end up. A box has an '''id''' (default box's id is the empty string) and a '''mode''' ("atomic", "cache", or "persistent"). Storage APIs that support boxes can associate themselves with a box by passing it in.
 +
 
 +
Note: going forward we might add the ability to associate UI labels with boxes so end-users can make decisions on a per-box basis as well.
 +
 
 +
// If the persistent permission is granted, you can add boxes that are persistent
 +
navigator.storage.addBox("the-good-wife-s01", { mode: "persistent" }).then(
 +
  box => self.caches.open("the-good-wife-s01", { box: box }) // it might be worth changing .open() to be simpler
 +
)
 +
 
 +
// Not all APIs will support "cache" boxes, but it's a natural fit for the Cache API
 +
// even though by default it will be "atomic"
 +
navigator.storage.addBox("localcache", { mode: "cache" }).then(
 +
  box => self.caches.open("localcache", { box: box })
 +
)
 +
 
 +
// Games might want to store levels in distinct boxes so that when the user needs to
 +
// free up space the game engine remains intact while levels not played for a while
 +
// get cleared.
 +
navigator.storage.addBox("game-level-1", { mode: "atomic" }).then(
 +
  box => indexedDB.open("game-level-1", 1, { box: box }) // again, it might be worth changing .open() to be simpler
 +
)
 +
 
 +
==== Atomic boxes ====
 +
 
 +
An "atomic" box is cleared in its entirety. User agents are encouraged to reason globally about all "atomic" boxes in such a way that boxes the user least likely needs are cleared before others without paying much heed to site or origin storage unit boundaries.
  
* Persistent elastic storage (music, photos, video)
+
User agents could potentially reason globally about "atomic" boxes and individual items of "cache" boxes together.
* Priority-based temporary storage (gaming, social)
 
  
== Rough plan ==
+
==== Cache boxes ====
  
Create two modes to back the current set of storage APIs: best effort and persistent. By default the mode for a given eTLD+1 is best effort. After the user opts in it is persistent. Persistent means the user agent does not touch the storage without user consent. It also means that the application gets quite a lot of space. (Ideally most of the hard drive, if that turns into a problem the user will have to delete it...)
+
A "cache" box consists of a ''sequence of items''. What exactly constitutes an ''item'' is up to the API that supports a "cache" box. User agents are encouraged to reason globally about all ''items'' in all "cache" boxes in such a way that ''items'' the user least likely needs are cleared before others.
  
Also create a new storage API for explicit priority-based temporary storage. A key/value/priority store, effectively. Application is in charge of determining and changing priorities over time.
+
Note: A "cache" box is distinct from an "atomic" (and "persistent") box in that it provides no all-or-nothing guarantees. It's ideal for storing resources that help improve performance but are not essential.
  
== API proposal ==
+
==== Persistent boxes ====
  
  partial interface Navigator {
+
Through a user-mediated action an origin can acquire persistent storage. This ensures that origin storage unit's default box is persistently stored and will not be wiped without user consent. Persistence can be granted through navigator.storage.requestPersistent(), through bookmarking, or a combination of sorts. The expectation is that user agents will converge on a suitable pattern over time.
 +
 
 +
A site storage unit's default box cannot obtain persistence and will always be atomic.
 +
 
 +
=== Quota ===
 +
 
 +
The '''global quota''' is the available hard drive space minus any space the user agent wants to reserve. The ''non-persistent global quota'' is the limit of storage available within the global quota to "atomic" and "cache".
 +
 
 +
A '''site storage unit quota''' is a segment of the non-persistent global quota. User agents are free to let factors such as navigation frequency, bookmarking, and other indications of popularity impact the site storage unit quota.
 +
 
 +
An '''origin storage unit quota''' is that of its site storage unit, with the exception of its boxes that use "persistent" rather than other types of storage. "Persistent" boxes may use the global quota.
 +
 
 +
 
 +
== API ==
 +
 
 +
[NoInterfaceObject,
 +
  Exposed=(Window,Worker)]
 +
  interface NavigatorStorage {
 
   readonly attribute StorageManager storage;
 
   readonly attribute StorageManager storage;
 
  };
 
  };
   
+
  Navigator implements NavigatorStorage;
 +
WorkerNavigator implements NavigatorStorage;
 +
 
 +
This ensures the API is available everywhere under <code>navigator.storage</code>. (At some point we should probably clean up this pattern.)
 +
 
 
  [Exposed=(Window,Worker)]
 
  [Exposed=(Window,Worker)]
 
  interface StorageManager {
 
  interface StorageManager {
   readonly attribute StorageMode mode;
+
   [Exposed=Window] Promise<PersistentStoragePermission> requestPersistent();
   Promise<void> requestPersistent();
+
   Promise<PersistentStoragePermission> persistentPermission();
 
   
 
   
   Promise<[https://dvcs.w3.org/hg/quota/raw-file/tip/Overview.html#idl-def-StorageInfo StorageInfo]> getEstimate();
+
   Promise<StorageInfo]> getEstimate();
 +
  Promise<StorageInfo> getPersistentEstimate();
 +
 
 +
  Promise<StorageBox> addBox(DOMString id, StorageBoxInit init);
 +
  Promise<boolean> deleteBox(optional DOMString id = "");
 +
  Promise<sequence<StorageBox>> boxes();
 +
};
 +
 
 +
<code>StorageManager</code> provides access to getting permission for "persistent" boxes (and changing the origin storage unit's ''default box'' to use it).
 +
 
 +
<code>getEstimate()</code> provides access to the estimated usage and quota for the origin storage unit's segment of the ''non-persistent global quota''.
 +
 
 +
<code>getPersistentEstimate()</code> provides access to the estimated usage and quota for the ''global quota'', provided the persistent permission is "granted".
 +
 
 +
The remainder provides access to ''boxes''. An origin storage unit's default box can be cleared by invoking <code>deleteBox()</code>. Invoking <code>deleteBox()</code> with a non-empty-string argument simply removes that box. <code>addBox()</code>'s first argument cannot be the empty string as it is reserved for the default box.
 +
 
 +
Note: No access is given to a site storage unit for now as it crosses the ordinary security boundary.
 +
 
 +
interface StorageBox {
 +
    readonly attribute DOMString id;
 +
    readonly  attribute StorageMode mode;
 +
    readonly attribute unsigned long long usageEstimate;
 +
};
 +
 +
dictionary StorageBoxInit {
 +
  StorageMode mode = "atomic";
 
  };
 
  };
 
   
 
   
  enum StorageMode { "best-effort", "persistent" };
+
interface StorageInfo {
 +
  readonly attribute unsigned long long usage;
 +
  readonly attribute unsigned long long quota;
 +
};
 +
 +
  enum StorageMode { "atomic", "cache", "persistent" };
 +
enum PersistentStoragePermission { "default", "denied", "granted" }; // merge with NotificationPermission?
  
<code>navigator.storage</code> returns a <code>StorageManager</code>.
+
=== Rollout ===
  
<code>mode</code> returns the storage mode for the storage scope (eTLD+1 or origin depending on the implementation), with "<code>best-effort</code>" as default.
+
If this is too much at once, we can partition it as such:
  
<code>requestPersistent()</code> changes the storage mode if the user agrees. Fulfills with undefined, rejects with a TypeError if the dialog is dismissed or not acted upon. (Or should it fulfill with <code>StorageMode</code> always?)
+
* v1: Persistent permission and usage/quota estimates.
 +
* v2: Boxes.
  
<code>getEstimate()</code> returns storage information for the storage scope.
+
== Existing APIs ==
 +
 
 +
These APIs need to be (re)written in terms of the new low-level storage primitives so that they have a common grounding.
 +
 
 +
* Cookies
 +
* localStorage
 +
* sessionStorage
 +
* Indexed DB
 +
* Cache API (from service workers, to be clear)
 +
* History API
 +
* Notifications API
 +
* Appcache (legacy)
 +
* Filesystem (legacy, might make a comeback)
 +
* WebSQL (legacy)
 +
 
 +
HTTP cache is probably distinct.
  
v2: Explicit temporary storage API (does not affect the best-effort/persistent area), or perhaps named storage areas (divide up a storage scope).
 
  
 
== FAQ ==
 
== FAQ ==
Line 71: Line 176:
 
=== Why no eviction event? ===
 
=== Why no eviction event? ===
  
This has been tried on other systems and they always end up with /tmp or some such. When you need space you don't want to ask a dozen applications to clear some before you can start allocating. You want to be able to wipe immediately. That is why priority-based temporary storage is preferable.
+
There are two main reasons to avoid this:
 +
 
 +
* Does not allow for global reasoning by the user agent
 +
* Tragedy of the commons
 +
 
 +
Also, this has been tried on other systems and they always end up with <code>/tmp</code> or some such. When you need space you don't want to ask a dozen applications to clear some before you can start allocating or ask the user to remove something.
 +
 
 +
=== What is the relationship to the Quota API? ===
 +
 
 +
The main difference is that requesting a specific quota from the user is not possible as making that work UX-wise is not possible. This API also puts emphasis on usage and quota being estimates.
 +
 
 +
 
 +
== Elsewhere ==
 +
 
 +
* https://github.com/w3ctag/spec-reviews/blob/master/2014/02/quota-management-api.md
 +
* https://w3c.github.io/quota-api/
 +
* https://github.com/w3c/quota-api/issues
 +
* https://github.com/slightlyoff/StorageDurability/issues
 +
* https://bugzilla.mozilla.org/show_bug.cgi?id=1147820

Latest revision as of 00:22, 27 June 2015

This document is obsolete.

For the current specification, see: Storage Standard


Goals

  • Provided a unified architecture for storage on the web.
  • Enable persistent storage with the same "unlimited" quota as native.
  • Enable applications to more effectively manage their storage.

Use cases

Music

User visits a music application and decides they want to sign up and store 200 music files so they are available offline as the user will travel abroad the next day. It is imperative that these songs cannot be removed without explicit consent from the user.

Games

User plays an RPG for which you need to be online. To improve the user experience the RPG wants to cache as much as possible of the user's current location, nearby locations, and further away locations in game, in decreasing order of priority. So that when the user agent requires space the locations the user is furthest away from will be wiped first. The engine is expected to be stored in persistent storage, though preferable without asking the user.

Photos & video

User wants to store photos taken on a trip abroad without connectivity without losing them. User wants to store photos and always have them available offline for sharing with others.

Social

A social network wants to temporarily cache the latest activities locally.

Maps

Have the directions to your destination abroad locally available so you can avoid roaming.

News & books

Have news articles or books available offline to read on the London Tube.

Infrastructure

A user agent has zero or more site storage units. A site storage unit's scope is eTLD+1. A site storage unit consists of a 'default box' and zero or more origin storage units.

An origin storage unit's cope is origin. An origin unit consists of a default box and zero or more boxes.

HTTP cookies are stored in a site storage unit's default box.

localStorage, sessionStorage, IDB, Service Workers, Cache API, History API, and Notifications API are stored in an origin storage unit's default box.

Boxes

A box is where the bytes end up. A box has an id (default box's id is the empty string) and a mode ("atomic", "cache", or "persistent"). Storage APIs that support boxes can associate themselves with a box by passing it in.

Note: going forward we might add the ability to associate UI labels with boxes so end-users can make decisions on a per-box basis as well.

// If the persistent permission is granted, you can add boxes that are persistent
navigator.storage.addBox("the-good-wife-s01", { mode: "persistent" }).then(
  box => self.caches.open("the-good-wife-s01", { box: box }) // it might be worth changing .open() to be simpler
)
// Not all APIs will support "cache" boxes, but it's a natural fit for the Cache API
// even though by default it will be "atomic"
navigator.storage.addBox("localcache", { mode: "cache" }).then(
  box => self.caches.open("localcache", { box: box })
)
// Games might want to store levels in distinct boxes so that when the user needs to
// free up space the game engine remains intact while levels not played for a while
// get cleared.
navigator.storage.addBox("game-level-1", { mode: "atomic" }).then(
  box => indexedDB.open("game-level-1", 1, { box: box }) // again, it might be worth changing .open() to be simpler
)

Atomic boxes

An "atomic" box is cleared in its entirety. User agents are encouraged to reason globally about all "atomic" boxes in such a way that boxes the user least likely needs are cleared before others without paying much heed to site or origin storage unit boundaries.

User agents could potentially reason globally about "atomic" boxes and individual items of "cache" boxes together.

Cache boxes

A "cache" box consists of a sequence of items. What exactly constitutes an item is up to the API that supports a "cache" box. User agents are encouraged to reason globally about all items in all "cache" boxes in such a way that items the user least likely needs are cleared before others.

Note: A "cache" box is distinct from an "atomic" (and "persistent") box in that it provides no all-or-nothing guarantees. It's ideal for storing resources that help improve performance but are not essential.

Persistent boxes

Through a user-mediated action an origin can acquire persistent storage. This ensures that origin storage unit's default box is persistently stored and will not be wiped without user consent. Persistence can be granted through navigator.storage.requestPersistent(), through bookmarking, or a combination of sorts. The expectation is that user agents will converge on a suitable pattern over time.

A site storage unit's default box cannot obtain persistence and will always be atomic.

Quota

The global quota is the available hard drive space minus any space the user agent wants to reserve. The non-persistent global quota is the limit of storage available within the global quota to "atomic" and "cache".

A site storage unit quota is a segment of the non-persistent global quota. User agents are free to let factors such as navigation frequency, bookmarking, and other indications of popularity impact the site storage unit quota.

An origin storage unit quota is that of its site storage unit, with the exception of its boxes that use "persistent" rather than other types of storage. "Persistent" boxes may use the global quota.


API

[NoInterfaceObject, 
 Exposed=(Window,Worker)]
interface NavigatorStorage {
  readonly attribute StorageManager storage;
};
Navigator implements NavigatorStorage;
WorkerNavigator implements NavigatorStorage;

This ensures the API is available everywhere under navigator.storage. (At some point we should probably clean up this pattern.)

[Exposed=(Window,Worker)]
interface StorageManager {
  [Exposed=Window] Promise<PersistentStoragePermission> requestPersistent();
  Promise<PersistentStoragePermission> persistentPermission();

  Promise<StorageInfo]> getEstimate();
  Promise<StorageInfo> getPersistentEstimate();
  
  Promise<StorageBox> addBox(DOMString id, StorageBoxInit init);
  Promise<boolean> deleteBox(optional DOMString id = "");
  Promise<sequence<StorageBox>> boxes();
};

StorageManager provides access to getting permission for "persistent" boxes (and changing the origin storage unit's default box to use it).

getEstimate() provides access to the estimated usage and quota for the origin storage unit's segment of the non-persistent global quota.

getPersistentEstimate() provides access to the estimated usage and quota for the global quota, provided the persistent permission is "granted".

The remainder provides access to boxes. An origin storage unit's default box can be cleared by invoking deleteBox(). Invoking deleteBox() with a non-empty-string argument simply removes that box. addBox()'s first argument cannot be the empty string as it is reserved for the default box.

Note: No access is given to a site storage unit for now as it crosses the ordinary security boundary.

interface StorageBox {
   readonly attribute DOMString id;
   readonly  attribute StorageMode mode;
   readonly attribute unsigned long long usageEstimate;
};

dictionary StorageBoxInit {
  StorageMode mode = "atomic";
};

interface StorageInfo {
  readonly attribute unsigned long long usage;
  readonly attribute unsigned long long quota;
};

enum StorageMode { "atomic", "cache", "persistent" };
enum PersistentStoragePermission { "default", "denied", "granted" }; // merge with NotificationPermission?

Rollout

If this is too much at once, we can partition it as such:

  • v1: Persistent permission and usage/quota estimates.
  • v2: Boxes.

Existing APIs

These APIs need to be (re)written in terms of the new low-level storage primitives so that they have a common grounding.

  • Cookies
  • localStorage
  • sessionStorage
  • Indexed DB
  • Cache API (from service workers, to be clear)
  • History API
  • Notifications API
  • Appcache (legacy)
  • Filesystem (legacy, might make a comeback)
  • WebSQL (legacy)

HTTP cache is probably distinct.


FAQ

Why no eviction event?

There are two main reasons to avoid this:

  • Does not allow for global reasoning by the user agent
  • Tragedy of the commons

Also, this has been tried on other systems and they always end up with /tmp or some such. When you need space you don't want to ask a dozen applications to clear some before you can start allocating or ask the user to remove something.

What is the relationship to the Quota API?

The main difference is that requesting a specific quota from the user is not possible as making that work UX-wise is not possible. This API also puts emphasis on usage and quota being estimates.


Elsewhere