A user account is required in order to edit this wiki, but we've had to disable public user registrations due to spam.

To request an account, ask an autoconfirmed user on IRC (such as one of these permanent autoconfirmed members).

Difference between revisions of "Storage"

From WHATWG Wiki
Jump to: navigation, search
(FAQ)
Line 1: Line 1:
== Existing APIs ==
 
  
* Cookies
+
== Goals ==
* localStorage
 
* sessionStorage
 
* Indexed DB
 
* Cache API (from service workers, to be clear)
 
* History API
 
* Notifications API
 
  
Legacy:
+
* Provided a unified architecture for storage on the web.
 +
* Enable persistent storage with the same "unlimited" quota as native.
 +
* Enable applications to more effectively manage their storage.
  
* Appcache
+
== Use cases ==  
* Filesystem (might make a comeback)
 
* WebSQL
 
 
 
HTTP cache is distinct (though might make sense if "best effort" strategies are aligned and such).
 
 
 
== Storage types ==
 
 
 
A storage area's scope is that of eTLD+1 or an origin (depending on the implementation). A storage area's mode is either best effort or persistent. A storage area is atomic and includes all the above APIs.
 
 
 
* Best effort: default on the web, can be upgraded to persistent through the user.
 
* Persistent: resources that need to be available in order for the application to run offline and cannot be removed without user consent. (A distinction can be made between app layer (e.g. anything required to play music) and data layer (offline-available music).)
 
 
 
v2 ideas:
 
 
 
* Temporary: resources that would be useful to cache to reduce latency/bandwidth but are not strictly required when offline. A priority system can be useful so the user agent can determine what to preserve from the temporary storage when running out of the global quota.
 
* Named storage areas: making it possible for a single scope to have several storage areas. Other APIs could be used to mark some temporary, attach priorities, etc.
 
 
 
== Stories ==
 
  
 
=== Music ===
 
=== Music ===
Line 47: Line 24:
 
A social network wants to temporarily cache the latest activities locally.
 
A social network wants to temporarily cache the latest activities locally.
  
== Requirements ==
 
  
* Persistent elastic storage (music, photos, video)
+
== Infrastructure ==
* Priority-based temporary storage (gaming, social)
+
 
 +
A user agent has zero or more '''site storage units'''. A site storage unit's scope is eTLD+1. A site storage unit consists of a 'default box' and zero or more '''origin storage units'''.
 +
 
 +
An origin storage unit's cope is origin. An origin unit consists of a ''default box'' and zero or more ''boxes''.
 +
 
 +
HTTP cookies are stored in a site storage unit's ''default box''.
 +
 
 +
localStorage, sessionStorage, IDB, Service Workers, Cache API, History API, and Notifications API are stored in an origin storage unit's ''default box''.
 +
 
 +
=== Boxes ===
 +
 
 +
A '''box''' is where the bytes end up. A box has an '''id''' (default box's id is the empty string) and a '''mode''' ("atomic", "cache", or "persistent"). Storage APIs that support boxes can associate themselves with a box by passing it in.
 +
 
 +
Note: going forward we might add the ability to associate UI labels with boxes so end-users can make decisions on a per-box basis as well.
 +
 
 +
// If the persistent permission is granted, you can add boxes that are persistent
 +
navigator.storage.addBox("the-good-wife-s01", { mode: "persistent" }).then(
 +
  box => self.caches.open("the-good-wife-s01", { box: box }) // it might be worth changing .open() to be simpler
 +
)
 +
 
 +
// Not all APIs will support "cache" boxes, but it's a natural fit for the Cache API
 +
// even though by default it will be "atomic"
 +
navigator.storage.addBox("localcache", { mode: "cache" }).then(
 +
  box => self.caches.open("localcache", { box: box })
 +
)
 +
 
 +
// Games might want to store levels in distinct boxes so that when the user needs to
 +
// free up space the game engine remains intact while levels not played for a while
 +
// get cleared.
 +
navigator.storage.addBox("game-level-1", { mode: "atomic" }).then(
 +
  box => indexedDB.open("game-level-1", 1, { box: box }) // again, it might be worth changing .open() to be simpler
 +
)
 +
 
 +
==== Atomic boxes ====
 +
 
 +
User agents are encouraged to reason globally about all "atomic" boxes in such a way that boxes the user least likely needs are cleared before others without paying much heed to site or origin storage unit boundaries. If an "atomic" box needs to be cleared it needs to be cleared entirely.
 +
 
 +
==== Cache boxes ====
 +
 
 +
User agents are encouraged to reason globally about all 'individual items' in "cache" boxes in such a way that items the user least likely needs are cleared before others without paying much heed to any perceived boundaries.
 +
 
 +
==== Persistent boxes ====
 +
 
 +
Through a user-mediated action an origin can acquire persistent storage. This ensures that origin storage unit's default box is persistently stored and will not be wiped without user consent. Persistence can be granted through navigator.storage.requestPersistent(), through bookmarking, or a combination of sorts. The expectation is that user agents will converge on a suitable pattern over time.
 +
 
 +
A site storage unit's default box cannot obtain persistence and will always be atomic.
 +
 
 +
=== Quota ===
 +
 
 +
The '''global quota''' is the available hard drive space minus any space the user agent wants to reserve. The ''non-persistent global quota'' is the limit of storage available within the global quota to "atomic" and "cache".
  
== Rough plan ==
+
A '''site storage unit quota''' is a segment of the non-persistent global quota. User agents are free to let factors such as navigation frequency, bookmarking, and other indications of popularity impact the site storage unit quota.
  
Create two modes to back the current set of storage APIs: best effort and persistent. By default the mode for a given eTLD+1 is best effort. After the user opts in it is persistent. Persistent means the user agent does not touch the storage without user consent. It also means that the application gets quite a lot of space. (Ideally most of the hard drive, if that turns into a problem the user will have to delete it...)
+
An '''origin storage unit quota''' is that of its site storage unit, with the exception of its boxes that use "persistent" rather than other types of storage. "Persistent" boxes may use the global quota.
  
v2 ideas: Create a new storage API for explicit priority-based temporary storage. A key/value/priority store, effectively. Application is in charge of determining and changing priorities over time. Or create named storage areas of sorts.
 
  
== API proposal ==
+
== API ==
  
  partial interface Navigator {
+
  [NoInterfaceObject,
 +
  Exposed=(Window,Worker)]
 +
interface NavigatorStorage {
 
   readonly attribute StorageManager storage;
 
   readonly attribute StorageManager storage;
 
  };
 
  };
 +
Navigator implements NavigatorStorage;
 +
WorkerNavigator implements NavigatorStorage;
 
   
 
   
 +
This ensures the API is available everywhere under <code>navigator.storage</code>. (At some point we should probably clean up this pattern.)
 +
 
  [Exposed=(Window,Worker)]
 
  [Exposed=(Window,Worker)]
 
  interface StorageManager {
 
  interface StorageManager {
   readonly attribute StorageMode mode;
+
   Promise<PersistentStoragePermission> requestPersistent();
   Promise<void> requestPersistent();
+
   Promise<PersistentStoragePermission> persistentPermission();
 
   
 
   
   Promise<[https://dvcs.w3.org/hg/quota/raw-file/tip/Overview.html#idl-def-StorageInfo StorageInfo]> getEstimate();
+
   Promise<StorageInfo]> getEstimate();
 +
  Promise<StorageInfo> getPersistentEstimate();
 +
 
 +
  Promise<StorageBox> addBox(DOMString id, StorageBoxInit init);
 +
  Promise<boolean> deleteBox(optional DOMString id = "");
 +
  Promise<sequence<StorageBox>> boxes();
 
  };
 
  };
 
   
 
   
enum StorageMode { "best-effort", "persistent" };
+
<code>StorageManager</code> provides access to getting permission for "persistent" boxes (and changing the origin storage unit's ''default box'' to use it).
  
<code>navigator.storage</code> returns a <code>StorageManager</code>.
+
<code>getEstimate()</code> provides access to the estimated usage and quota for the origin storage unit's segment of the ''non-persistent global quota''.
  
<code>mode</code> returns the storage mode for the storage scope (eTLD+1 or origin depending on the implementation), with "<code>best-effort</code>" as default.
+
<code>getPersistentEstimate()</code> provides access to the estimated usage and quota for the ''global quota'', provided the persistent permission is "granted".
  
<code>requestPersistent()</code> changes the storage mode if the user agrees. Fulfills with undefined, rejects with a TypeError if the dialog is dismissed or not acted upon. (Or should it fulfill with <code>StorageMode</code> always?)
+
The remainder provides access to ''boxes''. An origin storage unit's default box can be cleared by invoking <code>deleteBox()</code>. Invoking <code>deleteBox()</code> with a non-empty-string argument simply removes that box. <code>addBox()</code>'s first argument cannot be the empty string as it is reserved for the default box.
 +
 
 +
No access is given to a site storage unit for now as it crosses the ordinary security boundary.
 +
 
 +
interface StorageBox {
 +
    readonly attribute DOMString id;
 +
    readonly  attribute StorageMode mode;
 +
    readonly attribute unsigned long long usageEstimate;
 +
};
 +
 +
dictionary StorageBoxInit {
 +
  StorageMode mode = "atomic";
 +
};
 +
 +
interface StorageInfo {
 +
  readonly attribute unsigned long long usage;
 +
  readonly attribute unsigned long long quota;
 +
};
 +
 +
enum StorageMode { "atomic", "cache", "persistent" };
 +
enum PersistentStoragePermission { "default", "denied", "granted" }; // merge with NotificationPermission?
 +
 
 +
== Existing APIs ==
 +
 
 +
These APIs need to be (re)written in terms of the new low-level storage primitives so that they have a common grounding.
 +
 
 +
* Cookies
 +
* localStorage
 +
* sessionStorage
 +
* Indexed DB
 +
* Cache API (from service workers, to be clear)
 +
* History API
 +
* Notifications API
 +
* Appcache (legacy)
 +
* Filesystem (legacy, might make a comeback)
 +
* WebSQL (legacy)
 +
 
 +
HTTP cache is probably distinct.
  
<code>getEstimate()</code> returns storage information for the storage scope.
 
  
 
== FAQ ==
 
== FAQ ==
Line 91: Line 162:
  
 
The main difference is that requesting a specific quota from the user is not possible as making that work UX-wise is not possible. This API also puts emphasis on usage and quota being estimates.
 
The main difference is that requesting a specific quota from the user is not possible as making that work UX-wise is not possible. This API also puts emphasis on usage and quota being estimates.
 +
 +
 +
== Elsewhere ==
 +
 +
* https://github.com/w3ctag/spec-reviews/blob/master/2014/02/quota-management-api.md
 +
* https://w3c.github.io/quota-api/
 +
* https://github.com/w3c/quota-api/issues
 +
* https://github.com/slightlyoff/StorageDurability/issues

Revision as of 09:08, 7 April 2015

Goals

  • Provided a unified architecture for storage on the web.
  • Enable persistent storage with the same "unlimited" quota as native.
  • Enable applications to more effectively manage their storage.

Use cases

Music

User visits a music application and decides they want to sign up and store 200 music files so they are available offline as the user will travel abroad the next day. It is imperative that these songs cannot be removed without explicit consent from the user.

Games

User plays an RPG for which you need to be online. To improve the user experience the RPG wants to cache as much as possible of the user's current location, nearby locations, and further away locations in game, in decreasing order of priority. So that when the user agent requires space the locations the user is furthest away from will be wiped first. The engine is expected to be stored in persistent storage, though preferable without asking the user.

Photos & video

User wants to store photos taken on a trip abroad without connectivity without losing them. User wants to store photos and always have them available offline for sharing with others.

Social

A social network wants to temporarily cache the latest activities locally.


Infrastructure

A user agent has zero or more site storage units. A site storage unit's scope is eTLD+1. A site storage unit consists of a 'default box' and zero or more origin storage units.

An origin storage unit's cope is origin. An origin unit consists of a default box and zero or more boxes.

HTTP cookies are stored in a site storage unit's default box.

localStorage, sessionStorage, IDB, Service Workers, Cache API, History API, and Notifications API are stored in an origin storage unit's default box.

Boxes

A box is where the bytes end up. A box has an id (default box's id is the empty string) and a mode ("atomic", "cache", or "persistent"). Storage APIs that support boxes can associate themselves with a box by passing it in.

Note: going forward we might add the ability to associate UI labels with boxes so end-users can make decisions on a per-box basis as well.

// If the persistent permission is granted, you can add boxes that are persistent
navigator.storage.addBox("the-good-wife-s01", { mode: "persistent" }).then(
  box => self.caches.open("the-good-wife-s01", { box: box }) // it might be worth changing .open() to be simpler
)
// Not all APIs will support "cache" boxes, but it's a natural fit for the Cache API
// even though by default it will be "atomic"
navigator.storage.addBox("localcache", { mode: "cache" }).then(
  box => self.caches.open("localcache", { box: box })
)
// Games might want to store levels in distinct boxes so that when the user needs to
// free up space the game engine remains intact while levels not played for a while
// get cleared.
navigator.storage.addBox("game-level-1", { mode: "atomic" }).then(
  box => indexedDB.open("game-level-1", 1, { box: box }) // again, it might be worth changing .open() to be simpler
)

Atomic boxes

User agents are encouraged to reason globally about all "atomic" boxes in such a way that boxes the user least likely needs are cleared before others without paying much heed to site or origin storage unit boundaries. If an "atomic" box needs to be cleared it needs to be cleared entirely.

Cache boxes

User agents are encouraged to reason globally about all 'individual items' in "cache" boxes in such a way that items the user least likely needs are cleared before others without paying much heed to any perceived boundaries.

Persistent boxes

Through a user-mediated action an origin can acquire persistent storage. This ensures that origin storage unit's default box is persistently stored and will not be wiped without user consent. Persistence can be granted through navigator.storage.requestPersistent(), through bookmarking, or a combination of sorts. The expectation is that user agents will converge on a suitable pattern over time.

A site storage unit's default box cannot obtain persistence and will always be atomic.

Quota

The global quota is the available hard drive space minus any space the user agent wants to reserve. The non-persistent global quota is the limit of storage available within the global quota to "atomic" and "cache".

A site storage unit quota is a segment of the non-persistent global quota. User agents are free to let factors such as navigation frequency, bookmarking, and other indications of popularity impact the site storage unit quota.

An origin storage unit quota is that of its site storage unit, with the exception of its boxes that use "persistent" rather than other types of storage. "Persistent" boxes may use the global quota.


API

[NoInterfaceObject, 
 Exposed=(Window,Worker)]
interface NavigatorStorage {
  readonly attribute StorageManager storage;
};
Navigator implements NavigatorStorage;
WorkerNavigator implements NavigatorStorage;

This ensures the API is available everywhere under navigator.storage. (At some point we should probably clean up this pattern.)

[Exposed=(Window,Worker)]
interface StorageManager {
  Promise<PersistentStoragePermission> requestPersistent();
  Promise<PersistentStoragePermission> persistentPermission();

  Promise<StorageInfo]> getEstimate();
  Promise<StorageInfo> getPersistentEstimate();
  
  Promise<StorageBox> addBox(DOMString id, StorageBoxInit init);
  Promise<boolean> deleteBox(optional DOMString id = "");
  Promise<sequence<StorageBox>> boxes();
};

StorageManager provides access to getting permission for "persistent" boxes (and changing the origin storage unit's default box to use it).

getEstimate() provides access to the estimated usage and quota for the origin storage unit's segment of the non-persistent global quota.

getPersistentEstimate() provides access to the estimated usage and quota for the global quota, provided the persistent permission is "granted".

The remainder provides access to boxes. An origin storage unit's default box can be cleared by invoking deleteBox(). Invoking deleteBox() with a non-empty-string argument simply removes that box. addBox()'s first argument cannot be the empty string as it is reserved for the default box.

No access is given to a site storage unit for now as it crosses the ordinary security boundary.

interface StorageBox {
   readonly attribute DOMString id;
   readonly  attribute StorageMode mode;
   readonly attribute unsigned long long usageEstimate;
};

dictionary StorageBoxInit {
  StorageMode mode = "atomic";
};

interface StorageInfo {
  readonly attribute unsigned long long usage;
  readonly attribute unsigned long long quota;
};

enum StorageMode { "atomic", "cache", "persistent" };
enum PersistentStoragePermission { "default", "denied", "granted" }; // merge with NotificationPermission?

Existing APIs

These APIs need to be (re)written in terms of the new low-level storage primitives so that they have a common grounding.

  • Cookies
  • localStorage
  • sessionStorage
  • Indexed DB
  • Cache API (from service workers, to be clear)
  • History API
  • Notifications API
  • Appcache (legacy)
  • Filesystem (legacy, might make a comeback)
  • WebSQL (legacy)

HTTP cache is probably distinct.


FAQ

Why no eviction event?

This has been tried on other systems and they always end up with /tmp or some such. When you need space you don't want to ask a dozen applications to clear some before you can start allocating. You want to be able to wipe immediately. That is why priority-based temporary storage is preferable.

What is the relationship to the Quota API?

The main difference is that requesting a specific quota from the user is not possible as making that work UX-wise is not possible. This API also puts emphasis on usage and quota being estimates.


Elsewhere