WHATWG Wiki - User contributions [en]

OffscreenCanvas.requestAnimationFrame

2017-01-18T17:11:34Z

Junov: Adding the commit solution to the proposal

:''This proposal aims to provide a reliable mechanism for driving animations using OffscreenCanvas in a Worker''

== Use Case Description ==
An OffscreenCanvas is used in a worker to produce a sequence of frames that constitute an animation. The frames need to be produced at regular intervals that match the frame rate of the display

=== Current Limitations ===

=== Current Usage and Workarounds ===
Currently, it is possible to use setTimeout or setInterval in a worker to invoke an animation callback at a regular interval. However, since this mechanism is not driven by the display, the following issues arise:
* The display device's refresh rate needs to be guessed.
* Even if the correct rate is guessed, drift in the timing will cause dropped frames and skipped frames.
* In GPU-accelerated use cases, it is possible for the rendering script to run at a rate that the GPU cannot keep-up with, which may cause the accumulation of a multi-frame rendering backlog, resulting in high latency and eventually OOM crashes. A possible solution to this is for the user agent to implement a throttling mechanism, in which case the worker's event loop may be periodically de-scheduled while the GPU catches up. Such de-scheduling is bad because it prevents the worker from doing other work.

Another solution is to have a requestAnimationFrame loop in the browsing context's event loop that posts a message to the worker at each animation iteration.
* This mechanism may add undue latency to the signal, especially when the browsing context's event loop is busy, which completely destroys one of the key advantages of using OffscreenCanvas in a worker.
* The the frame rate in the browsing context's event loop may be higher than the worker can keep up which which will require a throttling mechanism to be implemented in script
* As with setTimeout/setInterval, it is possible for the rendering script to run at a rate that the GPU cannot keep-up with.

=== Benefits ===

New API to drive OffscreenCanvas animations can ensure optimal frame rate, minimize animation jank, prevent overdraw, and minimize display latency.

=== Requests for this Feature ===

* <cite>[https://github.com/w3ctag/spec-reviews/issues/141 W3C TAG review for ImageBitmapRenderingContext]</cite><blockquote><p>(...) there seems to be a good bit of concern about lack of things like requestAnimationFrame (...) -- L. David Barron, Mozilla</p></blockquote>
* <cite>[https://lists.w3.org/Archives/Public/public-whatwg-archive/2015Aug/0019.html whatwg mailing list thread "Worker requestAnimationFrame"]</cite> <blockquote><p>For OffscreenCanvas we need a way for a Worker to draw once per composited frame. --Robert O'Callahan, Mozilla
</p></blockquote>

== Proposed Solutions ==

=== OffscreenCanvas.requestAnimationFrame ===

Works much like window.requestAnimationFrame, except that the scheduling of callbacks is independent of the browsing context event loop, and therefore is not necessarily synchronized with graphics updates from the browsing context.

==== Processing Model ====

The requestAnimationFrame() and cancelAnimationFrame() methods shall be spec'ed almost identically to their Window interface counterparts, except that the callback list would be stored in the OffscreenCanvas object.

The main difference with respect to the Window.requestAnimationFrame processing model is in how the callbacks are scheduled. In a browsing context, the animation callbacks are coordinated with the graphics update in the [https://html.spec.whatwg.org/multipage/webappapis.html#event-loop-processing-model event loop processing model]. Such coordination shall not exist in workers.

For OffscreenCanvases, the user agent will schedule an "animation task" to run all of the OffscreenCanvas's pending animation callbacks. in a way the respects the following constraints:
* No overdraw: An animation frame committed from an animation task shall not replace an animation frame from a previous animation task until that previous frame has been rendered to the display device.
* animation tasks shall be scheduled at the highest possible rate that can be maintained without going into overdraw and without accumulating a backlog of more than one pending frame.

===== Special Cases =====

* When rAF is invoked on an OffscreenCanvas that does not have a placeholder canvas and is not linked to a VRLayer, throw an InvalidStateError. The OffscreenCanvas content must be composited for the rAF processing model to make sense.
* Attempting to transfer an OffscreenCanvas object with a non-empty animation callback list throws an InvalidStateError.
* Attempting to construct a VRLayer using an OffscreenCanvas object with a non-empty animation callback list throws an InvalidStateError.
* When the OffscreenCanvas is associated with a VRLayer, all calls to {request|cancel}AnimationFrame must be forwarded to the VRLayer's VRDisplay's {request|cancel}AnimationFrame methods. This implies that when the OffscreenCanvas simultaneously is visible through a placeholder canvas and a VR device, the animation loop is driven by the VR device.
* The animations tasks for different OffscreenCanvas objects that live in the same event loop are not necessarily synchronized.

==== Open issues ====
Calling commit() on a given OffscreenCanvas multiple times in the same animation frame is problematic. Possible way of handling the situation:
* Drop all commits but the last one (or the first one?)
* Queue multiple frames and wait for all of them to have be displayed before scheduling the next animation task
* Throw an exception

What to do if commit() is not called from within the animation callback? This is problematic because
* Do an implicit commit()
* Repeat the previous frame
* Schedule the next animation frame immediately.
* Prevent this from ever happening: Let OffscreenCanvas object have a needsCommit flag that is initially false. Set needsCommit to true at the beginning of an animation task. Set needsCommit to false when commit is called. When requestAnimationFrame is called, throw an exception if needsCommit is true.

Should it be possible to commit() the contents of other canvases from within a rAF callback?

=== OffscreenCanvas.commit() to return a promise ===

An alternate solution would be to have commit() return a promise that gets resolved when it is time to begin rendering the next frame. This single API entry-point provides the necessary flexibility to handle continuous animations as well as sporadic updates.

Continuous animation example:

<nowiki>function animationLoop() {
// draw stuff
(...)
ctx.commit().then(animationLoop);
// do post commit work
(...)
}</nowiki>

Another possibility is to use the async/await syntax:

<nowiki>async function animationLoop() {
var promise;
do {
//draw stuff
(...)
promise = ctx.commit()
// do post commit work
(...)
} while (await promise);
}</nowiki>

To animate multiple canvases in lock-step, one could do this, for eaxample:

<nowiki>function animationLoop() {
// draw stuff
(...)
Promise.all([ctx1.commit(), ctx2.commit()]).then(animationLoop);
}</nowiki>

For occasional update use cases, it is just a matter of ignoring the promise returned by commit() and to drive the animation using another signal, for example a network event. In the case where multiple calls to commit are made in the same frame interval, the user agent skips frames in order to avoid accumulating a multi-frame backlog, as described in the processing model below.

==== Processing Model ====

An OffscreenCanvas object has a ''pendingFrame'' internal slot that stores a reference to the frame that was captured by the last call to commit(). The reference is held until the frame is actually committed. 'pendingFrame' is initially unset.

An OffscreenCanvas object has a ''pendingPromise'' internal slot that stores a reference to the promise that was returned by the last call to commit(). [[pendingPromise]] is initially unset, and its reference is only retained while the promise is in the unresolved state.

The ''BeginFrame'' signal is a signal that is dispatched by the UserAgent to a specific OffscreenCanvas when it is time to render the next animation frame for the OffscreenCanvas.

When commit() is called:
# Let ''frame'' be a copy of the current contents of the canvas.
# If ''pendingPromise'' is set, then run theses substeps:
## Set ''pendingFrame'' to be a reference to ''frame''.
## Return ''pendingPromise''.
# Set ''pendingPromise'' to be a newly created unresolved promise object.
# Run the steps to '''commit a frame''', passing ''frame'' as an argument.
# Return ''pendingPromise''.

When the ''BeginFrame'' signal is to be dispatched to an OffscreenCanvas object, the UserAgent must queue a task on the OffscreenCanvas object's event loop that runs the following steps:
# If ''pendingFrame'' is set, then run the following substeps:
## Run the steps to '''commit a frame''', passing [[pendingFrame]] as an argument.
## Unset ''pendingFrame''.
## Abort these steps.
# If ''pendingPromise'' is not set then abort these steps.
# Resolve the promise referenced by ''pendingPromise''.
# Unset ''pendingPromise''.

When the user agent is required to run the steps to '''commit a frame''', it must do what is currently spec'ed as the steps for commit().

This processing model takes care the unresolved issues with the OffscreenCanvas.requestAnimationFrame solution because it makes it safe to call commit at any time by providing the following guarantees:
# In cases of overdraw (commit() called at a rate higher than can be displayed), frames may be dropped to ensure low latency (no more than one frame of backlog).
# The frame captured by the last call to commit after the end of an animation sequence is never dropped. In other words, when animation stops, it is always the most recent frame that is displayed.

==== Adoption ====

The idea of the commit API was discussed at a meeting of the WebVR working group and has support from multiple browser vendors.

[[Category:Proposals]]

OffscreenCanvas.requestAnimationFrame

2016-12-02T18:58:14Z

Junov: Created page with ":''This proposal aims to provide a reliable mechanism for driving animations using OffscreenCanvas in a Worker'' == Use Case Description == An OffscreenCanvas is used in a wo..."

OffscreenCanvas

2016-11-30T18:14:50Z

Junov:

'''This proposal is no longer active. OffscreenCanvas is now in the HTML specification [https://html.spec.whatwg.org/multipage/scripting.html#the-offscreencanvas-interface here].'''

:''Provides more control over how canvases are rendered. This is a follow-on to the [[WorkerCanvas]] proposal and will be merged once agreement is reached.''

== Use Case Description ==

Feedback from web application authors using canvases have shown the need for the following controls:

* (From ShaderToy, Sketchfab, Verold): need to be able to render to multiple regions on the page efficiently using a single canvas context. 3D model warehouse sites desire to show multiple live interactive models on the page, but creating multiple WebGL contexts per page is too inefficient. A single context should be able to render to multiple regions on the page.
* (From Google Maps): need to be able to render WebGL from a worker, transfer the rendered image to the main thread without making any copy of it, and composite it with other HTML on the page, guaranteeing that the updates are all seen in the same rendered frame.
* (From Mozilla and partners using Emscripten and asm.js): need to be able to render WebGL entirely asynchronously from a worker, displaying the results in a canvas owned by the main thread, without any synchronization with the main thread. In this mode, the entire application runs in the worker. The main thread only receives input events and sends them to the worker for processing.
* (From adopters of the Push API): need to be able to dynamically create images to use as notification icons, such as compositing avatars, or adding an unread count
* (From the Google Docs team): need to be able to layout and render text from a worker using CanvasRenderingContext2D and display those results on the main thread.
* (From the Google Slides team): want to layout and render the slide thumbnails from a worker. During initial load and heavy collaboration these update frequently, and currently cause slowdowns on the main thread.

=== Current Limitations ===

* [https://html.spec.whatwg.org/multipage/scripting.html#proxying-canvases-to-workers CanvasProxy] does not provide sufficient control to allow synchronization between workers' rendering and DOM updates on the main thread. Keeping this rendering in sync is a requirement from Google's Maps team.
* [[CanvasInWorkers]] does not allow a worker to render directly into a canvas on the main thread without running code on the main thread. Allowing completely unsynchronized rendering is a requirement from Mozilla and users of Emscripten such as Epic Games and Unity, in which the desire is to execute all of the game's rendering on a worker thread.
* [[WorkerCanvas]] mostly addresses these two use cases, but some implementers objected to the mechanism for displaying the rendering results in image elements. The specific objection was that image elements already have complex internal state (for example, the management of the image's "loaded" state), and this would make it more complex. It also did not precisely address the use case of producing new frames both on the main thread and in workers.

=== Current Usage and Workarounds ===

[https://blog.mozilla.org/research/2014/07/22/webgl-in-web-workers-today-and-faster-than-expected/ WebGL in Web Workers] details some work attempted in the Emscripten toolchain to address the lack of WebGL in workers. Due to the high volume of calls and large amount of data that is transferred to the graphics card in a typical high-end WebGL application, this approach is not sustainable. It's necessary for workers to be able to call the WebGL API directly, and present those results to the screen in a manner that does not introduce any copies of the rendering results.

=== Benefits ===

Making canvas rendering contexts available to workers will increase parallelism in web applications, leading to increased performance on multi-core systems.

=== Requests for this Feature ===

See the abovementioned use cases:

* Google's Maps team
* Emscripten users such as Epic Games and Unity
* Many others

== Web IDL ==

typedef (OffscreenCanvasRenderingContext2D or
WebGLRenderingContext or
WebGL2RenderingContext) OffscreenRenderingContext;

[Constructor(unsigned long width, unsigned long height),
Exposed=(Window,Worker)]
interface OffscreenCanvas {
attribute unsigned long width;
attribute unsigned long height;
OffscreenRenderingContext? getContext(DOMString contextId, any... arguments);

// OffscreenCanvas, like HTMLCanvasElement, maintains an origin-clean flag.
// ImageBitmaps created by calling this method also have an
// origin-clean flag which is set to the value of the OffscreenCanvas's
// flag at the time of their construction. Uses of the ImageBitmap
// in other APIs, such as CanvasRenderingContext2D or
// WebGLRenderingContext, propagate this flag like other
// CanvasImageSource types do, such as HTMLImageElement.
ImageBitmap transferToImageBitmap();

// Throws a SecurityError if the OffscreenCanvas's origin-clean flag
// is set to false.
Promise<Blob> convertToBlob(optional ImageEncodeOptions options);
};

dictionary ImageEncodeOptions {
DOMString type = "image/png";
unrestricted double quality = 1.0; // Defaults to 1.0 if value is outside 0:1 range
};

OffscreenCanvas implements Transferable;

partial interface HTMLCanvasElement {
OffscreenCanvas transferControlToOffscreen();
};

typedef (HTMLOrSVGImageElement or
HTMLVideoElement or
HTMLCanvasElement or
ImageBitmap or
OffscreenCanvas) CanvasImageSource;

[Exposed=Window, Worker]
interface OffscreenCanvasRenderingContext2D {
// commit() can only be used when HTMLCanvasElement has transferred Control
// to OffscreenCanvas. Otherwise, an InvalidStateError will be thrown.
// commit() can be invoked on main thread or worker thread. When it is invoked,
// it is expected to see the image drawn to OffscreenCanvasRenderingContext2D
// be displayed in the associated HTMLCanvasElement.
void commit();
// back-reference to the canvas
readonly attribute OffscreenCanvas canvas;
};
OffscreenCanvasRenderingContext2D implements CanvasState;
OffscreenCanvasRenderingContext2D implements CanvasTransform;
OffscreenCanvasRenderingContext2D implements CanvasCompositing;
OffscreenCanvasRenderingContext2D implements CanvasImageSmoothing;
OffscreenCanvasRenderingContext2D implements CanvasFillStrokeStyles;
OffscreenCanvasRenderingContext2D implements CanvasShadowStyles;
// Reference filters (e.g. 'url()') are not expected to work in Workers
OffscreenCanvasRenderingContext2D implements CanvasFilters;
OffscreenCanvasRenderingContext2D implements CanvasRect;
OffscreenCanvasRenderingContext2D implements CanvasDrawPath;
// Text support in workers poses very difficult technical challenges.
// Open issue: should we forgo text support in OffscreenCanvas v1?
OffscreenCanvasRenderingContext2D implements CanvasText;
OffscreenCanvasRenderingContext2D implements CanvasDrawImage;
OffscreenCanvasRenderingContext2D implements CanvasImageData;
OffscreenCanvasRenderingContext2D implements CanvasPathDrawingStyles;
OffscreenCanvasRenderingContext2D implements CanvasTextDrawingStyles;
OffscreenCanvasRenderingContext2D implements CanvasPath;

[Exposed=Window, Worker]
Partial interface CanvasPattern {
}

[Exposed=Window, Worker]
partial interface CanvasGradient {
}

partial interface WebGLRenderingContextBase {
// back-reference to the canvas
readonly attribute (HTMLCanvasElement or OffscreenCanvas) canvas;

// If this context is associated with an OffscreenCanvas that was
// created by HTMLCanvasElement's transferControlToOffscreen method,
// causes this context's current rendering results to be pushed
// to that canvas element. This has the same effect as returning
// control to the main loop in a single-threaded application. Otherwise,
// an InvalidStateError will be thrown.
void commit();
};

== Proposed Solutions ==

=== This Solution ===

This proposed API can be used in several ways to satisfy the use cases described above:

* It supports zero-copy transfer of canvases' rendering results between threads, for example from a worker to the main thread. In this model, the main thread controls when to display new frames produced by the worker, so synchronization with other DOM updates is achieved.

* It supports fully asynchronous rendering by a worker into a canvas displayed on the main thread. This satisfies certain Emscripten developers' full-screen use cases.

* It supports using a single WebGLRenderingContext or Canvas2DRenderingContext to efficiently render into multiple regions on the web page.

* It introduces ImageBitmapRenderingContext, a new canvas context type whose sole purpose is to efficiently display ImageBitmaps. This supersedes the [[WorkerCanvas]] proposal's use of HTMLImageElement for this purpose.

* It supports asynchronous encoding of OffscreenCanvases' rendering results into Blobs which can be consumed by various other web platform APIs.

==== Processing Model ====

This proposal introduces two primary processing models. The first involves ''synchronous'' display of new frames produced by the OffscreenCanvas. The application generates new frames using the RenderingContext obtained from the OffscreenCanvas. When the application is finished rendering each new frame, it calls transferToImageBitmap to "tear off" the most recently rendered image from the OffscreenCanvas -- like a Post-It note. The resulting ImageBitmap can then be used in any API receiving that data type; notably, it can be displayed in a second canvas without introducing a copy. An ImageBitmapRenderingContext is obtained from the second canvas by calling <code>getContext('bitmaprenderer')</code>. Each frame is displayed in the second canvas using the <code>transferImageBitmap</code> method on this rendering context. Note that the threads producing and consuming the frames may be the same, or they may be different. Note also that a single OffscreenCanvas may transfer frames into an arbitrary number of other ImageBitmapRenderingContexts.

The second processing model involves ''asynchronous'' display of new frames produced by the OffscreenCanvas. The main thread instantiates an HTMLCanvasElement and calls <code>transferControlToOffscreeen</code>. <code>getContext</code> is used to obtain a rendering context for that OffscreenCanvas, either on the main thread, or on a worker. The application calls <code>commit</code> against that rendering context in order to push frames to the original HTMLCanvasElement. In this rendering model, it is not defined when those frames become visible in the original canvas element. However, if the following situations apply:

* It is a worker thread which is calling commit(), and
* The worker is calling commit() repeatedly against exactly one rendering context

then it is required that the user agent synchronize the calls to commit() to the vsync interval. Calls to commit() conceptually enqueue frames for display, and after an implementation-defined number of frames have been enqueued, further calls to commit() will block until earlier frames have been presented to the screen. (This requirement allows porting of applications which drive their own main loop rather than using an event-driven loop.)

==== Limitations ====

* A known good way to drive an animation loop from a worker is needed. requestAnimationFrame or a similar API needs to be defined on worker threads.
* Some parts of the CanvasRenderingContext2D interface shall not be supported due OffscreenCanvas objects having no relation to the DOM or Frame: HitRegions, scrollPathIntoView, drawFocusIfNeeded.
* Due to technical challenges, some implementors [https://bugzilla.mozilla.org/show_bug.cgi?id=801176#c29 (Google and Mozilla)] have expressed a desire to ship without initially supporting text rendering in 2D contexts. Open Issue: Should text support be formally excluded from the specification until implementors are prepared to ship it (or until a more feasible API is designed)?

==== Implementation ====

This proposal has been vetted by developers of Apple's Safari, Google's Chrome, Microsoft's Internet Explorer, and Mozilla's Firefox browsers. All vendors agreed upon the basic form of the API, so it is likely it will be implemented widely and compatibly.

==== Adoption ====

Web page authors have demanded increased parallelism support from the web platform for multiple years. If support for multithreaded rendering is added, it is likely it will be rapidly adopted.

==== Example code ====

Jeff Gilbert from Mozilla has crafted some example code utilizing this API:

* [https://github.com/jdashg/snippets/tree/master/webgl-from-worker Rendering WebGL from a worker using the commit() API]
* [https://github.com/jdashg/snippets/blob/master/webgl-one-to-many/index.html Using one WebGL context to render to many Canvas elements]

[[Category:Proposals]]

OffscreenCanvas

2016-10-14T15:12:54Z

Junov: Added link to pull request

'''The specification is currently under review in this [https://github.com/whatwg/html/pull/1876 GitHub pull request].'''
:''Provides more control over how canvases are rendered. This is a follow-on to the [[WorkerCanvas]] proposal and will be merged once agreement is reached.''

== Use Case Description ==

Feedback from web application authors using canvases have shown the need for the following controls:

* (From ShaderToy, Sketchfab, Verold): need to be able to render to multiple regions on the page efficiently using a single canvas context. 3D model warehouse sites desire to show multiple live interactive models on the page, but creating multiple WebGL contexts per page is too inefficient. A single context should be able to render to multiple regions on the page.
* (From Google Maps): need to be able to render WebGL from a worker, transfer the rendered image to the main thread without making any copy of it, and composite it with other HTML on the page, guaranteeing that the updates are all seen in the same rendered frame.
* (From Mozilla and partners using Emscripten and asm.js): need to be able to render WebGL entirely asynchronously from a worker, displaying the results in a canvas owned by the main thread, without any synchronization with the main thread. In this mode, the entire application runs in the worker. The main thread only receives input events and sends them to the worker for processing.
* (From adopters of the Push API): need to be able to dynamically create images to use as notification icons, such as compositing avatars, or adding an unread count
* (From the Google Docs team): need to be able to layout and render text from a worker using CanvasRenderingContext2D and display those results on the main thread.
* (From the Google Slides team): want to layout and render the slide thumbnails from a worker. During initial load and heavy collaboration these update frequently, and currently cause slowdowns on the main thread.

=== Current Limitations ===

* [https://html.spec.whatwg.org/multipage/scripting.html#proxying-canvases-to-workers CanvasProxy] does not provide sufficient control to allow synchronization between workers' rendering and DOM updates on the main thread. Keeping this rendering in sync is a requirement from Google's Maps team.
* [[CanvasInWorkers]] does not allow a worker to render directly into a canvas on the main thread without running code on the main thread. Allowing completely unsynchronized rendering is a requirement from Mozilla and users of Emscripten such as Epic Games and Unity, in which the desire is to execute all of the game's rendering on a worker thread.
* [[WorkerCanvas]] mostly addresses these two use cases, but some implementers objected to the mechanism for displaying the rendering results in image elements. The specific objection was that image elements already have complex internal state (for example, the management of the image's "loaded" state), and this would make it more complex. It also did not precisely address the use case of producing new frames both on the main thread and in workers.

=== Current Usage and Workarounds ===

[https://blog.mozilla.org/research/2014/07/22/webgl-in-web-workers-today-and-faster-than-expected/ WebGL in Web Workers] details some work attempted in the Emscripten toolchain to address the lack of WebGL in workers. Due to the high volume of calls and large amount of data that is transferred to the graphics card in a typical high-end WebGL application, this approach is not sustainable. It's necessary for workers to be able to call the WebGL API directly, and present those results to the screen in a manner that does not introduce any copies of the rendering results.

=== Benefits ===

Making canvas rendering contexts available to workers will increase parallelism in web applications, leading to increased performance on multi-core systems.

=== Requests for this Feature ===

See the abovementioned use cases:

* Google's Maps team
* Emscripten users such as Epic Games and Unity
* Many others

== Web IDL ==

typedef (OffscreenCanvasRenderingContext2D or
WebGLRenderingContext or
WebGL2RenderingContext) OffscreenRenderingContext;

[Constructor(unsigned long width, unsigned long height),
Exposed=(Window,Worker)]
interface OffscreenCanvas {
attribute unsigned long width;
attribute unsigned long height;
OffscreenRenderingContext? getContext(DOMString contextId, any... arguments);

// OffscreenCanvas, like HTMLCanvasElement, maintains an origin-clean flag.
// ImageBitmaps created by calling this method also have an
// origin-clean flag which is set to the value of the OffscreenCanvas's
// flag at the time of their construction. Uses of the ImageBitmap
// in other APIs, such as CanvasRenderingContext2D or
// WebGLRenderingContext, propagate this flag like other
// CanvasImageSource types do, such as HTMLImageElement.
ImageBitmap transferToImageBitmap();

// Throws a SecurityError if the OffscreenCanvas's origin-clean flag
// is set to false.
Promise<Blob> convertToBlob(optional ImageEncodeOptions options);
};

dictionary ImageEncodeOptions {
DOMString type = "image/png";
unrestricted double quality = 1.0; // Defaults to 1.0 if value is outside 0:1 range
};

OffscreenCanvas implements Transferable;

partial interface HTMLCanvasElement {
OffscreenCanvas transferControlToOffscreen();
};

typedef (HTMLOrSVGImageElement or
HTMLVideoElement or
HTMLCanvasElement or
ImageBitmap or
OffscreenCanvas) CanvasImageSource;

[Exposed=Window, Worker]
interface OffscreenCanvasRenderingContext2D {
// commit() can only be used when HTMLCanvasElement has transferred Control
// to OffscreenCanvas. Otherwise, an InvalidStateError will be thrown.
// commit() can be invoked on main thread or worker thread. When it is invoked,
// it is expected to see the image drawn to OffscreenCanvasRenderingContext2D
// be displayed in the associated HTMLCanvasElement.
void commit();
// back-reference to the canvas
readonly attribute OffscreenCanvas canvas;
};
OffscreenCanvasRenderingContext2D implements CanvasState;
OffscreenCanvasRenderingContext2D implements CanvasTransform;
OffscreenCanvasRenderingContext2D implements CanvasCompositing;
OffscreenCanvasRenderingContext2D implements CanvasImageSmoothing;
OffscreenCanvasRenderingContext2D implements CanvasFillStrokeStyles;
OffscreenCanvasRenderingContext2D implements CanvasShadowStyles;
// Reference filters (e.g. 'url()') are not expected to work in Workers
OffscreenCanvasRenderingContext2D implements CanvasFilters;
OffscreenCanvasRenderingContext2D implements CanvasRect;
OffscreenCanvasRenderingContext2D implements CanvasDrawPath;
// Text support in workers poses very difficult technical challenges.
// Open issue: should we forgo text support in OffscreenCanvas v1?
OffscreenCanvasRenderingContext2D implements CanvasText;
OffscreenCanvasRenderingContext2D implements CanvasDrawImage;
OffscreenCanvasRenderingContext2D implements CanvasImageData;
OffscreenCanvasRenderingContext2D implements CanvasPathDrawingStyles;
OffscreenCanvasRenderingContext2D implements CanvasTextDrawingStyles;
OffscreenCanvasRenderingContext2D implements CanvasPath;

[Exposed=Window, Worker]
Partial interface CanvasPattern {
}

[Exposed=Window, Worker]
partial interface CanvasGradient {
}

partial interface WebGLRenderingContextBase {
// back-reference to the canvas
readonly attribute (HTMLCanvasElement or OffscreenCanvas) canvas;

// If this context is associated with an OffscreenCanvas that was
// created by HTMLCanvasElement's transferControlToOffscreen method,
// causes this context's current rendering results to be pushed
// to that canvas element. This has the same effect as returning
// control to the main loop in a single-threaded application. Otherwise,
// an InvalidStateError will be thrown.
void commit();
};

== Proposed Solutions ==

=== This Solution ===

This proposed API can be used in several ways to satisfy the use cases described above:

* It supports zero-copy transfer of canvases' rendering results between threads, for example from a worker to the main thread. In this model, the main thread controls when to display new frames produced by the worker, so synchronization with other DOM updates is achieved.

* It supports fully asynchronous rendering by a worker into a canvas displayed on the main thread. This satisfies certain Emscripten developers' full-screen use cases.

* It supports using a single WebGLRenderingContext or Canvas2DRenderingContext to efficiently render into multiple regions on the web page.

* It introduces ImageBitmapRenderingContext, a new canvas context type whose sole purpose is to efficiently display ImageBitmaps. This supersedes the [[WorkerCanvas]] proposal's use of HTMLImageElement for this purpose.

* It supports asynchronous encoding of OffscreenCanvases' rendering results into Blobs which can be consumed by various other web platform APIs.

==== Processing Model ====

This proposal introduces two primary processing models. The first involves ''synchronous'' display of new frames produced by the OffscreenCanvas. The application generates new frames using the RenderingContext obtained from the OffscreenCanvas. When the application is finished rendering each new frame, it calls transferToImageBitmap to "tear off" the most recently rendered image from the OffscreenCanvas -- like a Post-It note. The resulting ImageBitmap can then be used in any API receiving that data type; notably, it can be displayed in a second canvas without introducing a copy. An ImageBitmapRenderingContext is obtained from the second canvas by calling <code>getContext('bitmaprenderer')</code>. Each frame is displayed in the second canvas using the <code>transferImageBitmap</code> method on this rendering context. Note that the threads producing and consuming the frames may be the same, or they may be different. Note also that a single OffscreenCanvas may transfer frames into an arbitrary number of other ImageBitmapRenderingContexts.

The second processing model involves ''asynchronous'' display of new frames produced by the OffscreenCanvas. The main thread instantiates an HTMLCanvasElement and calls <code>transferControlToOffscreeen</code>. <code>getContext</code> is used to obtain a rendering context for that OffscreenCanvas, either on the main thread, or on a worker. The application calls <code>commit</code> against that rendering context in order to push frames to the original HTMLCanvasElement. In this rendering model, it is not defined when those frames become visible in the original canvas element. However, if the following situations apply:

* It is a worker thread which is calling commit(), and
* The worker is calling commit() repeatedly against exactly one rendering context

then it is required that the user agent synchronize the calls to commit() to the vsync interval. Calls to commit() conceptually enqueue frames for display, and after an implementation-defined number of frames have been enqueued, further calls to commit() will block until earlier frames have been presented to the screen. (This requirement allows porting of applications which drive their own main loop rather than using an event-driven loop.)

==== Limitations ====

* A known good way to drive an animation loop from a worker is needed. requestAnimationFrame or a similar API needs to be defined on worker threads.
* Some parts of the CanvasRenderingContext2D interface shall not be supported due OffscreenCanvas objects having no relation to the DOM or Frame: HitRegions, scrollPathIntoView, drawFocusIfNeeded.
* Due to technical challenges, some implementors [https://bugzilla.mozilla.org/show_bug.cgi?id=801176#c29 (Google and Mozilla)] have expressed a desire to ship without initially supporting text rendering in 2D contexts. Open Issue: Should text support be formally excluded from the specification until implementors are prepared to ship it (or until a more feasible API is designed)?

==== Implementation ====

This proposal has been vetted by developers of Apple's Safari, Google's Chrome, Microsoft's Internet Explorer, and Mozilla's Firefox browsers. All vendors agreed upon the basic form of the API, so it is likely it will be implemented widely and compatibly.

==== Adoption ====

Web page authors have demanded increased parallelism support from the web platform for multiple years. If support for multithreaded rendering is added, it is likely it will be rapidly adopted.

==== Example code ====

Jeff Gilbert from Mozilla has crafted some example code utilizing this API:

* [https://github.com/jdashg/snippets/tree/master/webgl-from-worker Rendering WebGL from a worker using the commit() API]
* [https://github.com/jdashg/snippets/blob/master/webgl-one-to-many/index.html Using one WebGL context to render to many Canvas elements]

[[Category:Proposals]]

OffscreenCanvas

2016-09-27T16:58:03Z

Junov: Removed definition of ImageBitmapSource because it already includes OffscreenCanvas via CanvasImageSource

:''Provides more control over how canvases are rendered. This is a follow-on to the [[WorkerCanvas]] proposal and will be merged once agreement is reached.''

== Use Case Description ==

Feedback from web application authors using canvases have shown the need for the following controls:

* (From ShaderToy, Sketchfab, Verold): need to be able to render to multiple regions on the page efficiently using a single canvas context. 3D model warehouse sites desire to show multiple live interactive models on the page, but creating multiple WebGL contexts per page is too inefficient. A single context should be able to render to multiple regions on the page.
* (From Google Maps): need to be able to render WebGL from a worker, transfer the rendered image to the main thread without making any copy of it, and composite it with other HTML on the page, guaranteeing that the updates are all seen in the same rendered frame.
* (From Mozilla and partners using Emscripten and asm.js): need to be able to render WebGL entirely asynchronously from a worker, displaying the results in a canvas owned by the main thread, without any synchronization with the main thread. In this mode, the entire application runs in the worker. The main thread only receives input events and sends them to the worker for processing.
* (From adopters of the Push API): need to be able to dynamically create images to use as notification icons, such as compositing avatars, or adding an unread count
* (From the Google Docs team): need to be able to layout and render text from a worker using CanvasRenderingContext2D and display those results on the main thread.
* (From the Google Slides team): want to layout and render the slide thumbnails from a worker. During initial load and heavy collaboration these update frequently, and currently cause slowdowns on the main thread.

=== Current Limitations ===

* [https://html.spec.whatwg.org/multipage/scripting.html#proxying-canvases-to-workers CanvasProxy] does not provide sufficient control to allow synchronization between workers' rendering and DOM updates on the main thread. Keeping this rendering in sync is a requirement from Google's Maps team.
* [[CanvasInWorkers]] does not allow a worker to render directly into a canvas on the main thread without running code on the main thread. Allowing completely unsynchronized rendering is a requirement from Mozilla and users of Emscripten such as Epic Games and Unity, in which the desire is to execute all of the game's rendering on a worker thread.
* [[WorkerCanvas]] mostly addresses these two use cases, but some implementers objected to the mechanism for displaying the rendering results in image elements. The specific objection was that image elements already have complex internal state (for example, the management of the image's "loaded" state), and this would make it more complex. It also did not precisely address the use case of producing new frames both on the main thread and in workers.

=== Current Usage and Workarounds ===

[https://blog.mozilla.org/research/2014/07/22/webgl-in-web-workers-today-and-faster-than-expected/ WebGL in Web Workers] details some work attempted in the Emscripten toolchain to address the lack of WebGL in workers. Due to the high volume of calls and large amount of data that is transferred to the graphics card in a typical high-end WebGL application, this approach is not sustainable. It's necessary for workers to be able to call the WebGL API directly, and present those results to the screen in a manner that does not introduce any copies of the rendering results.

=== Benefits ===

Making canvas rendering contexts available to workers will increase parallelism in web applications, leading to increased performance on multi-core systems.

=== Requests for this Feature ===

See the abovementioned use cases:

* Google's Maps team
* Emscripten users such as Epic Games and Unity
* Many others

== Web IDL ==

typedef (OffscreenCanvasRenderingContext2D or
WebGLRenderingContext or
WebGL2RenderingContext) OffscreenRenderingContext;

[Constructor(unsigned long width, unsigned long height),
Exposed=(Window,Worker)]
interface OffscreenCanvas {
attribute unsigned long width;
attribute unsigned long height;
OffscreenRenderingContext? getContext(DOMString contextId, any... arguments);

// OffscreenCanvas, like HTMLCanvasElement, maintains an origin-clean flag.
// ImageBitmaps created by calling this method also have an
// origin-clean flag which is set to the value of the OffscreenCanvas's
// flag at the time of their construction. Uses of the ImageBitmap
// in other APIs, such as CanvasRenderingContext2D or
// WebGLRenderingContext, propagate this flag like other
// CanvasImageSource types do, such as HTMLImageElement.
ImageBitmap transferToImageBitmap();

// Throws a SecurityError if the OffscreenCanvas's origin-clean flag
// is set to false.
Promise<Blob> convertToBlob(optional ImageEncodeOptions options);
};

dictionary ImageEncodeOptions {
DOMString type = "image/png";
unrestricted double quality = 1.0; // Defaults to 1.0 if value is outside 0:1 range
};

OffscreenCanvas implements Transferable;

partial interface HTMLCanvasElement {
OffscreenCanvas transferControlToOffscreen();
};

typedef (HTMLOrSVGImageElement or
HTMLVideoElement or
HTMLCanvasElement or
ImageBitmap or
OffscreenCanvas) CanvasImageSource;

[Exposed=Window, Worker]
interface OffscreenCanvasRenderingContext2D {
// commit() can only be used when HTMLCanvasElement has transferred Control
// to OffscreenCanvas. Otherwise, an InvalidStateError will be thrown.
// commit() can be invoked on main thread or worker thread. When it is invoked,
// it is expected to see the image drawn to OffscreenCanvasRenderingContext2D
// be displayed in the associated HTMLCanvasElement.
void commit();
// back-reference to the canvas
readonly attribute OffscreenCanvas canvas;
};
OffscreenCanvasRenderingContext2D implements CanvasState;
OffscreenCanvasRenderingContext2D implements CanvasTransform;
OffscreenCanvasRenderingContext2D implements CanvasCompositing;
OffscreenCanvasRenderingContext2D implements CanvasImageSmoothing;
OffscreenCanvasRenderingContext2D implements CanvasFillStrokeStyles;
OffscreenCanvasRenderingContext2D implements CanvasShadowStyles;
// Reference filters (e.g. 'url()') are not expected to work in Workers
OffscreenCanvasRenderingContext2D implements CanvasFilters;
OffscreenCanvasRenderingContext2D implements CanvasRect;
OffscreenCanvasRenderingContext2D implements CanvasDrawPath;
// Text support in workers poses very difficult technical challenges.
// Open issue: should we forgo text support in OffscreenCanvas v1?
OffscreenCanvasRenderingContext2D implements CanvasText;
OffscreenCanvasRenderingContext2D implements CanvasDrawImage;
OffscreenCanvasRenderingContext2D implements CanvasImageData;
OffscreenCanvasRenderingContext2D implements CanvasPathDrawingStyles;
OffscreenCanvasRenderingContext2D implements CanvasTextDrawingStyles;
OffscreenCanvasRenderingContext2D implements CanvasPath;

[Exposed=Window, Worker]
Partial interface CanvasPattern {
}

[Exposed=Window, Worker]
partial interface CanvasGradient {
}

partial interface WebGLRenderingContextBase {
// back-reference to the canvas
readonly attribute (HTMLCanvasElement or OffscreenCanvas) canvas;

// If this context is associated with an OffscreenCanvas that was
// created by HTMLCanvasElement's transferControlToOffscreen method,
// causes this context's current rendering results to be pushed
// to that canvas element. This has the same effect as returning
// control to the main loop in a single-threaded application. Otherwise,
// an InvalidStateError will be thrown.
void commit();
};

== Proposed Solutions ==

=== This Solution ===

This proposed API can be used in several ways to satisfy the use cases described above:

* It supports zero-copy transfer of canvases' rendering results between threads, for example from a worker to the main thread. In this model, the main thread controls when to display new frames produced by the worker, so synchronization with other DOM updates is achieved.

* It supports fully asynchronous rendering by a worker into a canvas displayed on the main thread. This satisfies certain Emscripten developers' full-screen use cases.

* It supports using a single WebGLRenderingContext or Canvas2DRenderingContext to efficiently render into multiple regions on the web page.

* It introduces ImageBitmapRenderingContext, a new canvas context type whose sole purpose is to efficiently display ImageBitmaps. This supersedes the [[WorkerCanvas]] proposal's use of HTMLImageElement for this purpose.

* It supports asynchronous encoding of OffscreenCanvases' rendering results into Blobs which can be consumed by various other web platform APIs.

==== Processing Model ====

This proposal introduces two primary processing models. The first involves ''synchronous'' display of new frames produced by the OffscreenCanvas. The application generates new frames using the RenderingContext obtained from the OffscreenCanvas. When the application is finished rendering each new frame, it calls transferToImageBitmap to "tear off" the most recently rendered image from the OffscreenCanvas -- like a Post-It note. The resulting ImageBitmap can then be used in any API receiving that data type; notably, it can be displayed in a second canvas without introducing a copy. An ImageBitmapRenderingContext is obtained from the second canvas by calling <code>getContext('bitmaprenderer')</code>. Each frame is displayed in the second canvas using the <code>transferImageBitmap</code> method on this rendering context. Note that the threads producing and consuming the frames may be the same, or they may be different. Note also that a single OffscreenCanvas may transfer frames into an arbitrary number of other ImageBitmapRenderingContexts.

The second processing model involves ''asynchronous'' display of new frames produced by the OffscreenCanvas. The main thread instantiates an HTMLCanvasElement and calls <code>transferControlToOffscreeen</code>. <code>getContext</code> is used to obtain a rendering context for that OffscreenCanvas, either on the main thread, or on a worker. The application calls <code>commit</code> against that rendering context in order to push frames to the original HTMLCanvasElement. In this rendering model, it is not defined when those frames become visible in the original canvas element. However, if the following situations apply:

* It is a worker thread which is calling commit(), and
* The worker is calling commit() repeatedly against exactly one rendering context

then it is required that the user agent synchronize the calls to commit() to the vsync interval. Calls to commit() conceptually enqueue frames for display, and after an implementation-defined number of frames have been enqueued, further calls to commit() will block until earlier frames have been presented to the screen. (This requirement allows porting of applications which drive their own main loop rather than using an event-driven loop.)

==== Limitations ====

* A known good way to drive an animation loop from a worker is needed. requestAnimationFrame or a similar API needs to be defined on worker threads.
* Some parts of the CanvasRenderingContext2D interface shall not be supported due OffscreenCanvas objects having no relation to the DOM or Frame: HitRegions, scrollPathIntoView, drawFocusIfNeeded.
* Due to technical challenges, some implementors [https://bugzilla.mozilla.org/show_bug.cgi?id=801176#c29 (Google and Mozilla)] have expressed a desire to ship without initially supporting text rendering in 2D contexts. Open Issue: Should text support be formally excluded from the specification until implementors are prepared to ship it (or until a more feasible API is designed)?

==== Implementation ====

This proposal has been vetted by developers of Apple's Safari, Google's Chrome, Microsoft's Internet Explorer, and Mozilla's Firefox browsers. All vendors agreed upon the basic form of the API, so it is likely it will be implemented widely and compatibly.

==== Adoption ====

Web page authors have demanded increased parallelism support from the web platform for multiple years. If support for multithreaded rendering is added, it is likely it will be rapidly adopted.

==== Example code ====

Jeff Gilbert from Mozilla has crafted some example code utilizing this API:

* [https://github.com/jdashg/snippets/tree/master/webgl-from-worker Rendering WebGL from a worker using the commit() API]
* [https://github.com/jdashg/snippets/blob/master/webgl-one-to-many/index.html Using one WebGL context to render to many Canvas elements]

[[Category:Proposals]]

OffscreenCanvas

2016-08-25T15:44:19Z

Junov: Bikeshed: getAsBlob -> convertToBlob, which sounds more async

:''Provides more control over how canvases are rendered. This is a follow-on to the [[WorkerCanvas]] proposal and will be merged once agreement is reached.''

== Use Case Description ==

Feedback from web application authors using canvases have shown the need for the following controls:

* (From ShaderToy, Sketchfab, Verold): need to be able to render to multiple regions on the page efficiently using a single canvas context. 3D model warehouse sites desire to show multiple live interactive models on the page, but creating multiple WebGL contexts per page is too inefficient. A single context should be able to render to multiple regions on the page.
* (From Google Maps): need to be able to render WebGL from a worker, transfer the rendered image to the main thread without making any copy of it, and composite it with other HTML on the page, guaranteeing that the updates are all seen in the same rendered frame.
* (From Mozilla and partners using Emscripten and asm.js): need to be able to render WebGL entirely asynchronously from a worker, displaying the results in a canvas owned by the main thread, without any synchronization with the main thread. In this mode, the entire application runs in the worker. The main thread only receives input events and sends them to the worker for processing.
* (From adopters of the Push API): need to be able to dynamically create images to use as notification icons, such as compositing avatars, or adding an unread count
* (From the Google Docs team): need to be able to layout and render text from a worker using CanvasRenderingContext2D and display those results on the main thread.
* (From the Google Slides team): want to layout and render the slide thumbnails from a worker. During initial load and heavy collaboration these update frequently, and currently cause slowdowns on the main thread.

=== Current Limitations ===

* [https://html.spec.whatwg.org/multipage/scripting.html#proxying-canvases-to-workers CanvasProxy] does not provide sufficient control to allow synchronization between workers' rendering and DOM updates on the main thread. Keeping this rendering in sync is a requirement from Google's Maps team.
* [[CanvasInWorkers]] does not allow a worker to render directly into a canvas on the main thread without running code on the main thread. Allowing completely unsynchronized rendering is a requirement from Mozilla and users of Emscripten such as Epic Games and Unity, in which the desire is to execute all of the game's rendering on a worker thread.
* [[WorkerCanvas]] mostly addresses these two use cases, but some implementers objected to the mechanism for displaying the rendering results in image elements. The specific objection was that image elements already have complex internal state (for example, the management of the image's "loaded" state), and this would make it more complex. It also did not precisely address the use case of producing new frames both on the main thread and in workers.

=== Current Usage and Workarounds ===

[https://blog.mozilla.org/research/2014/07/22/webgl-in-web-workers-today-and-faster-than-expected/ WebGL in Web Workers] details some work attempted in the Emscripten toolchain to address the lack of WebGL in workers. Due to the high volume of calls and large amount of data that is transferred to the graphics card in a typical high-end WebGL application, this approach is not sustainable. It's necessary for workers to be able to call the WebGL API directly, and present those results to the screen in a manner that does not introduce any copies of the rendering results.

=== Benefits ===

Making canvas rendering contexts available to workers will increase parallelism in web applications, leading to increased performance on multi-core systems.

=== Requests for this Feature ===

See the abovementioned use cases:

* Google's Maps team
* Emscripten users such as Epic Games and Unity
* Many others

== Web IDL ==

typedef (OffscreenCanvasRenderingContext2D or
WebGLRenderingContext or
WebGL2RenderingContext) OffscreenRenderingContext;

[Constructor(unsigned long width, unsigned long height),
Exposed=(Window,Worker)]
interface OffscreenCanvas {
attribute unsigned long width;
attribute unsigned long height;
OffscreenRenderingContext? getContext(DOMString contextId, any... arguments);

// OffscreenCanvas, like HTMLCanvasElement, maintains an origin-clean flag.
// ImageBitmaps created by calling this method also have an
// origin-clean flag which is set to the value of the OffscreenCanvas's
// flag at the time of their construction. Uses of the ImageBitmap
// in other APIs, such as CanvasRenderingContext2D or
// WebGLRenderingContext, propagate this flag like other
// CanvasImageSource types do, such as HTMLImageElement.
ImageBitmap transferToImageBitmap();

// Throws a SecurityError if the OffscreenCanvas's origin-clean flag
// is set to false.
Promise<Blob> convertToBlob(optional ImageEncodeOptions options);
};

dictionary ImageEncodeOptions {
DOMString type = "image/png";
unrestricted double quality = 1.0; // Defaults to 1.0 if value is outside 0:1 range
};

OffscreenCanvas implements Transferable;

partial interface HTMLCanvasElement {
OffscreenCanvas transferControlToOffscreen();
};

typedef (HTMLOrSVGImageElement or
HTMLVideoElement or
HTMLCanvasElement or
ImageBitmap or
OffscreenCanvas) CanvasImageSource;

typedef (CanvasImageSource or
Blob or
ImageData or
OffscreenCanvas) ImageBitmapSource;

[Exposed=Window, Worker]
interface OffscreenCanvasRenderingContext2D {
// commit() can only be used when HTMLCanvasElement has transferred Control
// to OffscreenCanvas. Otherwise, Exception will be raised.
// commit() can be invoked on main thread or worker thread. When it is invoked,
// it is expected to see the image drawn to OffscreenCanvasRenderingContext2D
// be displayed in the associated HTMLCanvasElement.
void commit();
// back-reference to the canvas
readonly attribute OffscreenCanvas canvas;
};
OffscreenCanvasRenderingContext2D implements CanvasState;
OffscreenCanvasRenderingContext2D implements CanvasTransform;
OffscreenCanvasRenderingContext2D implements CanvasCompositing;
OffscreenCanvasRenderingContext2D implements CanvasImageSmoothing;
OffscreenCanvasRenderingContext2D implements CanvasFillStrokeStyles;
OffscreenCanvasRenderingContext2D implements CanvasShadowStyles;
// Reference filters (e.g. 'url()') are not expected to work in Workers
OffscreenCanvasRenderingContext2D implements CanvasFilters;
OffscreenCanvasRenderingContext2D implements CanvasRect;
OffscreenCanvasRenderingContext2D implements CanvasDrawPath;
// Text support in workers poses very difficult technical challenges.
// Open issue: should we forgo text support in OffscreenCanvas v1?
OffscreenCanvasRenderingContext2D implements CanvasText;
OffscreenCanvasRenderingContext2D implements CanvasDrawImage;
OffscreenCanvasRenderingContext2D implements CanvasImageData;
OffscreenCanvasRenderingContext2D implements CanvasPathDrawingStyles;
OffscreenCanvasRenderingContext2D implements CanvasTextDrawingStyles;
OffscreenCanvasRenderingContext2D implements CanvasPath;

[Exposed=Window, Worker]
Partial interface CanvasPattern {
}

[Exposed=Window, Worker]
partial interface CanvasGradient {
}

partial interface WebGLRenderingContextBase {
// back-reference to the canvas
readonly attribute (HTMLCanvasElement or OffscreenCanvas) canvas;

// If this context is associated with an OffscreenCanvas that was
// created by HTMLCanvasElement's transferControlToOffscreen method,
// causes this context's current rendering results to be pushed
// to that canvas element. This has the same effect as returning
// control to the main loop in a single-threaded application. Otherwise,
// this call has no effect.
void commit();
};

== Proposed Solutions ==

=== This Solution ===

This proposed API can be used in several ways to satisfy the use cases described above:

* It supports zero-copy transfer of canvases' rendering results between threads, for example from a worker to the main thread. In this model, the main thread controls when to display new frames produced by the worker, so synchronization with other DOM updates is achieved.

* It supports fully asynchronous rendering by a worker into a canvas displayed on the main thread. This satisfies certain Emscripten developers' full-screen use cases.

* It supports using a single WebGLRenderingContext or Canvas2DRenderingContext to efficiently render into multiple regions on the web page.

* It introduces ImageBitmapRenderingContext, a new canvas context type whose sole purpose is to efficiently display ImageBitmaps. This supersedes the [[WorkerCanvas]] proposal's use of HTMLImageElement for this purpose.

* It supports asynchronous encoding of OffscreenCanvases' rendering results into Blobs which can be consumed by various other web platform APIs.

==== Processing Model ====

This proposal introduces two primary processing models. The first involves ''synchronous'' display of new frames produced by the OffscreenCanvas. The application generates new frames using the RenderingContext obtained from the OffscreenCanvas. When the application is finished rendering each new frame, it calls transferToImageBitmap to "tear off" the most recently rendered image from the OffscreenCanvas -- like a Post-It note. The resulting ImageBitmap can then be used in any API receiving that data type; notably, it can be displayed in a second canvas without introducing a copy. An ImageBitmapRenderingContext is obtained from the second canvas by calling <code>getContext('bitmaprenderer')</code>. Each frame is displayed in the second canvas using the <code>transferImageBitmap</code> method on this rendering context. Note that the threads producing and consuming the frames may be the same, or they may be different. Note also that a single OffscreenCanvas may transfer frames into an arbitrary number of other ImageBitmapRenderingContexts.

The second processing model involves ''asynchronous'' display of new frames produced by the OffscreenCanvas. The main thread instantiates an HTMLCanvasElement and calls <code>transferControlToOffscreeen</code>. <code>getContext</code> is used to obtain a rendering context for that OffscreenCanvas, either on the main thread, or on a worker. The application calls <code>commit</code> against that rendering context in order to push frames to the original HTMLCanvasElement. In this rendering model, it is not defined when those frames become visible in the original canvas element. However, if the following situations apply:

* It is a worker thread which is calling commit(), and
* The worker is calling commit() repeatedly against exactly one rendering context

then it is required that the user agent synchronize the calls to commit() to the vsync interval. Calls to commit() conceptually enqueue frames for display, and after an implementation-defined number of frames have been enqueued, further calls to commit() will block until earlier frames have been presented to the screen. (This requirement allows porting of applications which drive their own main loop rather than using an event-driven loop.)

==== Limitations ====

* A known good way to drive an animation loop from a worker is needed. requestAnimationFrame or a similar API needs to be defined on worker threads.
* Some parts of the CanvasRenderingContext2D interface shall not be supported due OffscreenCanvas objects having no relation to the DOM or Frame: HitRegions, scrollPathIntoView, drawFocusIfNeeded.
* Due to technical challenges, some implementors [https://bugzilla.mozilla.org/show_bug.cgi?id=801176#c29 (Google and Mozilla)] have expressed a desire to ship without initially supporting text rendering in 2D contexts. Open Issue: Should text support be formally excluded from the specification until implementors are prepared to ship it (or until a more feasible API is designed)?

==== Implementation ====

This proposal has been vetted by developers of Apple's Safari, Google's Chrome, Microsoft's Internet Explorer, and Mozilla's Firefox browsers. All vendors agreed upon the basic form of the API, so it is likely it will be implemented widely and compatibly.

==== Adoption ====

Web page authors have demanded increased parallelism support from the web platform for multiple years. If support for multithreaded rendering is added, it is likely it will be rapidly adopted.

==== Example code ====

Jeff Gilbert from Mozilla has crafted some example code utilizing this API:

* [https://github.com/jdashg/snippets/tree/master/webgl-from-worker Rendering WebGL from a worker using the commit() API]
* [https://github.com/jdashg/snippets/blob/master/webgl-one-to-many/index.html Using one WebGL context to render to many Canvas elements]

[[Category:Proposals]]

OffscreenCanvas

2016-08-25T15:38:51Z

Junov: /* Web IDL */

:''Provides more control over how canvases are rendered. This is a follow-on to the [[WorkerCanvas]] proposal and will be merged once agreement is reached.''

== Use Case Description ==

Feedback from web application authors using canvases have shown the need for the following controls:

* (From ShaderToy, Sketchfab, Verold): need to be able to render to multiple regions on the page efficiently using a single canvas context. 3D model warehouse sites desire to show multiple live interactive models on the page, but creating multiple WebGL contexts per page is too inefficient. A single context should be able to render to multiple regions on the page.
* (From Google Maps): need to be able to render WebGL from a worker, transfer the rendered image to the main thread without making any copy of it, and composite it with other HTML on the page, guaranteeing that the updates are all seen in the same rendered frame.
* (From Mozilla and partners using Emscripten and asm.js): need to be able to render WebGL entirely asynchronously from a worker, displaying the results in a canvas owned by the main thread, without any synchronization with the main thread. In this mode, the entire application runs in the worker. The main thread only receives input events and sends them to the worker for processing.
* (From adopters of the Push API): need to be able to dynamically create images to use as notification icons, such as compositing avatars, or adding an unread count
* (From the Google Docs team): need to be able to layout and render text from a worker using CanvasRenderingContext2D and display those results on the main thread.
* (From the Google Slides team): want to layout and render the slide thumbnails from a worker. During initial load and heavy collaboration these update frequently, and currently cause slowdowns on the main thread.

=== Current Limitations ===

* [https://html.spec.whatwg.org/multipage/scripting.html#proxying-canvases-to-workers CanvasProxy] does not provide sufficient control to allow synchronization between workers' rendering and DOM updates on the main thread. Keeping this rendering in sync is a requirement from Google's Maps team.
* [[CanvasInWorkers]] does not allow a worker to render directly into a canvas on the main thread without running code on the main thread. Allowing completely unsynchronized rendering is a requirement from Mozilla and users of Emscripten such as Epic Games and Unity, in which the desire is to execute all of the game's rendering on a worker thread.
* [[WorkerCanvas]] mostly addresses these two use cases, but some implementers objected to the mechanism for displaying the rendering results in image elements. The specific objection was that image elements already have complex internal state (for example, the management of the image's "loaded" state), and this would make it more complex. It also did not precisely address the use case of producing new frames both on the main thread and in workers.

=== Current Usage and Workarounds ===

[https://blog.mozilla.org/research/2014/07/22/webgl-in-web-workers-today-and-faster-than-expected/ WebGL in Web Workers] details some work attempted in the Emscripten toolchain to address the lack of WebGL in workers. Due to the high volume of calls and large amount of data that is transferred to the graphics card in a typical high-end WebGL application, this approach is not sustainable. It's necessary for workers to be able to call the WebGL API directly, and present those results to the screen in a manner that does not introduce any copies of the rendering results.

=== Benefits ===

Making canvas rendering contexts available to workers will increase parallelism in web applications, leading to increased performance on multi-core systems.

=== Requests for this Feature ===

See the abovementioned use cases:

* Google's Maps team
* Emscripten users such as Epic Games and Unity
* Many others

== Web IDL ==

typedef (OffscreenCanvasRenderingContext2D or
WebGLRenderingContext or
WebGL2RenderingContext) OffscreenRenderingContext;

[Constructor(unsigned long width, unsigned long height),
Exposed=(Window,Worker)]
interface OffscreenCanvas {
attribute unsigned long width;
attribute unsigned long height;
OffscreenRenderingContext? getContext(DOMString contextId, any... arguments);

// OffscreenCanvas, like HTMLCanvasElement, maintains an origin-clean flag.
// ImageBitmaps created by calling this method also have an
// origin-clean flag which is set to the value of the OffscreenCanvas's
// flag at the time of their construction. Uses of the ImageBitmap
// in other APIs, such as CanvasRenderingContext2D or
// WebGLRenderingContext, propagate this flag like other
// CanvasImageSource types do, such as HTMLImageElement.
ImageBitmap transferToImageBitmap();

// Throws a SecurityError if the OffscreenCanvas's origin-clean flag
// is set to false.
Promise<Blob> getAsBlob(optional ImageEncodeOptions options);
};

dictionary ImageEncodeOptions {
DOMString type = "image/png";
unrestricted double quality = 1.0; // Defaults to 1.0 if value is outside 0:1 range
};

OffscreenCanvas implements Transferable;

partial interface HTMLCanvasElement {
OffscreenCanvas transferControlToOffscreen();
};

typedef (HTMLOrSVGImageElement or
HTMLVideoElement or
HTMLCanvasElement or
ImageBitmap or
OffscreenCanvas) CanvasImageSource;

typedef (CanvasImageSource or
Blob or
ImageData or
OffscreenCanvas) ImageBitmapSource;

[Exposed=Window, Worker]
interface OffscreenCanvasRenderingContext2D {
// commit() can only be used when HTMLCanvasElement has transferred Control
// to OffscreenCanvas. Otherwise, Exception will be raised.
// commit() can be invoked on main thread or worker thread. When it is invoked,
// it is expected to see the image drawn to OffscreenCanvasRenderingContext2D
// be displayed in the associated HTMLCanvasElement.
void commit();
// back-reference to the canvas
readonly attribute OffscreenCanvas canvas;
};
OffscreenCanvasRenderingContext2D implements CanvasState;
OffscreenCanvasRenderingContext2D implements CanvasTransform;
OffscreenCanvasRenderingContext2D implements CanvasCompositing;
OffscreenCanvasRenderingContext2D implements CanvasImageSmoothing;
OffscreenCanvasRenderingContext2D implements CanvasFillStrokeStyles;
OffscreenCanvasRenderingContext2D implements CanvasShadowStyles;
// Reference filters (e.g. 'url()') are not expected to work in Workers
OffscreenCanvasRenderingContext2D implements CanvasFilters;
OffscreenCanvasRenderingContext2D implements CanvasRect;
OffscreenCanvasRenderingContext2D implements CanvasDrawPath;
// Text support in workers poses very difficult technical challenges.
// Open issue: should we forgo text support in OffscreenCanvas v1?
OffscreenCanvasRenderingContext2D implements CanvasText;
OffscreenCanvasRenderingContext2D implements CanvasDrawImage;
OffscreenCanvasRenderingContext2D implements CanvasImageData;
OffscreenCanvasRenderingContext2D implements CanvasPathDrawingStyles;
OffscreenCanvasRenderingContext2D implements CanvasTextDrawingStyles;
OffscreenCanvasRenderingContext2D implements CanvasPath;

[Exposed=Window, Worker]
Partial interface CanvasPattern {
}

[Exposed=Window, Worker]
partial interface CanvasGradient {
}

partial interface WebGLRenderingContextBase {
// back-reference to the canvas
readonly attribute (HTMLCanvasElement or OffscreenCanvas) canvas;

// If this context is associated with an OffscreenCanvas that was
// created by HTMLCanvasElement's transferControlToOffscreen method,
// causes this context's current rendering results to be pushed
// to that canvas element. This has the same effect as returning
// control to the main loop in a single-threaded application. Otherwise,
// this call has no effect.
void commit();
};

== Proposed Solutions ==

=== This Solution ===

This proposed API can be used in several ways to satisfy the use cases described above:

* It supports zero-copy transfer of canvases' rendering results between threads, for example from a worker to the main thread. In this model, the main thread controls when to display new frames produced by the worker, so synchronization with other DOM updates is achieved.

* It supports fully asynchronous rendering by a worker into a canvas displayed on the main thread. This satisfies certain Emscripten developers' full-screen use cases.

* It supports using a single WebGLRenderingContext or Canvas2DRenderingContext to efficiently render into multiple regions on the web page.

* It introduces ImageBitmapRenderingContext, a new canvas context type whose sole purpose is to efficiently display ImageBitmaps. This supersedes the [[WorkerCanvas]] proposal's use of HTMLImageElement for this purpose.

* It supports asynchronous encoding of OffscreenCanvases' rendering results into Blobs which can be consumed by various other web platform APIs.

==== Processing Model ====

This proposal introduces two primary processing models. The first involves ''synchronous'' display of new frames produced by the OffscreenCanvas. The application generates new frames using the RenderingContext obtained from the OffscreenCanvas. When the application is finished rendering each new frame, it calls transferToImageBitmap to "tear off" the most recently rendered image from the OffscreenCanvas -- like a Post-It note. The resulting ImageBitmap can then be used in any API receiving that data type; notably, it can be displayed in a second canvas without introducing a copy. An ImageBitmapRenderingContext is obtained from the second canvas by calling <code>getContext('bitmaprenderer')</code>. Each frame is displayed in the second canvas using the <code>transferImageBitmap</code> method on this rendering context. Note that the threads producing and consuming the frames may be the same, or they may be different. Note also that a single OffscreenCanvas may transfer frames into an arbitrary number of other ImageBitmapRenderingContexts.

The second processing model involves ''asynchronous'' display of new frames produced by the OffscreenCanvas. The main thread instantiates an HTMLCanvasElement and calls <code>transferControlToOffscreeen</code>. <code>getContext</code> is used to obtain a rendering context for that OffscreenCanvas, either on the main thread, or on a worker. The application calls <code>commit</code> against that rendering context in order to push frames to the original HTMLCanvasElement. In this rendering model, it is not defined when those frames become visible in the original canvas element. However, if the following situations apply:

* It is a worker thread which is calling commit(), and
* The worker is calling commit() repeatedly against exactly one rendering context

then it is required that the user agent synchronize the calls to commit() to the vsync interval. Calls to commit() conceptually enqueue frames for display, and after an implementation-defined number of frames have been enqueued, further calls to commit() will block until earlier frames have been presented to the screen. (This requirement allows porting of applications which drive their own main loop rather than using an event-driven loop.)

==== Limitations ====

* A known good way to drive an animation loop from a worker is needed. requestAnimationFrame or a similar API needs to be defined on worker threads.
* Some parts of the CanvasRenderingContext2D interface shall not be supported due OffscreenCanvas objects having no relation to the DOM or Frame: HitRegions, scrollPathIntoView, drawFocusIfNeeded.
* Due to technical challenges, some implementors [https://bugzilla.mozilla.org/show_bug.cgi?id=801176#c29 (Google and Mozilla)] have expressed a desire to ship without initially supporting text rendering in 2D contexts. Open Issue: Should text support be formally excluded from the specification until implementors are prepared to ship it (or until a more feasible API is designed)?

==== Implementation ====

This proposal has been vetted by developers of Apple's Safari, Google's Chrome, Microsoft's Internet Explorer, and Mozilla's Firefox browsers. All vendors agreed upon the basic form of the API, so it is likely it will be implemented widely and compatibly.

==== Adoption ====

Web page authors have demanded increased parallelism support from the web platform for multiple years. If support for multithreaded rendering is added, it is likely it will be rapidly adopted.

==== Example code ====

Jeff Gilbert from Mozilla has crafted some example code utilizing this API:

* [https://github.com/jdashg/snippets/tree/master/webgl-from-worker Rendering WebGL from a worker using the commit() API]
* [https://github.com/jdashg/snippets/blob/master/webgl-one-to-many/index.html Using one WebGL context to render to many Canvas elements]

[[Category:Proposals]]

OffscreenCanvas

2016-08-25T15:38:16Z

Junov: renamed toBlob -> getAsBlob. Make it use a dictionary so we can easily add more options in the future, such as colorApace, bitDepth, alpha, ...

:''Provides more control over how canvases are rendered. This is a follow-on to the [[WorkerCanvas]] proposal and will be merged once agreement is reached.''

== Use Case Description ==

Feedback from web application authors using canvases have shown the need for the following controls:

* (From ShaderToy, Sketchfab, Verold): need to be able to render to multiple regions on the page efficiently using a single canvas context. 3D model warehouse sites desire to show multiple live interactive models on the page, but creating multiple WebGL contexts per page is too inefficient. A single context should be able to render to multiple regions on the page.
* (From Google Maps): need to be able to render WebGL from a worker, transfer the rendered image to the main thread without making any copy of it, and composite it with other HTML on the page, guaranteeing that the updates are all seen in the same rendered frame.
* (From Mozilla and partners using Emscripten and asm.js): need to be able to render WebGL entirely asynchronously from a worker, displaying the results in a canvas owned by the main thread, without any synchronization with the main thread. In this mode, the entire application runs in the worker. The main thread only receives input events and sends them to the worker for processing.
* (From adopters of the Push API): need to be able to dynamically create images to use as notification icons, such as compositing avatars, or adding an unread count
* (From the Google Docs team): need to be able to layout and render text from a worker using CanvasRenderingContext2D and display those results on the main thread.
* (From the Google Slides team): want to layout and render the slide thumbnails from a worker. During initial load and heavy collaboration these update frequently, and currently cause slowdowns on the main thread.

=== Current Limitations ===

* [https://html.spec.whatwg.org/multipage/scripting.html#proxying-canvases-to-workers CanvasProxy] does not provide sufficient control to allow synchronization between workers' rendering and DOM updates on the main thread. Keeping this rendering in sync is a requirement from Google's Maps team.
* [[CanvasInWorkers]] does not allow a worker to render directly into a canvas on the main thread without running code on the main thread. Allowing completely unsynchronized rendering is a requirement from Mozilla and users of Emscripten such as Epic Games and Unity, in which the desire is to execute all of the game's rendering on a worker thread.
* [[WorkerCanvas]] mostly addresses these two use cases, but some implementers objected to the mechanism for displaying the rendering results in image elements. The specific objection was that image elements already have complex internal state (for example, the management of the image's "loaded" state), and this would make it more complex. It also did not precisely address the use case of producing new frames both on the main thread and in workers.

=== Current Usage and Workarounds ===

[https://blog.mozilla.org/research/2014/07/22/webgl-in-web-workers-today-and-faster-than-expected/ WebGL in Web Workers] details some work attempted in the Emscripten toolchain to address the lack of WebGL in workers. Due to the high volume of calls and large amount of data that is transferred to the graphics card in a typical high-end WebGL application, this approach is not sustainable. It's necessary for workers to be able to call the WebGL API directly, and present those results to the screen in a manner that does not introduce any copies of the rendering results.

=== Benefits ===

Making canvas rendering contexts available to workers will increase parallelism in web applications, leading to increased performance on multi-core systems.

=== Requests for this Feature ===

See the abovementioned use cases:

* Google's Maps team
* Emscripten users such as Epic Games and Unity
* Many others

== Web IDL ==

typedef (OffscreenCanvasRenderingContext2D or
WebGLRenderingContext or
WebGL2RenderingContext) OffscreenRenderingContext;

[Constructor(unsigned long width, unsigned long height),
Exposed=(Window,Worker)]
interface OffscreenCanvas {
attribute unsigned long width;
attribute unsigned long height;
OffscreenRenderingContext? getContext(DOMString contextId, any... arguments);

// OffscreenCanvas, like HTMLCanvasElement, maintains an origin-clean flag.
// ImageBitmaps created by calling this method also have an
// origin-clean flag which is set to the value of the OffscreenCanvas's
// flag at the time of their construction. Uses of the ImageBitmap
// in other APIs, such as CanvasRenderingContext2D or
// WebGLRenderingContext, propagate this flag like other
// CanvasImageSource types do, such as HTMLImageElement.
ImageBitmap transferToImageBitmap();

// Throws a SecurityError if the OffscreenCanvas's origin-clean flag
// is set to false.
Promise<Blob> getAsBlob(optional ImageEncodeOptions options);
};

dictionary ImageEncodeOptions {
DOMString type = "image/png";
unrestricted double quality = 1.0; // Defaults to 1.0 if value is outside 0:1 range
};

OffscreenCanvas implements Transferable;

partial interface HTMLCanvasElement {
OffscreenCanvas transferControlToOffscreen();
};

typedef (HTMLOrSVGImageElement or
HTMLVideoElement or
HTMLCanvasElement or
ImageBitmap or
OffscreenCanvas) CanvasImageSource;

typedef (CanvasImageSource or
Blob or
ImageData or
OffscreenCanvas) ImageBitmapSource;

[Exposed=Window, Worker]
interface OffscreenCanvasRenderingContext2D {
// commit() can only be used when HTMLCanvasElement has transferred Control
// to OffscreenCanvas. Otherwise, Exception will be raised.
// commit() can be invoked on main thread or worker thread. When it is invoked,
// it is expected to see the image drawn to OffscreenCanvasRenderingContext2D
// be displayed in the associated HTMLCanvasElement.
void commit();
// back-reference to the canvas
readonly attribute OffscreenCanvas canvas;
};
OffscreenCanvasRenderingContext2D implements CanvasState;
OffscreenCanvasRenderingContext2D implements CanvasTransform;
OffscreenCanvasRenderingContext2D implements CanvasCompositing;
OffscreenCanvasRenderingContext2D implements CanvasImageSmoothing;
OffscreenCanvasRenderingContext2D implements CanvasFillStrokeStyles;
OffscreenCanvasRenderingContext2D implements CanvasShadowStyles;
// Reference filters (e.g. 'url()') are not expected to work in Workers
OffscreenCanvasRenderingContext2D implements CanvasFilters;
OffscreenCanvasRenderingContext2D implements CanvasRect;
OffscreenCanvasRenderingContext2D implements CanvasDrawPath;
// Text support in workers poses very difficult technical challenges.
// Open issue: should we forgo text support in OffscreenCanvas v1?
OffscreenCanvasRenderingContext2D implements CanvasText;
OffscreenCanvasRenderingContext2D implements CanvasDrawImage;
OffscreenCanvasRenderingContext2D implements CanvasImageData;
OffscreenCanvasRenderingContext2D implements CanvasPathDrawingStyles;
OffscreenCanvasRenderingContext2D implements CanvasTextDrawingStyles;
OffscreenCanvasRenderingContext2D implements CanvasPath;

[Exposed=Window, Worker]
Partial interface CanvasPattern {
}

[Exposed=Window, Worker]
partial interface CanvasGradient {
}

partial interface WebGLRenderingContextBase {
// back-reference to the canvas
readonly attribute (HTMLCanvasElement or OffscreenCanvas) canvas;

// If this context is associated with an OffscreenCanvas that was
// created by HTMLCanvasElement's transferControlToOffscreen method,
// causes this context's current rendering results to be pushed
// to that canvas element. This has the same effect as returning
// control to the main loop in a single-threaded application. Otherwise,
// this call has no effect.
void commit();
};

== Proposed Solutions ==

=== This Solution ===

This proposed API can be used in several ways to satisfy the use cases described above:

* It supports zero-copy transfer of canvases' rendering results between threads, for example from a worker to the main thread. In this model, the main thread controls when to display new frames produced by the worker, so synchronization with other DOM updates is achieved.

* It supports fully asynchronous rendering by a worker into a canvas displayed on the main thread. This satisfies certain Emscripten developers' full-screen use cases.

* It supports using a single WebGLRenderingContext or Canvas2DRenderingContext to efficiently render into multiple regions on the web page.

* It introduces ImageBitmapRenderingContext, a new canvas context type whose sole purpose is to efficiently display ImageBitmaps. This supersedes the [[WorkerCanvas]] proposal's use of HTMLImageElement for this purpose.

* It supports asynchronous encoding of OffscreenCanvases' rendering results into Blobs which can be consumed by various other web platform APIs.

==== Processing Model ====

This proposal introduces two primary processing models. The first involves ''synchronous'' display of new frames produced by the OffscreenCanvas. The application generates new frames using the RenderingContext obtained from the OffscreenCanvas. When the application is finished rendering each new frame, it calls transferToImageBitmap to "tear off" the most recently rendered image from the OffscreenCanvas -- like a Post-It note. The resulting ImageBitmap can then be used in any API receiving that data type; notably, it can be displayed in a second canvas without introducing a copy. An ImageBitmapRenderingContext is obtained from the second canvas by calling <code>getContext('bitmaprenderer')</code>. Each frame is displayed in the second canvas using the <code>transferImageBitmap</code> method on this rendering context. Note that the threads producing and consuming the frames may be the same, or they may be different. Note also that a single OffscreenCanvas may transfer frames into an arbitrary number of other ImageBitmapRenderingContexts.

The second processing model involves ''asynchronous'' display of new frames produced by the OffscreenCanvas. The main thread instantiates an HTMLCanvasElement and calls <code>transferControlToOffscreeen</code>. <code>getContext</code> is used to obtain a rendering context for that OffscreenCanvas, either on the main thread, or on a worker. The application calls <code>commit</code> against that rendering context in order to push frames to the original HTMLCanvasElement. In this rendering model, it is not defined when those frames become visible in the original canvas element. However, if the following situations apply:

* It is a worker thread which is calling commit(), and
* The worker is calling commit() repeatedly against exactly one rendering context

then it is required that the user agent synchronize the calls to commit() to the vsync interval. Calls to commit() conceptually enqueue frames for display, and after an implementation-defined number of frames have been enqueued, further calls to commit() will block until earlier frames have been presented to the screen. (This requirement allows porting of applications which drive their own main loop rather than using an event-driven loop.)

==== Limitations ====

* A known good way to drive an animation loop from a worker is needed. requestAnimationFrame or a similar API needs to be defined on worker threads.
* Some parts of the CanvasRenderingContext2D interface shall not be supported due OffscreenCanvas objects having no relation to the DOM or Frame: HitRegions, scrollPathIntoView, drawFocusIfNeeded.
* Due to technical challenges, some implementors [https://bugzilla.mozilla.org/show_bug.cgi?id=801176#c29 (Google and Mozilla)] have expressed a desire to ship without initially supporting text rendering in 2D contexts. Open Issue: Should text support be formally excluded from the specification until implementors are prepared to ship it (or until a more feasible API is designed)?

==== Implementation ====

This proposal has been vetted by developers of Apple's Safari, Google's Chrome, Microsoft's Internet Explorer, and Mozilla's Firefox browsers. All vendors agreed upon the basic form of the API, so it is likely it will be implemented widely and compatibly.

==== Adoption ====

Web page authors have demanded increased parallelism support from the web platform for multiple years. If support for multithreaded rendering is added, it is likely it will be rapidly adopted.

==== Example code ====

Jeff Gilbert from Mozilla has crafted some example code utilizing this API:

* [https://github.com/jdashg/snippets/tree/master/webgl-from-worker Rendering WebGL from a worker using the commit() API]
* [https://github.com/jdashg/snippets/blob/master/webgl-one-to-many/index.html Using one WebGL context to render to many Canvas elements]

[[Category:Proposals]]

OffscreenCanvas

2016-08-18T21:24:12Z

Junov: Cleaning-out parts of the proposal that have already been added to the spec.

:''Provides more control over how canvases are rendered. This is a follow-on to the [[WorkerCanvas]] proposal and will be merged once agreement is reached.''

== Use Case Description ==

Feedback from web application authors using canvases have shown the need for the following controls:

* (From ShaderToy, Sketchfab, Verold): need to be able to render to multiple regions on the page efficiently using a single canvas context. 3D model warehouse sites desire to show multiple live interactive models on the page, but creating multiple WebGL contexts per page is too inefficient. A single context should be able to render to multiple regions on the page.
* (From Google Maps): need to be able to render WebGL from a worker, transfer the rendered image to the main thread without making any copy of it, and composite it with other HTML on the page, guaranteeing that the updates are all seen in the same rendered frame.
* (From Mozilla and partners using Emscripten and asm.js): need to be able to render WebGL entirely asynchronously from a worker, displaying the results in a canvas owned by the main thread, without any synchronization with the main thread. In this mode, the entire application runs in the worker. The main thread only receives input events and sends them to the worker for processing.
* (From adopters of the Push API): need to be able to dynamically create images to use as notification icons, such as compositing avatars, or adding an unread count
* (From the Google Docs team): need to be able to layout and render text from a worker using CanvasRenderingContext2D and display those results on the main thread.
* (From the Google Slides team): want to layout and render the slide thumbnails from a worker. During initial load and heavy collaboration these update frequently, and currently cause slowdowns on the main thread.

=== Current Limitations ===

* [https://html.spec.whatwg.org/multipage/scripting.html#proxying-canvases-to-workers CanvasProxy] does not provide sufficient control to allow synchronization between workers' rendering and DOM updates on the main thread. Keeping this rendering in sync is a requirement from Google's Maps team.
* [[CanvasInWorkers]] does not allow a worker to render directly into a canvas on the main thread without running code on the main thread. Allowing completely unsynchronized rendering is a requirement from Mozilla and users of Emscripten such as Epic Games and Unity, in which the desire is to execute all of the game's rendering on a worker thread.
* [[WorkerCanvas]] mostly addresses these two use cases, but some implementers objected to the mechanism for displaying the rendering results in image elements. The specific objection was that image elements already have complex internal state (for example, the management of the image's "loaded" state), and this would make it more complex. It also did not precisely address the use case of producing new frames both on the main thread and in workers.

=== Current Usage and Workarounds ===

[https://blog.mozilla.org/research/2014/07/22/webgl-in-web-workers-today-and-faster-than-expected/ WebGL in Web Workers] details some work attempted in the Emscripten toolchain to address the lack of WebGL in workers. Due to the high volume of calls and large amount of data that is transferred to the graphics card in a typical high-end WebGL application, this approach is not sustainable. It's necessary for workers to be able to call the WebGL API directly, and present those results to the screen in a manner that does not introduce any copies of the rendering results.

=== Benefits ===

Making canvas rendering contexts available to workers will increase parallelism in web applications, leading to increased performance on multi-core systems.

=== Requests for this Feature ===

See the abovementioned use cases:

* Google's Maps team
* Emscripten users such as Epic Games and Unity
* Many others

== Web IDL ==

typedef (OffscreenCanvasRenderingContext2D or
WebGLRenderingContext or
WebGL2RenderingContext) OffscreenRenderingContext;

[Constructor(unsigned long width, unsigned long height),
Exposed=(Window,Worker)]
interface OffscreenCanvas {
attribute unsigned long width;
attribute unsigned long height;
OffscreenRenderingContext? getContext(DOMString contextId, any... arguments);

// OffscreenCanvas, like HTMLCanvasElement, maintains an origin-clean flag.
// ImageBitmaps created by calling this method also have an
// origin-clean flag which is set to the value of the OffscreenCanvas's
// flag at the time of their construction. Uses of the ImageBitmap
// in other APIs, such as CanvasRenderingContext2D or
// WebGLRenderingContext, propagate this flag like other
// CanvasImageSource types do, such as HTMLImageElement.
ImageBitmap transferToImageBitmap();

// Throws a SecurityError if the OffscreenCanvas's origin-clean flag
// is set to false.
Promise<Blob> toBlob(optional DOMString type, any... arguments);
};

OffscreenCanvas implements Transferable;

partial interface HTMLCanvasElement {
OffscreenCanvas transferControlToOffscreen();
};

typedef (HTMLOrSVGImageElement or
HTMLVideoElement or
HTMLCanvasElement or
ImageBitmap or
OffscreenCanvas) CanvasImageSource;

typedef (CanvasImageSource or
Blob or
ImageData or
OffscreenCanvas) ImageBitmapSource;

[Exposed=Window, Worker]
interface OffscreenCanvasRenderingContext2D {
// commit() can only be used when HTMLCanvasElement has transferred Control
// to OffscreenCanvas. Otherwise, Exception will be raised.
// commit() can be invoked on main thread or worker thread. When it is invoked,
// it is expected to see the image drawn to OffscreenCanvasRenderingContext2D
// be displayed in the associated HTMLCanvasElement.
void commit();
// back-reference to the canvas
readonly attribute OffscreenCanvas canvas;
};
OffscreenCanvasRenderingContext2D implements CanvasState;
OffscreenCanvasRenderingContext2D implements CanvasTransform;
OffscreenCanvasRenderingContext2D implements CanvasCompositing;
OffscreenCanvasRenderingContext2D implements CanvasImageSmoothing;
OffscreenCanvasRenderingContext2D implements CanvasFillStrokeStyles;
OffscreenCanvasRenderingContext2D implements CanvasShadowStyles;
// Reference filters (e.g. 'url()') are not expected to work in Workers
OffscreenCanvasRenderingContext2D implements CanvasFilters;
OffscreenCanvasRenderingContext2D implements CanvasRect;
OffscreenCanvasRenderingContext2D implements CanvasDrawPath;
// Text support in workers poses very difficult technical challenges.
// Open issue: should we forgo text support in OffscreenCanvas v1?
OffscreenCanvasRenderingContext2D implements CanvasText;
OffscreenCanvasRenderingContext2D implements CanvasDrawImage;
OffscreenCanvasRenderingContext2D implements CanvasImageData;
OffscreenCanvasRenderingContext2D implements CanvasPathDrawingStyles;
OffscreenCanvasRenderingContext2D implements CanvasTextDrawingStyles;
OffscreenCanvasRenderingContext2D implements CanvasPath;

[Exposed=Window, Worker]
Partial interface CanvasPattern {
}

[Exposed=Window, Worker]
partial interface CanvasGradient {
}

partial interface WebGLRenderingContextBase {
// back-reference to the canvas
readonly attribute (HTMLCanvasElement or OffscreenCanvas) canvas;

// If this context is associated with an OffscreenCanvas that was
// created by HTMLCanvasElement's transferControlToOffscreen method,
// causes this context's current rendering results to be pushed
// to that canvas element. This has the same effect as returning
// control to the main loop in a single-threaded application. Otherwise,
// this call has no effect.
void commit();
};

== Proposed Solutions ==

=== This Solution ===

This proposed API can be used in several ways to satisfy the use cases described above:

* It supports zero-copy transfer of canvases' rendering results between threads, for example from a worker to the main thread. In this model, the main thread controls when to display new frames produced by the worker, so synchronization with other DOM updates is achieved.

* It supports fully asynchronous rendering by a worker into a canvas displayed on the main thread. This satisfies certain Emscripten developers' full-screen use cases.

* It supports using a single WebGLRenderingContext or Canvas2DRenderingContext to efficiently render into multiple regions on the web page.

* It introduces ImageBitmapRenderingContext, a new canvas context type whose sole purpose is to efficiently display ImageBitmaps. This supersedes the [[WorkerCanvas]] proposal's use of HTMLImageElement for this purpose.

* It supports asynchronous encoding of OffscreenCanvases' rendering results into Blobs which can be consumed by various other web platform APIs.

==== Processing Model ====

This proposal introduces two primary processing models. The first involves ''synchronous'' display of new frames produced by the OffscreenCanvas. The application generates new frames using the RenderingContext obtained from the OffscreenCanvas. When the application is finished rendering each new frame, it calls transferToImageBitmap to "tear off" the most recently rendered image from the OffscreenCanvas -- like a Post-It note. The resulting ImageBitmap can then be used in any API receiving that data type; notably, it can be displayed in a second canvas without introducing a copy. An ImageBitmapRenderingContext is obtained from the second canvas by calling <code>getContext('bitmaprenderer')</code>. Each frame is displayed in the second canvas using the <code>transferImageBitmap</code> method on this rendering context. Note that the threads producing and consuming the frames may be the same, or they may be different. Note also that a single OffscreenCanvas may transfer frames into an arbitrary number of other ImageBitmapRenderingContexts.

The second processing model involves ''asynchronous'' display of new frames produced by the OffscreenCanvas. The main thread instantiates an HTMLCanvasElement and calls <code>transferControlToOffscreeen</code>. <code>getContext</code> is used to obtain a rendering context for that OffscreenCanvas, either on the main thread, or on a worker. The application calls <code>commit</code> against that rendering context in order to push frames to the original HTMLCanvasElement. In this rendering model, it is not defined when those frames become visible in the original canvas element. However, if the following situations apply:

* It is a worker thread which is calling commit(), and
* The worker is calling commit() repeatedly against exactly one rendering context

then it is required that the user agent synchronize the calls to commit() to the vsync interval. Calls to commit() conceptually enqueue frames for display, and after an implementation-defined number of frames have been enqueued, further calls to commit() will block until earlier frames have been presented to the screen. (This requirement allows porting of applications which drive their own main loop rather than using an event-driven loop.)

==== Limitations ====

* A known good way to drive an animation loop from a worker is needed. requestAnimationFrame or a similar API needs to be defined on worker threads.
* Some parts of the CanvasRenderingContext2D interface shall not be supported due OffscreenCanvas objects having no relation to the DOM or Frame: HitRegions, scrollPathIntoView, drawFocusIfNeeded.
* Due to technical challenges, some implementors [https://bugzilla.mozilla.org/show_bug.cgi?id=801176#c29 (Google and Mozilla)] have expressed a desire to ship without initially supporting text rendering in 2D contexts. Open Issue: Should text support be formally excluded from the specification until implementors are prepared to ship it (or until a more feasible API is designed)?

==== Implementation ====

This proposal has been vetted by developers of Apple's Safari, Google's Chrome, Microsoft's Internet Explorer, and Mozilla's Firefox browsers. All vendors agreed upon the basic form of the API, so it is likely it will be implemented widely and compatibly.

==== Adoption ====

Web page authors have demanded increased parallelism support from the web platform for multiple years. If support for multithreaded rendering is added, it is likely it will be rapidly adopted.

==== Example code ====

Jeff Gilbert from Mozilla has crafted some example code utilizing this API:

* [https://github.com/jdashg/snippets/tree/master/webgl-from-worker Rendering WebGL from a worker using the commit() API]
* [https://github.com/jdashg/snippets/blob/master/webgl-one-to-many/index.html Using one WebGL context to render to many Canvas elements]

[[Category:Proposals]]

OffscreenCanvas

2016-08-18T20:35:43Z

Junov:

:''Provides more control over how canvases are rendered. This is a follow-on to the [[WorkerCanvas]] proposal.''

== Use Case Description ==

Feedback from web application authors using canvases have shown the need for the following controls:

* (From ShaderToy, Sketchfab, Verold): need to be able to render to multiple regions on the page efficiently using a single canvas context. 3D model warehouse sites desire to show multiple live interactive models on the page, but creating multiple WebGL contexts per page is too inefficient. A single context should be able to render to multiple regions on the page.
* (From Google Maps): need to be able to render WebGL from a worker, transfer the rendered image to the main thread without making any copy of it, and composite it with other HTML on the page, guaranteeing that the updates are all seen in the same rendered frame.
* (From Mozilla and partners using Emscripten and asm.js): need to be able to render WebGL entirely asynchronously from a worker, displaying the results in a canvas owned by the main thread, without any synchronization with the main thread. In this mode, the entire application runs in the worker. The main thread only receives input events and sends them to the worker for processing.
* (From adopters of the Push API): need to be able to dynamically create images to use as notification icons, such as compositing avatars, or adding an unread count
* (From the Google Docs team): need to be able to layout and render text from a worker using CanvasRenderingContext2D and display those results on the main thread.
* (From the Google Slides team): want to layout and render the slide thumbnails from a worker. During initial load and heavy collaboration these update frequently, and currently cause slowdowns on the main thread.

=== Current Limitations ===

* [https://html.spec.whatwg.org/multipage/scripting.html#proxying-canvases-to-workers CanvasProxy] does not provide sufficient control to allow synchronization between workers' rendering and DOM updates on the main thread. Keeping this rendering in sync is a requirement from Google's Maps team.
* [[CanvasInWorkers]] does not allow a worker to render directly into a canvas on the main thread without running code on the main thread. Allowing completely unsynchronized rendering is a requirement from Mozilla and users of Emscripten such as Epic Games and Unity, in which the desire is to execute all of the game's rendering on a worker thread.
* [[WorkerCanvas]] mostly addresses these two use cases, but some implementers objected to the mechanism for displaying the rendering results in image elements. The specific objection was that image elements already have complex internal state (for example, the management of the image's "loaded" state), and this would make it more complex. It also did not precisely address the use case of producing new frames both on the main thread and in workers.

=== Current Usage and Workarounds ===

[https://blog.mozilla.org/research/2014/07/22/webgl-in-web-workers-today-and-faster-than-expected/ WebGL in Web Workers] details some work attempted in the Emscripten toolchain to address the lack of WebGL in workers. Due to the high volume of calls and large amount of data that is transferred to the graphics card in a typical high-end WebGL application, this approach is not sustainable. It's necessary for workers to be able to call the WebGL API directly, and present those results to the screen in a manner that does not introduce any copies of the rendering results.

=== Benefits ===

Making canvas rendering contexts available to workers will increase parallelism in web applications, leading to increased performance on multi-core systems.

=== Requests for this Feature ===

See the abovementioned use cases:

* Google's Maps team
* Emscripten users such as Epic Games and Unity
* Many others

== Web IDL ==

typedef (OffscreenCanvasRenderingContext2D or
WebGLRenderingContext or
WebGL2RenderingContext) OffscreenRenderingContext;

[Constructor(unsigned long width, unsigned long height),
Exposed=(Window,Worker)]
interface OffscreenCanvas {
attribute unsigned long width;
attribute unsigned long height;
OffscreenRenderingContext? getContext(DOMString contextId, any... arguments);

// OffscreenCanvas, like HTMLCanvasElement, maintains an origin-clean flag.
// ImageBitmaps created by calling this method also have an
// origin-clean flag which is set to the value of the OffscreenCanvas's
// flag at the time of their construction. Uses of the ImageBitmap
// in other APIs, such as CanvasRenderingContext2D or
// WebGLRenderingContext, propagate this flag like other
// CanvasImageSource types do, such as HTMLImageElement.
ImageBitmap transferToImageBitmap();

// Throws a SecurityError if the OffscreenCanvas's origin-clean flag
// is set to false.
Promise<Blob> toBlob(optional DOMString type, any... arguments);
};

OffscreenCanvas implements Transferable;
ImageBitmap implements Transferable;

// It's crucial that there be a way to explicitly dispose of ImageBitmaps
// since they refer to potentially large graphics resources. Some uses
// of this API proposal will result in repeated allocations of ImageBitmaps,
// and garbage collection will not reliably reclaim them quickly enough.
// Here we reuse close(), which also exists on another Transferable type,
// MessagePort. Potentially, all Transferable types should inherit from a
// new interface type "Closeable".
partial interface ImageBitmap {
// Dispose of all graphical resources associated with this ImageBitmap.
void close();
};

partial interface HTMLCanvasElement {
OffscreenCanvas transferControlToOffscreen();
};

// Note that CanvasRenderingContext2D already has a commit() method
// from the CanvasProxy spec which this proposal obsoletes.
partial interface CanvasRenderingContext2D {
// back-reference to the canvas
readonly attribute HTMLCanvasElement canvas;
};

typedef (HTMLOrSVGImageElement or
HTMLVideoElement or
HTMLCanvasElement or
ImageBitmap or
OffscreenCanvas) CanvasImageSource;

typedef (CanvasImageSource or
Blob or
ImageData or
OffscreenCanvas) ImageBitmapSource;

[Exposed=Window, Worker]
interface OffscreenCanvasRenderingContext2D {
// commit() can only be used when HTMLCanvasElement has transferred Control
// to OffscreenCanvas. Otherwise, Exception will be raised.
// commit() can be invoked on main thread or worker thread. When it is invoked,
// it is expected to see the image drawn in OffscreenCanvasRenderingContext2D
// being displayed in the associated HTMLCanvasElement.
void commit();
// back-reference to the canvas
readonly attribute OffscreenCanvas canvas;
};
OffscreenCanvasRenderingContext2D implements CanvasState;
OffscreenCanvasRenderingContext2D implements CanvasTransform;
OffscreenCanvasRenderingContext2D implements CanvasCompositing;
OffscreenCanvasRenderingContext2D implements CanvasImageSmoothing;
OffscreenCanvasRenderingContext2D implements CanvasFillStrokeStyles;
OffscreenCanvasRenderingContext2D implements CanvasShadowStyles;
OffscreenCanvasRenderingContext2D implements CanvasFilters;
OffscreenCanvasRenderingContext2D implements CanvasRect;
OffscreenCanvasRenderingContext2D implements CanvasDrawPath;
OffscreenCanvasRenderingContext2D implements CanvasText;
OffscreenCanvasRenderingContext2D implements CanvasDrawImage;
OffscreenCanvasRenderingContext2D implements CanvasImageData;
OffscreenCanvasRenderingContext2D implements CanvasPathDrawingStyles;
OffscreenCanvasRenderingContext2D implements CanvasTextDrawingStyles;
OffscreenCanvasRenderingContext2D implements CanvasPath;

[Exposed=Window, Worker]
Partial interface CanvasPattern {
}

[Exposed=Window, Worker]
partial interface CanvasGradient {
}

partial interface WebGLRenderingContextBase {
// back-reference to the canvas
readonly attribute (HTMLCanvasElement or OffscreenCanvas) canvas;

// If this context is associated with an OffscreenCanvas that was
// created by HTMLCanvasElement's transferControlToOffscreen method,
// causes this context's current rendering results to be pushed
// to that canvas element. This has the same effect as returning
// control to the main loop in a single-threaded application. Otherwise,
// this call has no effect.
void commit();
};

// The new ImageBitmapRenderingContext is a canvas rendering context
// which only provides the functionality to replace the canvas's
// contents with the given ImageBitmap. Its context id (the first argument
// to getContext) is "bitmaprenderer".
//
// Note: this interface has already been incorporated into the current WHATWG
// specification at https://html.spec.whatwg.org/multipage/scripting.html#the-imagebitmap-rendering-context .
interface ImageBitmapRenderingContext {
// Displays the given ImageBitmap in the canvas associated with this
// rendering context. Ownership of the ImageBitmap is transferred to
// the canvas. The caller may not use its reference to the ImageBitmap
// after making this call. (This semantic is crucial to enable prompt
// reclamation of expensive graphics resources, rather than relying on
// garbage collection to do so.)
//
// The ImageBitmap conceptually replaces the canvas's bitmap, but
// it does not change the canvas's intrinsic width or height.
//
// The ImageBitmap, when displayed, is clipped to the rectangle
// defined by the canvas's instrinsic width and height. Pixels that
// would be covered by the canvas's bitmap which are not covered by
// the supplied ImageBitmap are rendered transparent black. Any CSS
// styles affecting the display of the canvas are applied as usual.
void transferFromImageBitmap(ImageBitmap bitmap);
};

== Proposed Solutions ==

=== This Solution ===

This proposed API can be used in several ways to satisfy the use cases described above:

* It supports zero-copy transfer of canvases' rendering results between threads, for example from a worker to the main thread. In this model, the main thread controls when to display new frames produced by the worker, so synchronization with other DOM updates is achieved.

* It supports fully asynchronous rendering by a worker into a canvas displayed on the main thread. This satisfies certain Emscripten developers' full-screen use cases.

* It supports using a single WebGLRenderingContext or Canvas2DRenderingContext to efficiently render into multiple regions on the web page.

* It introduces ImageBitmapRenderingContext, a new canvas context type whose sole purpose is to efficiently display ImageBitmaps. This supersedes the [[WorkerCanvas]] proposal's use of HTMLImageElement for this purpose.

* It supports asynchronous encoding of OffscreenCanvases' rendering results into Blobs which can be consumed by various other web platform APIs.

==== Processing Model ====

This proposal introduces two primary processing models. The first involves ''synchronous'' display of new frames produced by the OffscreenCanvas. The application generates new frames using the RenderingContext obtained from the OffscreenCanvas. When the application is finished rendering each new frame, it calls transferToImageBitmap to "tear off" the most recently rendered image from the OffscreenCanvas -- like a Post-It note. The resulting ImageBitmap can then be used in any API receiving that data type; notably, it can be displayed in a second canvas without introducing a copy. An ImageBitmapRenderingContext is obtained from the second canvas by calling <code>getContext('bitmaprenderer')</code>. Each frame is displayed in the second canvas using the <code>transferImageBitmap</code> method on this rendering context. Note that the threads producing and consuming the frames may be the same, or they may be different. Note also that a single OffscreenCanvas may transfer frames into an arbitrary number of other ImageBitmapRenderingContexts.

The second processing model involves ''asynchronous'' display of new frames produced by the OffscreenCanvas. The main thread instantiates an HTMLCanvasElement and calls <code>transferControlToOffscreeen</code>. <code>getContext</code> is used to obtain a rendering context for that OffscreenCanvas, either on the main thread, or on a worker. The application calls <code>commit</code> against that rendering context in order to push frames to the original HTMLCanvasElement. In this rendering model, it is not defined when those frames become visible in the original canvas element. However, if the following situations apply:

* It is a worker thread which is calling commit(), and
* The worker is calling commit() repeatedly against exactly one rendering context

then it is required that the user agent synchronize the calls to commit() to the vsync interval. Calls to commit() conceptually enqueue frames for display, and after an implementation-defined number of frames have been enqueued, further calls to commit() will block until earlier frames have been presented to the screen. (This requirement allows porting of applications which drive their own main loop rather than using an event-driven loop.)

==== Limitations ====

* A known good way to drive an animation loop from a worker is needed. requestAnimationFrame or a similar API needs to be defined on worker threads.
* Some parts of the CanvasRenderingContext2D interface shall not be supported due OffscreenCanvas objects having no relation to the DOM or Frame: HitRegions, scrollPathIntoView, drawFocusIfNeeded.
* Due to technical challenges, some implementors [https://bugzilla.mozilla.org/show_bug.cgi?id=801176#c29 (Google and Mozilla)] have expressed a desire to ship without initially supporting text rendering in 2D contexts. Open Issue: Should text support be formally excluded from the specification until implementors are prepared to ship it (or until a more feasible API is designed)?

==== Implementation ====

This proposal has been vetted by developers of Apple's Safari, Google's Chrome, Microsoft's Internet Explorer, and Mozilla's Firefox browsers. All vendors agreed upon the basic form of the API, so it is likely it will be implemented widely and compatibly.

==== Adoption ====

Web page authors have demanded increased parallelism support from the web platform for multiple years. If support for multithreaded rendering is added, it is likely it will be rapidly adopted.

==== Example code ====

Jeff Gilbert from Mozilla has crafted some example code utilizing this API:

* [https://github.com/jdashg/snippets/tree/master/webgl-from-worker Rendering WebGL from a worker using the commit() API]
* [https://github.com/jdashg/snippets/blob/master/webgl-one-to-many/index.html Using one WebGL context to render to many Canvas elements]

[[Category:Proposals]]

WorkerCanvas

2016-08-18T20:35:11Z

Junov:

'''This proposal has been abandonned. Please refer to the [[OffscreenCanvas]] proposal.'''

:''Provides more control over how canvases are rendered.''

== Use Case Description ==

Feedback from web application authors using canvases have shown the need for the following controls:

* (From ShaderToy, Sketchfab, Verold): need to be able to render to multiple regions on the page efficiently using a single canvas context. 3D model warehouse sites desire to show multiple live interactive models on the page, but creating multiple WebGL contexts per page is too inefficient. A single context should be able to render to multiple regions on the page.
* (From Google Maps): need to be able to render WebGL from a worker, transfer the rendered image to the main thread without making any copy of it, and composite it with other HTML on the page, guaranteeing that the updates are all seen in the same rendered frame.
* (From Mozilla and partners using Emscripten and asm.js): need to be able to render WebGL entirely asynchronously from a worker, displaying the results in a canvas owned by the main thread, without any synchronization with the main thread. In this mode, the entire application runs in the worker. The main thread only receives input events and sends them to the worker for processing.

== WebIDL ==

[Constructor(unsigned long width, unsigned long height)]
interface WorkerCanvas {
attribute unsigned long width;
attribute unsigned long height;
RenderingContext? getContext(DOMString contextId, any... arguments);
void toBlob(FileCallback? _callback, optional DOMString type, any... arguments);
ImageBitmap transferToImageBitmap();
};

WorkerCanvas implements Transferable;
ImageBitmap implements Transferable;

partial interface HTMLCanvasElement {
WorkerCanvas transferControlToWorker();
};

partial interface ImageBitmap {
void transferToImage(HTMLImageElement image);
};

== Spec changes ==

Transferring of ImageBitmaps has to be defined. It should neuter the ImageBitmap in the sending thread. Neutering sets the ImageBitmap's width and height to 0.

HTMLCanvasElement.transferControlToWorker behaves like transferControlToProxy in the current WHATWG spec. WorkerCanvas is Transferable, but transfer fails if transferred other than from the main thread to a worker. All its methods throw if not called on a worker, or if it's neutered.

ImageBitmap.transferToImage removes the image element's "src" attribute and makes the image display the contents of the ImageBitmap (until the next transferToImage to that image, or until the image's "src" attribute is set). The ImageBitmap is neutered.

[[Category:Proposals]]

CanvasInWorkers

2016-08-18T20:34:34Z

Junov:

'''This proposal has been abandoned. Please refer to the [[OffscreenCanvas]] proposal.'''

:''This proposal is trying to solve 2 issues: (1) being able to render to a canvas from a worker and (2) being able to render to multiple canvases using a single rendering context''

== Use Case Description ==

There are 2 common uses cases.

Use case #1: Building a multiple view 3D editor (like Blender, or Maya). WebGL, being based on OpenGL, has the limitation that resources belong to a single WebGLRenderingContext. That means if you have 2 or more canvases (left view, top view, perspective view) etc. You currently need a separate context for each one which means you need to load 100s of megabytes of resources multiple times. Allowing a single WebGLRenderingContext to be used with more than 1 canvas would solve this problem.

Use case #2: You'll like to be able to make better use of multiple cores and avoid jank when drawing to a canvas. Many canvas apps (webgl or canvas2d) need to make thousands of API calls per frame at 60 frames a second. Being able to move those calls to a worker would potentially free up the main thread to do other things. It would also potentially help jank by keeping work off the main thread so it does not block the UI.

=== Current Usage and Workarounds ===

Games in HMTL are becoming more common and we'd like to support developers making even more complex games. A few 3D editors are starting to appear and we'd like to help them be as good as their native app counter parts.

== Goals ==

# Allow rendering using the Canvas2D api in a worker.
# Allow rendering using the WebGL api in a worker.
# Allow synchronization of canvas rendering with DOM manipulation
# Allow using 1 Canvas2DRenderingContext with multiple destinations without losing state
# Allow using 1 WebGLRenderingContext with multiple destinations without losing state
# Don't waste memory.
# Do not break existing content (existing APIs still work as is)

== Non Goals ==

# Sharing WebGL Resources between contexts. That is an orthogonal issue

== Proposed Solutions ==

One proposed solution involves <tt>CanvasProxy</tt> and a <tt>commit</tt> method. This solution does not meet the goals above. Specifically it does not handle synchronization issues and may waste memory.

=== Suggested Solution ===
Allow rendering contexts to be created by constructor

var ctx = new Canvas2DRenderingContext();
var gl = new WebGLRenderingContext();

Define `DrawingBuffer`. DrawingBuffer can be considered a ‘handle’ to a single texture (or bucket of pixels). A DrawingBuffer can be passed anywhere a Canvas can be passed. In particular drawImage, texImage2D and texSubImage2D. A DrawingBuffer also has a toDataURL method that is similar to the Canvas’s toDataURL method. A DrawingBuffer can be transferred to and from a worker using the transfer of ownership concept similar to an ArrayBuffer.

A DrawingBuffer is created by constructor as in

var db = new DrawingBuffer(context, {...creation-parameters...});

The context associated with a DrawingBuffer at creation is the only context that may render to that DrawingBuffer.

Canvas becomes a ‘shell’ whose sole purpose is to display DrawingBuffers.

Add 2 functions to `Canvas`. Canvas.transferDrawingBufferToCanvas() and Canvas.copyDrawingBuffer()

Canvas.transferDrawingBufferToCanvas effectively transfers ownership of the DrawingBuffer. The user's DrawingBuffer is neutered. This similar to how transferring a DrawingBuffer from the main thread to a worker makes the main thread no longer able to use it.

A single threaded app that wanted to emulate the existing workflow using DrawingBuffers would do something like this.

<pre>
var gl = new WebGLRenderingContext();
function render() {
var db = new DrawingBuffer(...);
gl.setDrawingBuffer(db);
gl.drawXXX();
canvas.transferDrawingBufferToCanvas(db);
requestAnimationFrame(render);
}
</pre>

Canvas.copyDrawingBuffer() on the other hand copies the DrawingBuffer’s texture/backingstore to the canvas. This is a slower path but emulates the standard Canvas2D persistent backingstore style. The canvas will have to allocate a texture or bucket of pixel to hold the copy if it does not already have one.

Disallow ‘multi-sampled’ / “anti-aliased” DrawingBuffers and instead expose GL_ANGLE_framebuffer_blit
GL_ANGLE_framebuffer_multisample. (webgl specific)

Define ‘DepthStencilBuffer’. Add a function, WebGLRenderingContext.setDepthStencilBuffer (webgl specific)

== Suggested IDL ==

<pre>
interface HTMLCanvas {
...
void transferDrawingBufferToCanvas(DrawingBuffer b);
void copyDrawingBuffer(DrawingBuffer b);
}

interface Canvas2DRenderingContext {
readonly attribute DrawingBuffer drawingBuffer;
void setDrawingBuffer(DrawingBuffer buffer);
CanvasPattern createPattern(DrawingBuffer buffer, ...);
void drawImage(DrawingBuffer buffer,
unrestricted double dx,
unrestricted double dy);
void drawImage(DrawingBuffer buffer,
unrestricted double dx, unrestricted double dy,
unrestricted double dw, unrestricted double dh);
void drawImage(DrawingBuffer buffer,
unrestricted double sx, unrestricted double sy,
unrestricted double sw, unrestricted double sh,
unrestricted double dx, unrestricted double dy,
unrestricted double dw, unrestricted double dh);
}

interface WebGLRenderingContext {
...
readonly attribute DrawingBuffer drawingBuffer;
readonly attribute DepthStencilBuffer depthStencilBuffer;
void setDrawingBuffer(DrawingBuffer buffer);
void setDepthStencilBuffer(DepthStencilBuffer buffer);
void texImage2D(GLenum target, GLint level, GLenum internalformat,
GLenum format, GLenum type, DrawingBuffer buffer);
void texSubImage2D(GLenum target, GLint level,
GLint xoffset, GLint yoffset,
GLenum format, GLenum type,
DrawingBuffer buffer);
}

[ Constructor(RenderingContext c, any contextCreationParameters) ]
interface DrawingBuffer {
readonly attribute long width;
readonly attribute long height;
void setSize(long width, long height);
DOMString toDataURL(in DOMString type)
raises(???Exception);
}

[ Constructor(RenderingContext c, any contextCreationParameters) ]
interface DepthStencilBuffer {
attribute long width;
attribute long height;
readonly attribute long width;
readonly attribute long height;
void setSize(long width, long height);
}
</pre>

== Rationale: ==

'''Q:''' Why get rid of a commit method in workers to propagate changes from a context rendered in a worker to a canvas in the main page?

'''A:''' Using commit there is no way to synchronize updates in a worker with updates to the DOM in the main thread. This solution makes it possible to make sure that DOM objects positioned in the main thread stay in sync with images rendered by a worker. The worker transfers the DrawingBuffer to the main thread via postMessage, and the main thread calls canvas.transferDrawingBufferToCanvas. This solution also avoids unnecessary blits of the canvas’s contents, which is essential for performance.

<br />
'''Q:''' Why disallow anti-aliased DrawingBuffers?

'''A:''' In the existing model when you create a webgl context by calling canvas.getContext() a single multi-sampled renderbuffer is created by the browser. When the browser implicitly does a ‘swapBuffers’ for you it resolves or “blits” this multi-sampled renderbuffer into a texture.

On 30inch display (or a HD-DPI Macbook Pro) a fullscreen multi-sampled renderbuffer requires

<pre>
Bytes per pixel = 4 (rgba) * 4 (depth-stencil)

2560(width) *
1600(height) *
8(bytes per pixel) *
4 (multi-samples)
-----------------------------
= 125meg
</pre>

In the new model, a typical animating application will create a minimum of 2 DrawingBuffers (for double buffering so a worker can render to one while the other is passed back to the main thread for compositing) or possibly 3 (for Triple Buffering). If all the DrawingBuffers are antialiased that’s 375meg of VRAM used up immediately.

On the other hand, if instead we disallow anti-aliased DrawBuffers and expose GL_ANGLE_framebuffer_blit and GL_ANGLE_framebuffer_multisample as WebGL extensions, then a typical app that wants to support anti-aliasing will create a single multisampled renderbuffer and do its own blit to non-multi-sampled DrawingBuffers. For a triple buffered app that would be 218 meg of VRAM.

Another considered solution is to some how magically share a single multi-sample buffer. In a double buffered app you'd transfer a DrawingBuffer from the work to a the main thread. Since the DrawBuffer is intended to be given to a canvas and since you can't render to the drawing buffer because its context is back in the worker then, at transfer time, you could resolve the multi-sampled buffer. And give that multi-sampled buffer to the next DrawBuffer. Unfortunately that won't work under this design. Nothing prevents the user from transferring the DrawingBuffer from a worker to another thread and back to the worker. It should come back as it started. If it was resolved on transfer it would not come back. Another issue is there's nothing preventing the user from making 3 or 4 DrawingBuffers for a single canvas for triple or quadruple buffering. Since you have no idea what a user is going to use a DrawingBuffer or how they are going to use it there's no easy way for them all to magically share the same multi-sample buffer.

<br />
'''Q:''' Should we allow anti-aliased DrawingBuffers?

# Yes, developers who care about memory can create non anti-aliased buffers. Developers who don’t care can avoid the hassle of needing to make a multi-sampled renderbuffer and blitting
# No, all developers that want to use DrawingBuffers and get anti-aliasing must use GL_ANGLE_framebuffer_multisample and GL_ANGLE_framebuffer_blit

Resolution: #2. Saving memory is especially important in situations like tablets, multiple tabs, and systems without virtual memory. Rather than let the bad behavior be the easy path we chose to encourage the good behavior.

<br />
'''Q:''' Why separate out DepthStencilBuffer?

'''A:''' For similar reasons as disallowing anti-aliasing. (see above)

Given that DrawingBuffers are transferred by transferring ownership, and given that in the common case of transferring a DrawingBuffer to the main thread to be composited, there is no reason to also transfer the depth/stencil buffers. Doing so would mean multiple depth and stencil buffers would need to be allocated so one thread can render to them while the main thread is compositing.

Apps that use GL_ANGLE_framebuffer_multisample and GL_ANGLE_framebuffer_blit to support anti-aliasing will never need to create a ‘DepthStencilBuffer’ as they will end up creating a gl.DEPTH_STENCIL texture or renderbuffer

Separating them out also makes more sense for Canvas2D which never needs a depth/stencil buffer.

<br />
'''Q:''' For a worker based animated app what’s the expected code flow?

'''A:'''

<pre>
// render.js: -- worker --
var gl = new WebGLRenderingContext();
var dsBuffer = new DepthStencilBuffer(gl, …);
gl.setDepthStencilBuffer(dsBuffer);

var render = function(self) {
// make a new DrawingBuffer
var db = new DrawingBuffer(gl, ...);

// Render to drawing buffer.
gl.setDrawingBuffer(db);
gl.drawXXX(...);

// Pass the drawing buffer to the main thread for compositing
self.postMessage(db, [db]);

// request the next frame.
self.requestAnimationFrame(render);
}
render();

// Main thread:
var canvas = document.getElementById(“someCanvas”);
var worker = new Worker(“render.js”);
worker.addEventListener(‘message’, function(db) {
canvas.transferDrawingBufferToCanvas(db);
}, false);
</pre>

The thing to notice is the worker is creating a new DrawingBuffer every requestAnimationFrame. It then transfers ownership to the main thread. The main thread transfers it to the canvas. The browser can, behind the scenes, keep a queue of DrawingBuffers so that allocation of new ones is fast.

<br />
'''Q:''' Why does a DrawingBuffer’s constructor take a context?

'''A:''' DrawingBuffers can only be used with the context they are created with. Putting the context in the constructor spells out this relationship. The following is illegal

var gl1 = new WebGLRenderingContext();
var gl2 = new WebGLRenderingContext();
var db = new DrawingBuffer(gl1);
gl1.setDrawingBuffer(db);
gl2.setDrawingBuffer(db); // error. db belongs to gl1

<br />
'''Q:''' Can you use a Canvas2DRenderingContext without a DrawingBuffer?

'''A:''' Yes. But only to create patterns, gradients, etc. All methods that rasterize will throw an exception until the canvas is associated with a DrawingBuffer by calling Canvas2DRenderingContext.setDrawingBuffer()

<br />
'''Q:''' Can you use a WebGLRenderingContext without a DrawingBuffer?

'''A:''' Yes, you can create WebGL resources (textures, buffers, programs, etc..) You can render to framebuffer objects and call readPixels on them. Rendering to the default framebuffer <tt>null</tt> bind target will generate gl.INVALID_FRAMEBUFFER_OPERATION if no valid DrawingBuffer is set. A neutered DrawingBuffer, one that has been transferred to another thread, or one which has been transferred to a canvas, is not a valid drawing buffer.

<br />
'''Q:''' Do you need to call setDrawingBuffer if there is only 1 buffer?

'''A:''' No, creating a DrawingBuffer implicity calls setDrawingBuffer.

gl = new WebGLRenderingContext():
db = new DrawingBuffer(gl, …);
gl.clear(gl.COLOR_BUFFER_BIT); // renders to db.

<br />
'''Q:''' Is any context state lost when setDrawingBuffer is called?

'''A:''' No. The context’s state is preserved across calls to setDrawingBuffer for both Canvas2DRenderingContext and WebGLRenderingContext.

<br />
'''Q:''' Can you render to a DrawingBuffer that has been passed to another thread?

'''A:''' No, DrawingBuffers pass ownership. The DrawingBuffer in the thread that passed it is now neutered, just like an transferred ArrayBuffer is neutered.

<br />
'''Q:''' Can you transfer a DrawingBuffer to 2 canvases?

'''A:''' No, Canvas.transferDrawingBufferToCanvas takes ownership of the DrawingBuffer. The DrawingBuffer left for the user has been neutered.

<br />
'''Q:''' What happens if I transferDrawingBufferToCanvas DrawingBuffers of different sizes?

'''A:''' The canvas does not change its display size, it just displays the DrawingBuffer transferred at the size defined by css or if no css is specified then the canvas’ original size.

<br />
'''Q:''' Can you call getContext on a canvas that has had transferDrawingBufferToCanvas called on it?

'''A:''' No

<br />
'''Q:''' Can you call transferDrawingBufferToCanvas on a canvas that has had its getContext method called?

'''A:''' No. It might be possible to make this work but it’s probably not worth it?

<br />
'''Q:''' Can you use these features in “shared workers”?

'''A:''' No (or at least not for now)

<br />
'''Q:''' What happens to the Canvas2DRenderingContext.canvas and WebGLRenderingContext.canvas properties?

'''A:''' For contexts created by constructor they are set to undefined

<br />
'''Q:''' Should you be able to reference the current DrawingBuffer on a RenderingContext?

In other words, should there be a getDrawingBuffer or a ‘drawingBuffer’ property?

'''A:''' Yes, but it’s only set if you call setDrawingBuffer. In other words if you call getContext to make your context then this property would be undefined or if it’s a function it would return undefined.

<br />
'''Q:''' Should you be able to change the size of a DrawingBuffer?

'''A:'''

# Yes, set width and/or height and its size will change<br /> Issue. Allocating DrawingBuffers is a slow operation so implementations would like to avoid re-allocting once when width is set and again when height is set. Deferring that allocation is no fun to implement.
# Yes, use a setSize(width, height) method<br /> This avoids the complications of using the writable properties
# No, just allocate a new DrawingBuffer<br /> The only issue here is quick GCing.

Resolution: #2

== Issues: ==

=== Workers can flood the graphics system with too much work. ===

In the main thread you can write code like this

// render as fast as you can
function render() {
for (var i = 0; i < 1000; +i) {
gl.drawXXX(...);
}
}
setInterval(render, 0);

This will DoS many systems by saturating the GPU with draw calls. The solution on the main thread is the implicit ‘SwapBuffers’. Every time JavaScript exits the interval event the system can pause or block. But this is not true in workers. As there is no implicit swap and workers can run in infinite loops there is no way to prevent this situation. While preventing infinite loops is outside the scope of what we can deal with consider a worker that generates frames at 90fps and a main thread that composites them at 60fps. There is nothing to stop the worker from generating too much work or a giant backlog of GPU work.

Ideas

# So what. The worker will run out of memory.<br /> Unfortunately until that happens the entire system may be unresponsive (not just the browser)
# Allow rendering in workers only inside some callback.<br /> For example, if it is only possible to render inside a worker during a requestAnimationFrame event the browser can throttle the worker by sending less events.<br /> The minor problem with this solution is it makes non animating apps slightly convoluted to write. Let’s say you want to make a Maya or Blender type app so you only render on demand. You end up getting a mousemove event, posting a message to a worker, the worker would issue raf so the raf can do the rendering. Maybe that’s not too convoluted.
# Other?

Note: Exposing DrawingBuffer in the main thread causes the same problem

Which suggests that even the main thread should not be allowed to render to contexts created by a constructor except in requestAnimationFrame?

=== How should GL_ANGLE_framebuffer_multisample be specified w.r.t. number of samples? (webgl specific) ===

We’d like apps not to fail based on the “samples” parameter to renderbufferStorageMultisample but GL_ANGLE_framebuffer_multisample is specified that it must allocate a renderbuffer with the user specified number of “samples” or greater. That means if an app passes the wrong number in (say hardcodes a 4) and the user’s GPU does not support 4 samples or the user’s GPU multi-sample support is blacklisted the app will fail.

We’d prefer a more permissive API by letting the implementation choose the number of samples so that more apps will succeed.

Ideas

# Leave the API as is. Apps may suddenly fail on different hardware or the same hardware when multi-sampling is blacklisted
# Leave the API the same but let the implementation choose the actual number of samples. Apps that need to know how many samples were chosen can query how many they got with getRenderbufferParameter.
# Change the API slightly by providing an enum (high, medium, low, none) as a quality input to renderbufferStorageMultisample instead of specifying the specific number of samples. Implementation can choose their own interpretation of ‘low’, ‘medium’, ‘high’

[[Category:Proposals]]

CanvasInWorkers

2016-08-18T20:34:21Z

Junov:

'''This proposal has been abandoned. Please refer to the more [[OffscreenCanvas]] proposal.'''

:''This proposal is trying to solve 2 issues: (1) being able to render to a canvas from a worker and (2) being able to render to multiple canvases using a single rendering context''

== Use Case Description ==

There are 2 common uses cases.

Use case #1: Building a multiple view 3D editor (like Blender, or Maya). WebGL, being based on OpenGL, has the limitation that resources belong to a single WebGLRenderingContext. That means if you have 2 or more canvases (left view, top view, perspective view) etc. You currently need a separate context for each one which means you need to load 100s of megabytes of resources multiple times. Allowing a single WebGLRenderingContext to be used with more than 1 canvas would solve this problem.

Use case #2: You'll like to be able to make better use of multiple cores and avoid jank when drawing to a canvas. Many canvas apps (webgl or canvas2d) need to make thousands of API calls per frame at 60 frames a second. Being able to move those calls to a worker would potentially free up the main thread to do other things. It would also potentially help jank by keeping work off the main thread so it does not block the UI.

=== Current Usage and Workarounds ===

Games in HMTL are becoming more common and we'd like to support developers making even more complex games. A few 3D editors are starting to appear and we'd like to help them be as good as their native app counter parts.

== Goals ==

# Allow rendering using the Canvas2D api in a worker.
# Allow rendering using the WebGL api in a worker.
# Allow synchronization of canvas rendering with DOM manipulation
# Allow using 1 Canvas2DRenderingContext with multiple destinations without losing state
# Allow using 1 WebGLRenderingContext with multiple destinations without losing state
# Don't waste memory.
# Do not break existing content (existing APIs still work as is)

== Non Goals ==

# Sharing WebGL Resources between contexts. That is an orthogonal issue

== Proposed Solutions ==

One proposed solution involves <tt>CanvasProxy</tt> and a <tt>commit</tt> method. This solution does not meet the goals above. Specifically it does not handle synchronization issues and may waste memory.

=== Suggested Solution ===
Allow rendering contexts to be created by constructor

var ctx = new Canvas2DRenderingContext();
var gl = new WebGLRenderingContext();

Define `DrawingBuffer`. DrawingBuffer can be considered a ‘handle’ to a single texture (or bucket of pixels). A DrawingBuffer can be passed anywhere a Canvas can be passed. In particular drawImage, texImage2D and texSubImage2D. A DrawingBuffer also has a toDataURL method that is similar to the Canvas’s toDataURL method. A DrawingBuffer can be transferred to and from a worker using the transfer of ownership concept similar to an ArrayBuffer.

A DrawingBuffer is created by constructor as in

var db = new DrawingBuffer(context, {...creation-parameters...});

The context associated with a DrawingBuffer at creation is the only context that may render to that DrawingBuffer.

Canvas becomes a ‘shell’ whose sole purpose is to display DrawingBuffers.

Add 2 functions to `Canvas`. Canvas.transferDrawingBufferToCanvas() and Canvas.copyDrawingBuffer()

Canvas.transferDrawingBufferToCanvas effectively transfers ownership of the DrawingBuffer. The user's DrawingBuffer is neutered. This similar to how transferring a DrawingBuffer from the main thread to a worker makes the main thread no longer able to use it.

A single threaded app that wanted to emulate the existing workflow using DrawingBuffers would do something like this.

<pre>
var gl = new WebGLRenderingContext();
function render() {
var db = new DrawingBuffer(...);
gl.setDrawingBuffer(db);
gl.drawXXX();
canvas.transferDrawingBufferToCanvas(db);
requestAnimationFrame(render);
}
</pre>

Canvas.copyDrawingBuffer() on the other hand copies the DrawingBuffer’s texture/backingstore to the canvas. This is a slower path but emulates the standard Canvas2D persistent backingstore style. The canvas will have to allocate a texture or bucket of pixel to hold the copy if it does not already have one.

Disallow ‘multi-sampled’ / “anti-aliased” DrawingBuffers and instead expose GL_ANGLE_framebuffer_blit
GL_ANGLE_framebuffer_multisample. (webgl specific)

Define ‘DepthStencilBuffer’. Add a function, WebGLRenderingContext.setDepthStencilBuffer (webgl specific)

== Suggested IDL ==

<pre>
interface HTMLCanvas {
...
void transferDrawingBufferToCanvas(DrawingBuffer b);
void copyDrawingBuffer(DrawingBuffer b);
}

interface Canvas2DRenderingContext {
readonly attribute DrawingBuffer drawingBuffer;
void setDrawingBuffer(DrawingBuffer buffer);
CanvasPattern createPattern(DrawingBuffer buffer, ...);
void drawImage(DrawingBuffer buffer,
unrestricted double dx,
unrestricted double dy);
void drawImage(DrawingBuffer buffer,
unrestricted double dx, unrestricted double dy,
unrestricted double dw, unrestricted double dh);
void drawImage(DrawingBuffer buffer,
unrestricted double sx, unrestricted double sy,
unrestricted double sw, unrestricted double sh,
unrestricted double dx, unrestricted double dy,
unrestricted double dw, unrestricted double dh);
}

interface WebGLRenderingContext {
...
readonly attribute DrawingBuffer drawingBuffer;
readonly attribute DepthStencilBuffer depthStencilBuffer;
void setDrawingBuffer(DrawingBuffer buffer);
void setDepthStencilBuffer(DepthStencilBuffer buffer);
void texImage2D(GLenum target, GLint level, GLenum internalformat,
GLenum format, GLenum type, DrawingBuffer buffer);
void texSubImage2D(GLenum target, GLint level,
GLint xoffset, GLint yoffset,
GLenum format, GLenum type,
DrawingBuffer buffer);
}

[ Constructor(RenderingContext c, any contextCreationParameters) ]
interface DrawingBuffer {
readonly attribute long width;
readonly attribute long height;
void setSize(long width, long height);
DOMString toDataURL(in DOMString type)
raises(???Exception);
}

[ Constructor(RenderingContext c, any contextCreationParameters) ]
interface DepthStencilBuffer {
attribute long width;
attribute long height;
readonly attribute long width;
readonly attribute long height;
void setSize(long width, long height);
}
</pre>

== Rationale: ==

'''Q:''' Why get rid of a commit method in workers to propagate changes from a context rendered in a worker to a canvas in the main page?

'''A:''' Using commit there is no way to synchronize updates in a worker with updates to the DOM in the main thread. This solution makes it possible to make sure that DOM objects positioned in the main thread stay in sync with images rendered by a worker. The worker transfers the DrawingBuffer to the main thread via postMessage, and the main thread calls canvas.transferDrawingBufferToCanvas. This solution also avoids unnecessary blits of the canvas’s contents, which is essential for performance.

<br />
'''Q:''' Why disallow anti-aliased DrawingBuffers?

'''A:''' In the existing model when you create a webgl context by calling canvas.getContext() a single multi-sampled renderbuffer is created by the browser. When the browser implicitly does a ‘swapBuffers’ for you it resolves or “blits” this multi-sampled renderbuffer into a texture.

On 30inch display (or a HD-DPI Macbook Pro) a fullscreen multi-sampled renderbuffer requires

<pre>
Bytes per pixel = 4 (rgba) * 4 (depth-stencil)

2560(width) *
1600(height) *
8(bytes per pixel) *
4 (multi-samples)
-----------------------------
= 125meg
</pre>

In the new model, a typical animating application will create a minimum of 2 DrawingBuffers (for double buffering so a worker can render to one while the other is passed back to the main thread for compositing) or possibly 3 (for Triple Buffering). If all the DrawingBuffers are antialiased that’s 375meg of VRAM used up immediately.

On the other hand, if instead we disallow anti-aliased DrawBuffers and expose GL_ANGLE_framebuffer_blit and GL_ANGLE_framebuffer_multisample as WebGL extensions, then a typical app that wants to support anti-aliasing will create a single multisampled renderbuffer and do its own blit to non-multi-sampled DrawingBuffers. For a triple buffered app that would be 218 meg of VRAM.

Another considered solution is to some how magically share a single multi-sample buffer. In a double buffered app you'd transfer a DrawingBuffer from the work to a the main thread. Since the DrawBuffer is intended to be given to a canvas and since you can't render to the drawing buffer because its context is back in the worker then, at transfer time, you could resolve the multi-sampled buffer. And give that multi-sampled buffer to the next DrawBuffer. Unfortunately that won't work under this design. Nothing prevents the user from transferring the DrawingBuffer from a worker to another thread and back to the worker. It should come back as it started. If it was resolved on transfer it would not come back. Another issue is there's nothing preventing the user from making 3 or 4 DrawingBuffers for a single canvas for triple or quadruple buffering. Since you have no idea what a user is going to use a DrawingBuffer or how they are going to use it there's no easy way for them all to magically share the same multi-sample buffer.

<br />
'''Q:''' Should we allow anti-aliased DrawingBuffers?

# Yes, developers who care about memory can create non anti-aliased buffers. Developers who don’t care can avoid the hassle of needing to make a multi-sampled renderbuffer and blitting
# No, all developers that want to use DrawingBuffers and get anti-aliasing must use GL_ANGLE_framebuffer_multisample and GL_ANGLE_framebuffer_blit

Resolution: #2. Saving memory is especially important in situations like tablets, multiple tabs, and systems without virtual memory. Rather than let the bad behavior be the easy path we chose to encourage the good behavior.

<br />
'''Q:''' Why separate out DepthStencilBuffer?

'''A:''' For similar reasons as disallowing anti-aliasing. (see above)

Given that DrawingBuffers are transferred by transferring ownership, and given that in the common case of transferring a DrawingBuffer to the main thread to be composited, there is no reason to also transfer the depth/stencil buffers. Doing so would mean multiple depth and stencil buffers would need to be allocated so one thread can render to them while the main thread is compositing.

Apps that use GL_ANGLE_framebuffer_multisample and GL_ANGLE_framebuffer_blit to support anti-aliasing will never need to create a ‘DepthStencilBuffer’ as they will end up creating a gl.DEPTH_STENCIL texture or renderbuffer

Separating them out also makes more sense for Canvas2D which never needs a depth/stencil buffer.

<br />
'''Q:''' For a worker based animated app what’s the expected code flow?

'''A:'''

<pre>
// render.js: -- worker --
var gl = new WebGLRenderingContext();
var dsBuffer = new DepthStencilBuffer(gl, …);
gl.setDepthStencilBuffer(dsBuffer);

var render = function(self) {
// make a new DrawingBuffer
var db = new DrawingBuffer(gl, ...);

// Render to drawing buffer.
gl.setDrawingBuffer(db);
gl.drawXXX(...);

// Pass the drawing buffer to the main thread for compositing
self.postMessage(db, [db]);

// request the next frame.
self.requestAnimationFrame(render);
}
render();

// Main thread:
var canvas = document.getElementById(“someCanvas”);
var worker = new Worker(“render.js”);
worker.addEventListener(‘message’, function(db) {
canvas.transferDrawingBufferToCanvas(db);
}, false);
</pre>

The thing to notice is the worker is creating a new DrawingBuffer every requestAnimationFrame. It then transfers ownership to the main thread. The main thread transfers it to the canvas. The browser can, behind the scenes, keep a queue of DrawingBuffers so that allocation of new ones is fast.

<br />
'''Q:''' Why does a DrawingBuffer’s constructor take a context?

'''A:''' DrawingBuffers can only be used with the context they are created with. Putting the context in the constructor spells out this relationship. The following is illegal

var gl1 = new WebGLRenderingContext();
var gl2 = new WebGLRenderingContext();
var db = new DrawingBuffer(gl1);
gl1.setDrawingBuffer(db);
gl2.setDrawingBuffer(db); // error. db belongs to gl1

<br />
'''Q:''' Can you use a Canvas2DRenderingContext without a DrawingBuffer?

'''A:''' Yes. But only to create patterns, gradients, etc. All methods that rasterize will throw an exception until the canvas is associated with a DrawingBuffer by calling Canvas2DRenderingContext.setDrawingBuffer()

<br />
'''Q:''' Can you use a WebGLRenderingContext without a DrawingBuffer?

'''A:''' Yes, you can create WebGL resources (textures, buffers, programs, etc..) You can render to framebuffer objects and call readPixels on them. Rendering to the default framebuffer <tt>null</tt> bind target will generate gl.INVALID_FRAMEBUFFER_OPERATION if no valid DrawingBuffer is set. A neutered DrawingBuffer, one that has been transferred to another thread, or one which has been transferred to a canvas, is not a valid drawing buffer.

<br />
'''Q:''' Do you need to call setDrawingBuffer if there is only 1 buffer?

'''A:''' No, creating a DrawingBuffer implicity calls setDrawingBuffer.

gl = new WebGLRenderingContext():
db = new DrawingBuffer(gl, …);
gl.clear(gl.COLOR_BUFFER_BIT); // renders to db.

<br />
'''Q:''' Is any context state lost when setDrawingBuffer is called?

'''A:''' No. The context’s state is preserved across calls to setDrawingBuffer for both Canvas2DRenderingContext and WebGLRenderingContext.

<br />
'''Q:''' Can you render to a DrawingBuffer that has been passed to another thread?

'''A:''' No, DrawingBuffers pass ownership. The DrawingBuffer in the thread that passed it is now neutered, just like an transferred ArrayBuffer is neutered.

<br />
'''Q:''' Can you transfer a DrawingBuffer to 2 canvases?

'''A:''' No, Canvas.transferDrawingBufferToCanvas takes ownership of the DrawingBuffer. The DrawingBuffer left for the user has been neutered.

<br />
'''Q:''' What happens if I transferDrawingBufferToCanvas DrawingBuffers of different sizes?

'''A:''' The canvas does not change its display size, it just displays the DrawingBuffer transferred at the size defined by css or if no css is specified then the canvas’ original size.

<br />
'''Q:''' Can you call getContext on a canvas that has had transferDrawingBufferToCanvas called on it?

'''A:''' No

<br />
'''Q:''' Can you call transferDrawingBufferToCanvas on a canvas that has had its getContext method called?

'''A:''' No. It might be possible to make this work but it’s probably not worth it?

<br />
'''Q:''' Can you use these features in “shared workers”?

'''A:''' No (or at least not for now)

<br />
'''Q:''' What happens to the Canvas2DRenderingContext.canvas and WebGLRenderingContext.canvas properties?

'''A:''' For contexts created by constructor they are set to undefined

<br />
'''Q:''' Should you be able to reference the current DrawingBuffer on a RenderingContext?

In other words, should there be a getDrawingBuffer or a ‘drawingBuffer’ property?

'''A:''' Yes, but it’s only set if you call setDrawingBuffer. In other words if you call getContext to make your context then this property would be undefined or if it’s a function it would return undefined.

<br />
'''Q:''' Should you be able to change the size of a DrawingBuffer?

'''A:'''

# Yes, set width and/or height and its size will change<br /> Issue. Allocating DrawingBuffers is a slow operation so implementations would like to avoid re-allocting once when width is set and again when height is set. Deferring that allocation is no fun to implement.
# Yes, use a setSize(width, height) method<br /> This avoids the complications of using the writable properties
# No, just allocate a new DrawingBuffer<br /> The only issue here is quick GCing.

Resolution: #2

== Issues: ==

=== Workers can flood the graphics system with too much work. ===

In the main thread you can write code like this

// render as fast as you can
function render() {
for (var i = 0; i < 1000; +i) {
gl.drawXXX(...);
}
}
setInterval(render, 0);

This will DoS many systems by saturating the GPU with draw calls. The solution on the main thread is the implicit ‘SwapBuffers’. Every time JavaScript exits the interval event the system can pause or block. But this is not true in workers. As there is no implicit swap and workers can run in infinite loops there is no way to prevent this situation. While preventing infinite loops is outside the scope of what we can deal with consider a worker that generates frames at 90fps and a main thread that composites them at 60fps. There is nothing to stop the worker from generating too much work or a giant backlog of GPU work.

Ideas

# So what. The worker will run out of memory.<br /> Unfortunately until that happens the entire system may be unresponsive (not just the browser)
# Allow rendering in workers only inside some callback.<br /> For example, if it is only possible to render inside a worker during a requestAnimationFrame event the browser can throttle the worker by sending less events.<br /> The minor problem with this solution is it makes non animating apps slightly convoluted to write. Let’s say you want to make a Maya or Blender type app so you only render on demand. You end up getting a mousemove event, posting a message to a worker, the worker would issue raf so the raf can do the rendering. Maybe that’s not too convoluted.
# Other?

Note: Exposing DrawingBuffer in the main thread causes the same problem

Which suggests that even the main thread should not be allowed to render to contexts created by a constructor except in requestAnimationFrame?

=== How should GL_ANGLE_framebuffer_multisample be specified w.r.t. number of samples? (webgl specific) ===

We’d like apps not to fail based on the “samples” parameter to renderbufferStorageMultisample but GL_ANGLE_framebuffer_multisample is specified that it must allocate a renderbuffer with the user specified number of “samples” or greater. That means if an app passes the wrong number in (say hardcodes a 4) and the user’s GPU does not support 4 samples or the user’s GPU multi-sample support is blacklisted the app will fail.

We’d prefer a more permissive API by letting the implementation choose the number of samples so that more apps will succeed.

Ideas

# Leave the API as is. Apps may suddenly fail on different hardware or the same hardware when multi-sampling is blacklisted
# Leave the API the same but let the implementation choose the actual number of samples. Apps that need to know how many samples were chosen can query how many they got with getRenderbufferParameter.
# Change the API slightly by providing an enum (high, medium, low, none) as a quality input to renderbufferStorageMultisample instead of specifying the specific number of samples. Implementation can choose their own interpretation of ‘low’, ‘medium’, ‘high’

[[Category:Proposals]]

WorkerCanvas

2016-08-18T20:33:02Z

Junov:

'''This proposal is obsolete. Please refer to the more up-to-date [[OffscreenCanvas]] proposal.'''

:''Provides more control over how canvases are rendered.''

== Use Case Description ==

Feedback from web application authors using canvases have shown the need for the following controls:

* (From ShaderToy, Sketchfab, Verold): need to be able to render to multiple regions on the page efficiently using a single canvas context. 3D model warehouse sites desire to show multiple live interactive models on the page, but creating multiple WebGL contexts per page is too inefficient. A single context should be able to render to multiple regions on the page.
* (From Google Maps): need to be able to render WebGL from a worker, transfer the rendered image to the main thread without making any copy of it, and composite it with other HTML on the page, guaranteeing that the updates are all seen in the same rendered frame.
* (From Mozilla and partners using Emscripten and asm.js): need to be able to render WebGL entirely asynchronously from a worker, displaying the results in a canvas owned by the main thread, without any synchronization with the main thread. In this mode, the entire application runs in the worker. The main thread only receives input events and sends them to the worker for processing.

== WebIDL ==

[Constructor(unsigned long width, unsigned long height)]
interface WorkerCanvas {
attribute unsigned long width;
attribute unsigned long height;
RenderingContext? getContext(DOMString contextId, any... arguments);
void toBlob(FileCallback? _callback, optional DOMString type, any... arguments);
ImageBitmap transferToImageBitmap();
};

WorkerCanvas implements Transferable;
ImageBitmap implements Transferable;

partial interface HTMLCanvasElement {
WorkerCanvas transferControlToWorker();
};

partial interface ImageBitmap {
void transferToImage(HTMLImageElement image);
};

== Spec changes ==

Transferring of ImageBitmaps has to be defined. It should neuter the ImageBitmap in the sending thread. Neutering sets the ImageBitmap's width and height to 0.

HTMLCanvasElement.transferControlToWorker behaves like transferControlToProxy in the current WHATWG spec. WorkerCanvas is Transferable, but transfer fails if transferred other than from the main thread to a worker. All its methods throw if not called on a worker, or if it's neutered.

ImageBitmap.transferToImage removes the image element's "src" attribute and makes the image display the contents of the ImageBitmap (until the next transferToImage to that image, or until the image's "src" attribute is set). The ImageBitmap is neutered.

[[Category:Proposals]]

CanvasColorSpace

2016-05-14T03:26:24Z

Junov:

== Latest version of the proposal has moved to [https://github.com/junov/CanvasColorSpace/blob/master/CanvasColorSpaceProposal.md here] ==

== Use Case Description ==
* Contents displayed through a canvas element should be color managed in order to minimize differences in appearance across browsers and display devices. Improving color fidelity matters a lot for artistic uses (e.g. photo and paint apps) and for e-commerce (product presentation).
* Canvases should be able to take advantage of the full color gamut of the display device.
* Creative apps that do image manipulation generally prefer compositing, filtering and interpolation calculations to be performed in a linear color space.

=== Current Limitations ===
* The color space of canvases is undefined in the current specification.
* The bit-depth of canvases is currently fixed to 8 bits per component, which is below the capabilities of some monitors. Monitors with higher contrast ratios require more bits per component to avoid banding.

=== Current Usage and Workarounds ===
The lack of color space interoperability is hard to work around. With some browser implementations that color correct images drawn to canvases by applying the display profile, apps that want to use canvases for color corrected image processing are stuck doing convoluted workarounds, such as:
* reverse-engineer the display profile by drawing test pattern images to the canvas and inspecting the color corrected result via getImageData
* bypass CanvasRenderingContext2D.drawImage() and use image decoders implemented in JavaScript to extract raw image data that was not tainted by the browser's color correction behavior.''

An aspect of current implementations that is interoperable is that colors match between CSS/HTML and canvases:
* A color value used as a canvas drawing style will have the same appearance as if the same color value were used as a CSS style
* An image resource drawn to a canvas element will have the same appearance as if it were displayed as the replaced content of an HTML element or used as a CSS style value.

This color matching behavior needs to be preserved to avoid breaking pre-existing content.

Some implementations convert images drawn to canvases to the sRGB color space. This has the advantage of making the color correction behavior device independent, but it clamps the gamuts of the rendered content to the sRGB gamut, which is significantly narrower than the gamuts of some current consumer devices.

=== Requests for this Feature ===

* <cite>[https://github.com/whatwg/html/issues/299]</cite> <blockquote><p>Allow 2dcontexts to use deeper color buffers</p></blockquote>
* <cite>[https://bugs.chromium.org/p/chromium/issues/detail?id=425935]</cite> <blockquote><p>Wrong color profile with 2D canvas</p></blockquote>
* Engineers from the Google Photos, Maps and Sheets teams have expressed a desire for canvases to become color managed. Particularly for the use case of resizing an imaging, using a canvas, prior to uploading it to the server, to save bandwidth. The problem is that the images retrieved from a canvas are in an undefined color space and no color space information is encoded by toDataURL or toBlob.

== Proposed Solutions ==

=== Proposed solution: CanvasColorSpace ===
:Add a canvas color space creation parameter that allows user code to chose between backwards compatible behavior and color managed behaviors The same color space option would exist in the ImageData and ImageBitmap interfaces.

==== Processing Model ====

===== The color-space canvas creation parameter =====

IDL:
<code><pre>
enum CanvasColorSpace {
"legacy-srgb",
"srgb",
"linear-rec-2020",
"optimal"
};

dictionary CanvasRenderingContext2DSettings {
boolean alpha = true;
CanvasColorSpace color-space = "legacy-srgb";
};
</pre></code>

Example:
<code><pre>
canvas.getContext('2d', { color-space: "srgb"})
</pre></code>

===== The legacy-srgb color space =====

* Assures backwards compatible behavior
* Guarantees color matching with CSS and HTML content
* Color management behavior is implementation specific, may not use strict sRGB space, but is expected to be near sRGB. For example, could be display referred color space.
* toDataURL/toBlob produce resources with no color profile (backwards compat)
* Image resources with no color profile are never color corrected (backwards compat). This rule and the previous one allow for lossless toDataURL/drawImage round trips, which is a significant use case.

===== The srgb color space =====

* May break color matching with CSS on implementations that do not color-manage CSS.
* 8 bit unsigned integers per color component.
* All content drawn into the canvas must be color corrected to sRGB
* displayed canvases must be color corrected for the display if a display color profile is available. This color correction happens downstream at the compositing stage, and has no script-visible side-effects.
* Compositing, filtering and interpolation operations must perform all arithmetic in '''linear''' sRGB space.
* toDataURL/toBlob produce resources tagged as being in the sRGB colorspace
* Images with no color profile, when drawn to the canvas, are assumed to already be in the sRGB color space.

===== The rec-2020 color space =====

* Color space provided for wide gamut support without increasing memory cost.
* 8 bit unsigned integers per color component.
* All content drawn into the canvas must be color corrected to the rec-2020 color space.
* Displayed canvases must be color corrected for the display if a display color profile is available. This color correction happens downstream at the compositing stage, and has no script-visible side-effects. If there is no display color profile, the user agent should assume the display uses sRGB.
* Compositing, filtering and interpolation operations must perform all arithmetic in '''linear''' rec-2020 space.
* toDataURL/toBlob produce image resources in the rec-2020 colorspace.
* Images with no color profile, when drawn to the canvas, are assumed to be in the sRGB color space, and must therefore be converted from sRGB to rec-2020.

===== The linear-rec-2020 color space =====
* Color space provided for wide gamut and high dynamic range rendering.
* User agents may decide not to support the mode, based on host machine capabilities
* Uses 16-bit floating point representation.
* The color space corresponds to ITU-R Recommendation BT.2020, '''without gamma compression'''.
* toDataURL/toBlob convert image data to the rec-2020 color space (with gamma), and produce image resources with at least 12 bits per color component, if the format supports it. Thus, in the case of the png format, which supports 8 or 16 bits per component, 16bpc would be used.
* Image with no color profile, when drawn to the canvas, are assumed to be in the sRGB color space, and are converted to linear-rec-2020 for the purpose of the draw.

===== The optimal color space =====
The "optimal" option lets the user agent decide which space is optimal for the current display device based on the device's capabilities and color profile characteristics.
* This option may not select "legacy-srgb"
* Graphics devices with color gamuts that extend significantly beyond the sRGB color space should cause the UA to favor rec-2020; and linear linear-rec-2020 should be used for displays with high dynamic ranges.
* Graphics devices that would not produce noticeably higher quality visual results in rec-2020 or linear-rec-2020 should cause the UA to favor "srgb".

===== Feature detection =====

Rendering context objects are to expose a new "settings" attribute, which represents the settings that were successfully applied at context creation time.

Note: An alternative approach that was considered was to augment the probablySupportsContext() API by making it check the second argument. That approach is difficult to consolidate with how dictionary argument are meant to work, where unsupported entries are just ignored.

===== ImageBitmap =====

ImageBitmap objects are augmented to have an internal color space attribute of type CanvasColorSpace. The colorSpaceConversion creation attribute is to be augmented with new enum values for coercing conversions to a specific CanvasColorSpace at creation time.

===== ImageData =====

IDL
<code><pre>
typedef (Uint8ClampedArray or Float32Array) ImageDataArray;

[Constructor(unsigned long sw, unsigned long sh, optional CanvasColorSpace colorSpace = "legacy-srgb"),
Constructor(ImageDataArray data, unsigned long sw, optional unsigned long sh, optional CanvasColorSpace colorSpace),
Exposed=(Window,Worker)]
interface ImageData {
readonly attribute unsigned long width;
readonly attribute unsigned long height;
readonly attribute ImageDataArray data;
readonly attribute CanvasColorSpace colorSpace;
};
</pre></code>

* <code>data</code> is a Uint8ClampedArray if colorSpace is "srgb" or "legacy-srgb"
* <code>data</code> is a Float32Array if colorSpace is "linear-rec-2020"
* getImageData() produces an ImageData object in the same color space as the source canvas
* putImageData() performs a color space conversion to the color space of the destination canvas.

==== Limitations ====
* No support for arbitrary color spaces and bit depth. This capability could be added in the future. The current proposal attempts to solve the problem with a minimal API surface, and keeps the implementation scope reasonable. The extensible design will allow us to extend the capabilities in the future if necessary. The rec-2020 space was chosen for its very wide gamut and its non-virtual primary colors, which strikes a balance that is deemed practical.
* toDataURL is lossy when us on a canvas that is in the linear-rec-2020 space. Possible future improvements could solve or mitigate this issue by adding more file formats or adding options to specify the resource color space.
* ImageData uses float32, which is inefficient due to memory consumption and necessary conversion operations. Float32 was chosen because it is convenient for manipulation (e.g. image processing) due to its native support in JavaScript (and current CPUs). A possible extension would be to add and option for rec-2020 content to be encoded as float16s packed into Uint16 values.

==== Security and privacy issues ====
Some current implementations of CanvasRenderingContext2D color correct image resources for the display as they are drawn to the canvas. In other words, the canvas is in output referred color space. This is a known fingerprinting vulnerability since it exposes the user's display's color profile to scripts via getImageData. The current proposal does not solve the fingerprinting issue because it will still exist in legacy-srgb. To solve the problem, implementations must color-correct CSS colors, then by extension, legacy-srgb mode will be in the true sRGB color space by virtue of the color matching rules outlined above. When that becomes the case, images drawn to canvases will be color corrected to sRGB, which solves the problem. There is resistance to adopting this model because going through an sRGB intermediate is lossy compared to directly color correcting images for the display in a single pass (may cause banding and gamut clipping). This feature proposal mitigates the lossiness argument thanks to the linear-rec-2020 option.

==== Implementation notes ====
* Because float16 arithmetic is supported by many GPUs, but not by CPUs, implementations should probably opt to not support rec-2020 on hardware that does not provide any native support.
* When available, the srgb colorspace should use GPU API extension for SRGB support. This will streamline the conversion overhead for performing filtering and compositing in linear space.

==== Adoption ====
Lack of color management and color interoperability is a longstanding complaint about the canvas API.
Authors of games and imaging apps are expected to be enthusiastic adopters.

==== History ====
This proposal was originally incubated in the Khronos 3D Web group, with the participation of engineers from Google, Microsoft, Apple, Nvidia, and others.

[[Category:Proposals]]

CanvasColorSpace

2016-05-10T21:20:52Z

Junov:

:Color managing canvas contents

== Use Case Description ==
* Contents displayed through a canvas element should be color managed in order to minimize differences in appearance across browsers and display devices. Improving color fidelity matters a lot for artistic uses (e.g. photo and paint apps) and for e-commerce (product presentation).
* Canvases should be able to take advantage of the full color gamut of the display device.
* Creative apps that do image manipulation generally prefer compositing, filtering and interpolation calculations to be performed in a linear color space.

=== Current Limitations ===
* The color space of canvases is undefined in the current specification.
* The bit-depth of canvases is currently fixed to 8 bits per component, which is below the capabilities of some monitors. Monitors with higher contrast ratios require more bits per component to avoid banding.

=== Current Usage and Workarounds ===
The lack of color space interoperability is hard to work around. With some browser implementations that color correct images drawn to canvases by applying the display profile, apps that want to use canvases for color corrected image processing are stuck doing convoluted workarounds, such as:
* reverse-engineer the display profile by drawing test pattern images to the canvas and inspecting the color corrected result via getImageData
* bypass CanvasRenderingContext2D.drawImage() and use image decoders implemented in JavaScript to extract raw image data that was not tainted by the browser's color correction behavior.''

An aspect of current implementations that is interoperable is that colors match between CSS/HTML and canvases:
* A color value used as a canvas drawing style will have the same appearance as if the same color value were used as a CSS style
* An image resource drawn to a canvas element will have the same appearance as if it were displayed as the replaced content of an HTML element or used as a CSS style value.

This color matching behavior needs to be preserved to avoid breaking pre-existing content.

Some implementations convert images drawn to canvases to the sRGB color space. This has the advantage of making the color correction behavior device independent, but it clamps the gamuts of the rendered content to the sRGB gamut, which is significantly narrower than the gamuts of some current consumer devices.

=== Requests for this Feature ===

* <cite>[https://github.com/whatwg/html/issues/299]</cite> <blockquote><p>Allow 2dcontexts to use deeper color buffers</p></blockquote>
* <cite>[https://bugs.chromium.org/p/chromium/issues/detail?id=425935]</cite> <blockquote><p>Wrong color profile with 2D canvas</p></blockquote>
* Engineers from the Google Photos, Maps and Sheets teams have expressed a desire for canvases to become color managed. Particularly for the use case of resizing an imaging, using a canvas, prior to uploading it to the server, to save bandwidth. The problem is that the images retrieved from a canvas are in an undefined color space and no color space information is encoded by toDataURL or toBlob.

== Proposed Solutions ==

=== Proposed solution: CanvasColorSpace ===
:Add a canvas color space creation parameter that allows user code to chose between backwards compatible behavior and color managed behaviors The same color space option would exist in the ImageData and ImageBitmap interfaces.

==== Processing Model ====

===== The color-space canvas creation parameter =====

IDL:
<code><pre>
enum CanvasColorSpace {
"legacy-srgb",
"srgb",
"linear-rec-2020",
"optimal"
};

dictionary CanvasRenderingContext2DSettings {
boolean alpha = true;
CanvasColorSpace color-space = "legacy-srgb";
};
</pre></code>

Example:
<code><pre>
canvas.getContext('2d', { color-space: "srgb"})
</pre></code>

===== The legacy-srgb color space =====

* Assures backwards compatible behavior
* Guarantees color matching with CSS and HTML content
* Color management behavior is implementation specific, may not use strict sRGB space, but is expected to be near sRGB. For example, could be display referred color space.
* toDataURL/toBlob produce resources with no color profile (backwards compat)
* Image resources with no color profile are never color corrected (backwards compat). This rule and the previous one allow for lossless toDataURL/drawImage round trips, which is a significant use case.

===== The srgb color space =====

* May break color matching with CSS on implementations that do not color-manage CSS.
* 8 bit unsigned integers per color component.
* All content drawn into the canvas must be color corrected to sRGB
* displayed canvases must be color corrected for the display if a display color profile is available. This color correction happens downstream at the compositing stage, and has no script-visible side-effects.
* Compositing, filtering and interpolation operations must perform all arithmetic in '''linear''' sRGB space.
* toDataURL/toBlob produce resources tagged as being in the sRGB colorspace
* Images with no color profile, when drawn to the canvas, are assumed to already be in the sRGB color space.

===== The rec-2020 color space =====

* Color space provided for wide gamut support without increasing memory cost.
* 8 bit unsigned integers per color component.
* All content drawn into the canvas must be color corrected to the rec-2020 color space.
* Displayed canvases must be color corrected for the display if a display color profile is available. This color correction happens downstream at the compositing stage, and has no script-visible side-effects. If there is no display color profile, the user agent should assume the display uses sRGB.
* Compositing, filtering and interpolation operations must perform all arithmetic in '''linear''' rec-2020 space.
* toDataURL/toBlob produce image resources in the rec-2020 colorspace.
* Images with no color profile, when drawn to the canvas, are assumed to be in the sRGB color space, and must therefore be converted from sRGB to rec-2020.

===== The linear-rec-2020 color space =====
* Color space provided for wide gamut and high dynamic range rendering.
* User agents may decide not to support the mode, based on host machine capabilities
* Uses 16-bit floating point representation.
* The color space corresponds to ITU-R Recommendation BT.2020, '''without gamma compression'''.
* toDataURL/toBlob convert image data to the rec-2020 color space (with gamma), and produce image resources with at least 12 bits per color component, if the format supports it. Thus, in the case of the png format, which supports 8 or 16 bits per component, 16bpc would be used.
* Image with no color profile, when drawn to the canvas, are assumed to be in the sRGB color space, and are converted to linear-rec-2020 for the purpose of the draw.

===== The optimal color space =====
The "optimal" option lets the user agent decide which space is optimal for the current display device based on the device's capabilities and color profile characteristics.
* This option may not select "legacy-srgb"
* Graphics devices with color gamuts that extend significantly beyond the sRGB color space should cause the UA to favor rec-2020; and linear linear-rec-2020 should be used for displays with high dynamic ranges.
* Graphics devices that would not produce noticeably higher quality visual results in rec-2020 or linear-rec-2020 should cause the UA to favor "srgb".

===== Feature detection =====

Rendering context objects are to expose a new "settings" attribute, which represents the settings that were successfully applied at context creation time.

Note: An alternative approach that was considered was to augment the probablySupportsContext() API by making it check the second argument. That approach is difficult to consolidate with how dictionary argument are meant to work, where unsupported entries are just ignored.

===== ImageBitmap =====

ImageBitmap objects are augmented to have an internal color space attribute of type CanvasColorSpace. The colorSpaceConversion creation attribute is to be augmented with new enum values for coercing conversions to a specific CanvasColorSpace at creation time.

===== ImageData =====

IDL
<code><pre>
typedef (Uint8ClampedArray or Float32Array) ImageDataArray;

[Constructor(unsigned long sw, unsigned long sh, optional CanvasColorSpace colorSpace = "legacy-srgb"),
Constructor(ImageDataArray data, unsigned long sw, optional unsigned long sh, optional CanvasColorSpace colorSpace),
Exposed=(Window,Worker)]
interface ImageData {
readonly attribute unsigned long width;
readonly attribute unsigned long height;
readonly attribute ImageDataArray data;
readonly attribute CanvasColorSpace colorSpace;
};
</pre></code>

* <code>data</code> is a Uint8ClampedArray if colorSpace is "srgb" or "legacy-srgb"
* <code>data</code> is a Float32Array if colorSpace is "linear-rec-2020"
* getImageData() produces an ImageData object in the same color space as the source canvas
* putImageData() performs a color space conversion to the color space of the destination canvas.

==== Limitations ====
* No support for arbitrary color spaces and bit depth. This capability could be added in the future. The current proposal attempts to solve the problem with a minimal API surface, and keeps the implementation scope reasonable. The extensible design will allow us to extend the capabilities in the future if necessary. The rec-2020 space was chosen for its very wide gamut and its non-virtual primary colors, which strikes a balance that is deemed practical.
* toDataURL is lossy when us on a canvas that is in the linear-rec-2020 space. Possible future improvements could solve or mitigate this issue by adding more file formats or adding options to specify the resource color space.
* ImageData uses float32, which is inefficient due to memory consumption and necessary conversion operations. Float32 was chosen because it is convenient for manipulation (e.g. image processing) due to its native support in JavaScript (and current CPUs). A possible extension would be to add and option for rec-2020 content to be encoded as float16s packed into Uint16 values.

==== Security and privacy issues ====
Some current implementations of CanvasRenderingContext2D color correct image resources for the display as they are drawn to the canvas. In other words, the canvas is in output referred color space. This is a known fingerprinting vulnerability since it exposes the user's display's color profile to scripts via getImageData. The current proposal does not solve the fingerprinting issue because it will still exist in legacy-srgb. To solve the problem, implementations must color-correct CSS colors, then by extension, legacy-srgb mode will be in the true sRGB color space by virtue of the color matching rules outlined above. When that becomes the case, images drawn to canvases will be color corrected to sRGB, which solves the problem. There is resistance to adopting this model because going through an sRGB intermediate is lossy compared to directly color correcting images for the display in a single pass (may cause banding and gamut clipping). This feature proposal mitigates the lossiness argument thanks to the linear-rec-2020 option.

==== Implementation notes ====
* Because float16 arithmetic is supported by many GPUs, but not by CPUs, implementations should probably opt to not support rec-2020 on hardware that does not provide any native support.
* When available, the srgb colorspace should use GPU API extension for SRGB support. This will streamline the conversion overhead for performing filtering and compositing in linear space.

==== Adoption ====
Lack of color management and color interoperability is a longstanding complaint about the canvas API.
Authors of games and imaging apps are expected to be enthusiastic adopters.

==== History ====
This proposal was originally incubated in the Khronos 3D Web group, with the participation of engineers from Google, Microsoft, Apple, Nvidia, and others.

[[Category:Proposals]]

CanvasColorSpace

2016-05-09T15:05:35Z

Junov:

:Color managing canvas contents

== Use Case Description ==
* Contents displayed through a canvas element should be color managed in order to minimize differences in appearance across browsers and display devices. Improving color fidelity matters a lot for artistic uses (e.g. photo and paint apps) and for e-commerce (product presentation).
* Canvases should be able to take advantage of the full color gamut of the display device.
* Creative apps that do image manipulation generally prefer compositing, filtering and interpolation calculations to be performed in a linear color space.

=== Current Limitations ===
* The color space of canvases is undefined in the current specification.
* The bit-depth of canvases is currently fixed to 8 bits per component, which is below the capabilities of some monitors. Monitors with higher contrast ratios require more bits per component to avoid banding.

=== Current Usage and Workarounds ===
The lack of color space interoperability is hard to work around. With some browser implementations that color correct images drawn to canvases by applying the display profile, apps that want to use canvases for color corrected image processing are stuck doing convoluted workarounds, such as:
* reverse-engineer the display profile by drawing test pattern images to the canvas and inspecting the color corrected result via getImageData
* bypass CanvasRenderingContext2D.drawImage() and use image decoders implemented in JavaScript to extract raw image data that was not tainted by the browser's color correction behavior.''

An aspect of current implementations that is interoperable is that colors match between CSS/HTML and canvases:
* A color value used as a canvas drawing style will have the same appearance as if the same color value were used as a CSS style
* An image resource drawn to a canvas element will have the same appearance as if it were displayed as the replaced content of an HTML element or used as a CSS style value.

This color matching behavior needs to be preserved to avoid breaking pre-existing content.

Some implementations convert images drawn to canvases to the sRGB color space. This has the advantage of making the color correction behavior device independent, but it clamps the gamuts of the rendered content to the sRGB gamut, which is significantly narrower than the gamuts of some current consumer devices.

=== Requests for this Feature ===

* <cite>[https://github.com/whatwg/html/issues/299]</cite> <blockquote><p>Allow 2dcontexts to use deeper color buffers</p></blockquote>
* <cite>[https://bugs.chromium.org/p/chromium/issues/detail?id=425935]</cite> <blockquote><p>Wrong color profile with 2D canvas</p></blockquote>
* Engineers from the Google Photos, Maps and Sheets teams have expressed a desire for canvases to become color managed. Particularly for the use case of resizing an imaging, using a canvas, prior to uploading it to the server, to save bandwidth. The problem is that the images retrieved from a canvas are in an undefined color space and no color space information is encoded by toDataURL or toBlob.

== Proposed Solutions ==

=== Proposed solution: CanvasColorSpace ===
:Add a canvas color space creation parameter that allows user code to chose between backwards compatible behavior and color managed behaviors The same color space option would exist in the ImageData and ImageBitmap interfaces.

==== Processing Model ====

===== The color-space canvas creation parameter =====

IDL:
<code><pre>
enum CanvasColorSpace {
"legacy-srgb",
"srgb",
"linear-rec-2020",
"optimal"
};

dictionary CanvasRenderingContext2DSettings {
boolean alpha = true;
CanvasColorSpace color-space = "legacy-srgb";
};
</pre></code>

Example:
<code><pre>
canvas.getContext('2d', { color-space: "srgb"})
</pre></code>

===== The legacy-srgb color space =====

* Assures backwards compatible behavior
* Guarantees color matching with CSS and HTML content
* Color management behavior is implementation specific, may not use strict sRGB space, but is expected to be near sRGB. For example, could be display referred color space.
* toDataURL/toBlob produce resources with no color profile (backwards compat)
* Image resources with no color profile are never color corrected (backwards compat). This rule and the previous one allow for lossless toDataURL/drawImage round trips, which is a significant use case.

===== The srgb color space =====

* May break color matching with CSS on implementations that do not color-manage CSS.
* 8 bit unsigned integers per color component.
* All content drawn into the canvas must be color corrected to sRGB
* displayed canvases must be color corrected for the display if a display color profile is available. This color correction happens downstream at the compositing stage, and has no script-visible side-effects.
* Compositing, filtering and interpolation operations must perform all arithmetic in '''linear''' sRGB space.
* toDataURL/toBlob produce resources tagged as being in the sRGB colorspace
* Images with no color profile, when drawn to the canvas, are assumed to already be in the sRGB color space.

===== The linear-rec-2020 color space =====
* Color space provided for wide gamut and high dynamic range rendering.
* User agents may decide not to support the mode, based on host machine capabilities
* Uses 16-bit floating point representation.
* The color space corresponds to ITU-R Recommendation BT.2020, '''without gamma compression'''.
* toDataURL/toBlob convert image data to the rec-2020 color space (with gamma), and produce image resources with at least 12 bits per color component, if the format supports it. Thus, in the case of the png format, which supports 8 or 16 bits per component, 16bpc would be used.
* Image with no color profile, when drawn to the canvas, are assumed to be in the sRGB color space, and are converted to linear-rec-2020 for the purpose of the draw.

===== The optimal color space =====
The "optimal" option lets the user agent decide which space is optimal for the current display device based on the device's capabilities and color profile characteristics.
* This option may not select "legacy-srgb"
* Graphics devices with color gamuts and/or contrast ratios that extend significantly beyond the sRGB color space should cause the UA to favor linear-rec-2020 to avoid undue gamut clipping and/or banding.
* Graphics devices that would not produce noticeably higher quality visual results in the linear-rec-2020 should cause the UA to favor "srgb" to save on memory consumption

===== Feature detection =====

Rendering context objects are to expose a new "settings" attribute, which represents the settings that were successfully applied at context creation time.

Note: An alternative approach that was considered was to augment the probablySupportsContext() API by making it check the second argument. That approach is difficult to consolidate with how dictionary argument are meant to work, where unsupported entries are just ignored.

===== ImageBitmap =====

ImageBitmap objects are augmented to have an internal color space attribute of type CanvasColorSpace. The colorSpaceConversion creation attribute is to be augmented with new enum values for coercing conversions to a specific CanvasColorSpace at creation time.

===== ImageData =====

IDL
<code><pre>
typedef (Uint8ClampedArray or Float32Array) ImageDataArray;

[Constructor(unsigned long sw, unsigned long sh, optional CanvasColorSpace colorSpace = "legacy-srgb"),
Constructor(ImageDataArray data, unsigned long sw, optional unsigned long sh, optional CanvasColorSpace colorSpace),
Exposed=(Window,Worker)]
interface ImageData {
readonly attribute unsigned long width;
readonly attribute unsigned long height;
readonly attribute ImageDataArray data;
readonly attribute CanvasColorSpace colorSpace;
};
</pre></code>

* <code>data</code> is a Uint8ClampedArray if colorSpace is "srgb" or "legacy-srgb"
* <code>data</code> is a Float32Array if colorSpace is "linear-rec-2020"
* getImageData() produces an ImageData object in the same color space as the source canvas
* putImageData() performs a color space conversion to the color space of the destination canvas.

==== Limitations ====
* No support for arbitrary color spaces and bit depth. This capability could be added in the future. The current proposal attempts to solve the problem with a minimal API surface, and keeps the implementation scope reasonable. The extensible design will allow us to extend the capabilities in the future if necessary. The rec-2020 space was chosen for its very wide gamut and its non-virtual primary colors, which strikes a balance that is deemed practical.
* toDataURL is lossy when us on a canvas that is in the linear-rec-2020 space. Possible future improvements could solve or mitigate this issue by adding more file formats or adding options to specify the resource color space.
* ImageData uses float32, which is inefficient due to memory consumption and necessary conversion operations. Float32 was chosen because it is convenient for manipulation (e.g. image processing) due to its native support in JavaScript (and current CPUs). A possible extension would be to add and option for rec-2020 content to be encoded as float16s packed into Uint16 values.

==== Security and privacy issues ====
Some current implementations of CanvasRenderingContext2D color correct image resources for the display as they are drawn to the canvas. In other words, the canvas is in output referred color space. This is a known fingerprinting vulnerability since it exposes the user's display's color profile to scripts via getImageData. The current proposal does not solve the fingerprinting issue because it will still exist in legacy-srgb. To solve the problem, implementations must color-correct CSS colors, then by extension, legacy-srgb mode will be in the true sRGB color space by virtue of the color matching rules outlined above. When that becomes the case, images drawn to canvases will be color corrected to sRGB, which solves the problem. There is resistance to adopting this model because going through an sRGB intermediate is lossy compared to directly color correcting images for the display in a single pass (may cause banding and gamut clipping). This feature proposal mitigates the lossiness argument thanks to the linear-rec-2020 option.

==== Implementation notes ====
* Because float16 arithmetic is supported by many GPUs, but not by CPUs, implementations should probably opt to not support rec-2020 on hardware that does not provide any native support.
* When available, the srgb colorspace should use GPU API extension for SRGB support. This will streamline the conversion overhead for performing filtering and compositing in linear space.

==== Adoption ====
Lack of color management and color interoperability is a longstanding complaint about the canvas API.
Authors of games and imaging apps are expected to be enthusiastic adopters.

==== History ====
This proposal was originally incubated in the Khronos 3D Web group, with the participation of engineers from Google, Microsoft, Apple, Nvidia, and others.

[[Category:Proposals]]

CanvasColorSpace

2016-05-07T15:32:27Z

Junov: /* The linear-rec-2020 color space */

:Color managing canvas contents

== Use Case Description ==
* Contents displayed through a canvas element should be color managed in order to minimize differences in appearance across browsers and display devices. Improving color fidelity matters a lot for artistic uses (e.g. photo and paint apps) and for e-commerce (product presentation).
* Canvases should be able to take advantage of the full color gamut of the display device.
* Creative apps that do image manipulation generally prefer compositing, filtering and interpolation calculations to be performed in a linear color space.

=== Current Limitations ===
* The color space of canvases is undefined in the current specification.
* The bit-depth of canvases is currently fixed to 8 bits per component, which is below the capabilities of some monitors. Monitors with higher contrast ratios require more bits per component to avoid banding.

=== Current Usage and Workarounds ===
The lack of color space interoperability is hard to work around. With some browser implementations that color correct images drawn to canvases by applying the display profile, apps that want to use canvases for color corrected image processing are stuck doing convoluted workarounds, such as:
* reverse-engineer the display profile by drawing test pattern images to the canvas and inspecting the color corrected result via getImageData
* bypass CanvasRenderingContext2D.drawImage() and use image decoders implemented in JavaScript to extract raw image data that was not tainted by the browser's color correction behavior.''

An aspect of current implementations that is interoperable is that colors match between CSS/HTML and canvases:
* A color value used as a canvas drawing style will have the same appearance as if the same color value were used as a CSS style
* An image resource drawn to a canvas element will have the same appearance as if it were displayed as the replaced content of an HTML element or used as a CSS style value.

This color matching behavior needs to be preserved to avoid breaking pre-existing content.

Some implementations convert images drawn to canvases to the sRGB color space. This has the advantage of making the color correction behavior device independent, but it clamps the gamuts of the rendered content to the sRGB gamut, which is significantly narrower than the gamuts of some current consumer devices.

=== Requests for this Feature ===

* <cite>[https://github.com/whatwg/html/issues/299]</cite> <blockquote><p>Allow 2dcontexts to use deeper color buffers</p></blockquote>
* <cite>[https://bugs.chromium.org/p/chromium/issues/detail?id=425935]</cite> <blockquote><p>Wrong color profile with 2D canvas</p></blockquote>
* Engineers from the Google Photos, Maps and Sheets teams have expressed a desire for canvases to become color managed. Particularly for the use case of resizing an imaging, using a canvas, prior to uploading it to the server, to save bandwidth. The problem is that the images retrieved from a canvas are in an undefined color space and no color space information is encoded by toDataURL or toBlob.

== Proposed Solutions ==

=== Proposed solution: CanvasColorSpace ===
:Add a canvas color space creation parameter that allows user code to chose between backwards compatible behavior and color managed behaviors The same color space option would exist in the ImageData and ImageBitmap interfaces.

==== Processing Model ====

===== The color-space canvas creation parameter =====

IDL:
<code><pre>
enum CanvasColorSpace {
"legacy-srgb",
"srgb",
"linear-rec-2020",
"optimal"
};

dictionary CanvasRenderingContext2DSettings {
boolean alpha = true;
CanvasColorSpace color-space = "legacy-srgb";
};
</pre></code>

Example:
<code><pre>
canvas.getContext('2d', { color-space: "srgb"})
</pre></code>

===== The legacy-srgb color space =====

* Assures backwards compatible behavior
* Guarantees color matching with CSS and HTML content
* Color management behavior is implementation specific, may not use strict sRGB space, but is expected to be near sRGB. For example, could be display referred color space.
* toDataURL/toBlob produce resources with no color profile (backwards compat)
* Image resources with no color profile are never color corrected (backwards compat). This rule and the previous one allow for lossless toDataURL/drawImage round trips, which is a significant use case.

===== The srgb color space =====

* May break color matching with CSS on implementations that do not color-manage CSS.
* 8 bit unsigned integers per color component.
* All content drawn into the canvas must be color corrected to sRGB
* displayed canvases must be color corrected for the display if a display color profile is available. This color correction happens downstream at the compositing stage, and has no script-visible side-effects.
* Compositing, filtering and interpolation operations must perform all arithmetic in '''linear''' sRGB space.
* toDataURL/toBlob produce resources tagged as being in the sRGB colorspace
* Images with no color profile, when drawn to the canvas, are assumed to already be in the sRGB color space.

===== The linear-rec-2020 color space =====
* Color space provided for wide gamut and high dynamic range rendering.
* User agents may decide not to support the mode, based on host machine capabilities
* Uses 16-bit floating point representation.
* The color space corresponds to ITU-R Recommendation BT.2020, '''without gamma compression'''.
* toDataURL/toBlob convert image data to sRGB and produce image resources tagged as being in the sRGB color space.
* Image with no color profile, when drawn to the canvas, are assumed to be in the sRGB color space, and are converted to linear-rec-2020 for the purpose of the draw.

===== The optimal color space =====
The "optimal" option lets the user agent decide which space is optimal for the current display device based on the device's capabilities and color profile characteristics.
* This option may not select "legacy-srgb"
* Graphics devices with color gamuts and/or contrast ratios that extend significantly beyond the sRGB color space should cause the UA to favor linear-rec-2020 to avoid undue gamut clipping and/or banding.
* Graphics devices that would not produce noticeably higher quality visual results in the linear-rec-2020 should cause the UA to favor "srgb" to save on memory consumption

===== Feature detection =====

Rendering context objects are to expose a new "settings" attribute, which represents the settings that were successfully applied at context creation time.

Note: An alternative approach that was considered was to augment the probablySupportsContext() API by making it check the second argument. That approach is difficult to consolidate with how dictionary argument are meant to work, where unsupported entries are just ignored.

===== ImageBitmap =====

ImageBitmap objects are augmented to have an internal color space attribute of type CanvasColorSpace. The colorSpaceConversion creation attribute is to be augmented with new enum values for coercing conversions to a specific CanvasColorSpace at creation time.

===== ImageData =====

IDL
<code><pre>
typedef (Uint8ClampedArray or Float32Array) ImageDataArray;

[Constructor(unsigned long sw, unsigned long sh, optional CanvasColorSpace colorSpace = "legacy-srgb"),
Constructor(ImageDataArray data, unsigned long sw, optional unsigned long sh, optional CanvasColorSpace colorSpace),
Exposed=(Window,Worker)]
interface ImageData {
readonly attribute unsigned long width;
readonly attribute unsigned long height;
readonly attribute ImageDataArray data;
readonly attribute CanvasColorSpace colorSpace;
};
</pre></code>

* <code>data</code> is a Uint8ClampedArray if colorSpace is "srgb" or "legacy-srgb"
* <code>data</code> is a Float32Array if colorSpace is "linear-rec-2020"
* getImageData() produces an ImageData object in the same color space as the source canvas
* putImageData() performs a color space conversion to the color space of the destination canvas.

==== Limitations ====
* No support for arbitrary color spaces and bit depth. This capability could be added in the future. The current proposal attempts to solve the problem with a minimal API surface, and keeps the implementation scope reasonable. The extensible design will allow us to extend the capabilities in the future if necessary. The rec-2020 space was chosen for its very wide gamut and its non-virtual primary colors, which strikes a balance that is deemed practical.
* toDataURL is lossy when us on a canvas that is in the linear-rec-2020 space. Possible future improvements could solve or mitigate this issue by adding more file formats or adding options to specify the resource color space.
* ImageData uses float32, which is inefficient due to memory consumption and necessary conversion operations. Float32 was chosen because it is convenient for manipulation (e.g. image processing) due to its native support in JavaScript (and current CPUs). A possible extension would be to add and option for rec-2020 content to be encoded as float16s packed into Uint16 values.

==== Security and privacy issues ====
Some current implementations of CanvasRenderingContext2D color correct image resources for the display as they are drawn to the canvas. In other words, the canvas is in output referred color space. This is a known fingerprinting vulnerability since it exposes the user's display's color profile to scripts via getImageData. The current proposal does not solve the fingerprinting issue because it will still exist in legacy-srgb. To solve the problem, implementations must color-correct CSS colors, then by extension, legacy-srgb mode will be in the true sRGB color space by virtue of the color matching rules outlined above. When that becomes the case, images drawn to canvases will be color corrected to sRGB, which solves the problem. There is resistance to adopting this model because going through an sRGB intermediate is lossy compared to directly color correcting images for the display in a single pass (may cause banding and gamut clipping). This feature proposal mitigates the lossiness argument thanks to the linear-rec-2020 option.

==== Implementation notes ====
* Because float16 arithmetic is supported by many GPUs, but not by CPUs, implementations should probably opt to not support rec-2020 on hardware that does not provide any native support.
* When available, the srgb colorspace should use GPU API extension for SRGB support. This will streamline the conversion overhead for performing filtering and compositing in linear space.

==== Adoption ====
Lack of color management and color interoperability is a longstanding complaint about the canvas API.
Authors of games and imaging apps are expected to be enthusiastic adopters.

==== History ====
This proposal was originally incubated in the Khronos 3D Web group, with the participation of engineers from Google, Microsoft, Apple, Nvidia, and others.

[[Category:Proposals]]

CanvasColorSpace

2016-05-07T14:12:57Z

Junov:

:Color managing canvas contents

== Use Case Description ==
* Contents displayed through a canvas element should be color managed in order to minimize differences in appearance across browsers and display devices. Improving color fidelity matters a lot for artistic uses (e.g. photo and paint apps) and for e-commerce (product presentation).
* Canvases should be able to take advantage of the full color gamut of the display device.
* Creative apps that do image manipulation generally prefer compositing, filtering and interpolation calculations to be performed in a linear color space.

=== Current Limitations ===
* The color space of canvases is undefined in the current specification.
* The bit-depth of canvases is currently fixed to 8 bits per component, which is below the capabilities of some monitors. Monitors with higher contrast ratios require more bits per component to avoid banding.

=== Current Usage and Workarounds ===
The lack of color space interoperability is hard to work around. With some browser implementations that color correct images drawn to canvases by applying the display profile, apps that want to use canvases for color corrected image processing are stuck doing convoluted workarounds, such as:
* reverse-engineer the display profile by drawing test pattern images to the canvas and inspecting the color corrected result via getImageData
* bypass CanvasRenderingContext2D.drawImage() and use image decoders implemented in JavaScript to extract raw image data that was not tainted by the browser's color correction behavior.''

An aspect of current implementations that is interoperable is that colors match between CSS/HTML and canvases:
* A color value used as a canvas drawing style will have the same appearance as if the same color value were used as a CSS style
* An image resource drawn to a canvas element will have the same appearance as if it were displayed as the replaced content of an HTML element or used as a CSS style value.

This color matching behavior needs to be preserved to avoid breaking pre-existing content.

Some implementations convert images drawn to canvases to the sRGB color space. This has the advantage of making the color correction behavior device independent, but it clamps the gamuts of the rendered content to the sRGB gamut, which is significantly narrower than the gamuts of some current consumer devices.

=== Requests for this Feature ===

* <cite>[https://github.com/whatwg/html/issues/299]</cite> <blockquote><p>Allow 2dcontexts to use deeper color buffers</p></blockquote>
* <cite>[https://bugs.chromium.org/p/chromium/issues/detail?id=425935]</cite> <blockquote><p>Wrong color profile with 2D canvas</p></blockquote>
* Engineers from the Google Photos, Maps and Sheets teams have expressed a desire for canvases to become color managed. Particularly for the use case of resizing an imaging, using a canvas, prior to uploading it to the server, to save bandwidth. The problem is that the images retrieved from a canvas are in an undefined color space and no color space information is encoded by toDataURL or toBlob.

== Proposed Solutions ==

=== Proposed solution: CanvasColorSpace ===
:Add a canvas color space creation parameter that allows user code to chose between backwards compatible behavior and color managed behaviors The same color space option would exist in the ImageData and ImageBitmap interfaces.

==== Processing Model ====

===== The color-space canvas creation parameter =====

IDL:
<code><pre>
enum CanvasColorSpace {
"legacy-srgb",
"srgb",
"linear-rec-2020",
"optimal"
};

dictionary CanvasRenderingContext2DSettings {
boolean alpha = true;
CanvasColorSpace color-space = "legacy-srgb";
};
</pre></code>

Example:
<code><pre>
canvas.getContext('2d', { color-space: "srgb"})
</pre></code>

===== The legacy-srgb color space =====

* Assures backwards compatible behavior
* Guarantees color matching with CSS and HTML content
* Color management behavior is implementation specific, may not use strict sRGB space, but is expected to be near sRGB. For example, could be display referred color space.
* toDataURL/toBlob produce resources with no color profile (backwards compat)
* Image resources with no color profile are never color corrected (backwards compat). This rule and the previous one allow for lossless toDataURL/drawImage round trips, which is a significant use case.

===== The srgb color space =====

* May break color matching with CSS on implementations that do not color-manage CSS.
* 8 bit unsigned integers per color component.
* All content drawn into the canvas must be color corrected to sRGB
* displayed canvases must be color corrected for the display if a display color profile is available. This color correction happens downstream at the compositing stage, and has no script-visible side-effects.
* Compositing, filtering and interpolation operations must perform all arithmetic in '''linear''' sRGB space.
* toDataURL/toBlob produce resources tagged as being in the sRGB colorspace
* Images with no color profile, when drawn to the canvas, are assumed to already be in the sRGB color space.

===== The linear-rec-2020 color space =====
* Space provide for wide gamut and high dynamic range rendering
* User agents may decide not to support the mode, based on host machine capabilities
* Uses 16-bit floating point representation.
* The color space corresponds to ITU-R Recommendation BT.2020, '''without gamma compression'''.
* toDataURL/toBlob convert image data to sRGB and produce image resources tagged as being in the sRGB color space.
* Image with no color profile, when drawn to the canvas, are assumed to be in the sRGB color space, and are converted to linear-rec-2020 for the purpose of the draw.

===== The optimal color space =====
The "optimal" option lets the user agent decide which space is optimal for the current display device based on the device's capabilities and color profile characteristics.
* This option may not select "legacy-srgb"
* Graphics devices with color gamuts and/or contrast ratios that extend significantly beyond the sRGB color space should cause the UA to favor linear-rec-2020 to avoid undue gamut clipping and/or banding.
* Graphics devices that would not produce noticeably higher quality visual results in the linear-rec-2020 should cause the UA to favor "srgb" to save on memory consumption

===== Feature detection =====

Rendering context objects are to expose a new "settings" attribute, which represents the settings that were successfully applied at context creation time.

Note: An alternative approach that was considered was to augment the probablySupportsContext() API by making it check the second argument. That approach is difficult to consolidate with how dictionary argument are meant to work, where unsupported entries are just ignored.

===== ImageBitmap =====

ImageBitmap objects are augmented to have an internal color space attribute of type CanvasColorSpace. The colorSpaceConversion creation attribute is to be augmented with new enum values for coercing conversions to a specific CanvasColorSpace at creation time.

===== ImageData =====

IDL
<code><pre>
typedef (Uint8ClampedArray or Float32Array) ImageDataArray;

[Constructor(unsigned long sw, unsigned long sh, optional CanvasColorSpace colorSpace = "legacy-srgb"),
Constructor(ImageDataArray data, unsigned long sw, optional unsigned long sh, optional CanvasColorSpace colorSpace),
Exposed=(Window,Worker)]
interface ImageData {
readonly attribute unsigned long width;
readonly attribute unsigned long height;
readonly attribute ImageDataArray data;
readonly attribute CanvasColorSpace colorSpace;
};
</pre></code>

* <code>data</code> is a Uint8ClampedArray if colorSpace is "srgb" or "legacy-srgb"
* <code>data</code> is a Float32Array if colorSpace is "linear-rec-2020"
* getImageData() produces an ImageData object in the same color space as the source canvas
* putImageData() performs a color space conversion to the color space of the destination canvas.

==== Limitations ====
* No support for arbitrary color spaces and bit depth. This capability could be added in the future. The current proposal attempts to solve the problem with a minimal API surface, and keeps the implementation scope reasonable. The extensible design will allow us to extend the capabilities in the future if necessary. The rec-2020 space was chosen for its very wide gamut and its non-virtual primary colors, which strikes a balance that is deemed practical.
* toDataURL is lossy when us on a canvas that is in the linear-rec-2020 space. Possible future improvements could solve or mitigate this issue by adding more file formats or adding options to specify the resource color space.
* ImageData uses float32, which is inefficient due to memory consumption and necessary conversion operations. Float32 was chosen because it is convenient for manipulation (e.g. image processing) due to its native support in JavaScript (and current CPUs). A possible extension would be to add and option for rec-2020 content to be encoded as float16s packed into Uint16 values.

==== Security and privacy issues ====
Some current implementations of CanvasRenderingContext2D color correct image resources for the display as they are drawn to the canvas. In other words, the canvas is in output referred color space. This is a known fingerprinting vulnerability since it exposes the user's display's color profile to scripts via getImageData. The current proposal does not solve the fingerprinting issue because it will still exist in legacy-srgb. To solve the problem, implementations must color-correct CSS colors, then by extension, legacy-srgb mode will be in the true sRGB color space by virtue of the color matching rules outlined above. When that becomes the case, images drawn to canvases will be color corrected to sRGB, which solves the problem. There is resistance to adopting this model because going through an sRGB intermediate is lossy compared to directly color correcting images for the display in a single pass (may cause banding and gamut clipping). This feature proposal mitigates the lossiness argument thanks to the linear-rec-2020 option.

==== Implementation notes ====
* Because float16 arithmetic is supported by many GPUs, but not by CPUs, implementations should probably opt to not support rec-2020 on hardware that does not provide any native support.
* When available, the srgb colorspace should use GPU API extension for SRGB support. This will streamline the conversion overhead for performing filtering and compositing in linear space.

==== Adoption ====
Lack of color management and color interoperability is a longstanding complaint about the canvas API.
Authors of games and imaging apps are expected to be enthusiastic adopters.

==== History ====
This proposal was originally incubated in the Khronos 3D Web group, with the participation of engineers from Google, Microsoft, Apple, Nvidia, and others.

[[Category:Proposals]]

CanvasColorSpace

2016-05-07T13:27:36Z

Junov:

:Color managing canvas contents

== Use Case Description ==
* Contents displayed through a canvas element should be color managed in order to minimize differences in appearance across browsers and display devices. Improving color fidelity matters a lot for artistic uses (e.g. photo and paint apps) and for e-commerce (product presentation).
* Canvases should be able to take advantage of the full color gamut of the display device.
* Creative apps that do image manipulation generally prefer compositing, filtering and interpolation calculations to be performed in a linear color space.

=== Current Limitations ===
* The color space of canvases is undefined in the current specification.
* The bit-depth of canvases is currently fixed to 8 bits per component, which is below the capabilities of some monitors. Monitors with higher contrast ratios require more bits per component to avoid banding.

=== Current Usage and Workarounds ===
The lack of color space interoperability is hard to work around. With some browser implementations that color correct images drawn to canvases by applying the display profile, apps that want to use canvases for color corrected image processing are stuck doing convoluted workarounds, such as:
* reverse-engineer the display profile by drawing test pattern images to the canvas and inspecting the color corrected result via getImageData
* bypass CanvasRenderingContext2D.drawImage() and use image decoders implemented in JavaScript to extract raw image data that was not tainted by the browser's color correction behavior.''

An aspect of current implementations that is interoperable is that colors match between CSS/HTML and canvases:
* A color value used as a canvas drawing style will have the same appearance as if the same color value were used as a CSS style
* An image resource drawn to a canvas element will have the same appearance as if it were displayed as the replaced content of an HTML element or used as a CSS style value.

This color matching behavior needs to be preserved to avoid breaking pre-existing content.

Some implementations convert images drawn to canvases to the sRGB color space. This has the advantage of making the color correction behavior device independent, but it clamps the gamuts of the rendered content to the sRGB gamut, which is significantly narrower than the gamuts of some current consumer devices.

=== Requests for this Feature ===

* <cite>[https://github.com/whatwg/html/issues/299]</cite> <blockquote><p>Allow 2dcontexts to use deeper color buffers</p></blockquote>
* <cite>[https://bugs.chromium.org/p/chromium/issues/detail?id=425935]</cite> <blockquote><p>Wrong color profile with 2D canvas</p></blockquote>
* Engineers from the Google Photos, Maps and Sheets teams have expressed a desire for canvases to become color managed. Particularly for the use case of resizing an imaging, using a canvas, prior to uploading it to the server, to save bandwidth. The problem is that the images retrieved from a canvas are in an undefined color space and no color space information is encoded by toDataURL or toBlob.

== Proposed Solutions ==

=== Proposed solution: CanvasColorSpace ===
:Add a canvas color space creation parameter that allows user code to chose between backwards compatible behavior and color managed behaviors The same color space option would exist in the ImageData and ImageBitmap interfaces.

==== Processing Model ====

===== The color-space canvas creation parameter =====

IDL:
<code><pre>
enum CanvasColorSpace {
"legacy-srgb",
"srgb",
"linear-rec-2020"
};

dictionary CanvasRenderingContext2DSettings {
boolean alpha = true;
CanvasColorSpace color-space = "legacy-srgb";
};
</pre></code>

Example:
<code><pre>
canvas.getContext('2d', { color-space: "srgb"})
</pre></code>

===== The legacy-srgb color space =====

* Assures backwards compatible behavior
* Guarantees color matching with CSS and HTML content
* Color management behavior is implementation specific, may not use strict sRGB space, but is expected to be near sRGB. For example, could be display referred color space.
* toDataURL/toBlob produce resources with no color profile (backwards compat)
* Image resources with no color profile are never color corrected (backwards compat). This rule and the previous one allow for lossless toDataURL/drawImage round trips, which is a significant use case.

===== The srgb color space =====

* May break color matching with CSS on implementations that do not color-manage CSS.
* 8 bit unsigned integers per color component.
* All content drawn into the canvas must be color corrected to sRGB
* displayed canvases must be color corrected for the display if a display color profile is available. This color correction happens downstream at the compositing stage, and has no script-visible side-effects.
* Compositing, filtering and interpolation operations must perform all arithmetic in '''linear''' sRGB space.
* toDataURL/toBlob produce resources tagged as being in the sRGB colorspace
* Images with no color profile, when drawn to the canvas, are assumed to already be in the sRGB color space.

===== The linear-rec-2020 color space =====

* Space provide for wide gamut and high dynamic range rendering
* User agents may decide not to support the mode, based on host machine capabilities
* Uses 16-bit floating point representation.
* The color space corresponds to ITU-R Recommendation BT.2020, '''without gamma compression'''.
* toDataURL/toBlob convert image data to sRGB and produce image resources tagged as being in the sRGB color space.
* Image with no color profile, when drawn to the canvas, are assumed to be in the sRGB color space, and are converted to linear-rec-2020 for the purpose of the draw.

===== Feature detection =====

Rendering context objects are to expose a new "settings" attribute, which represents the settings that were successfully applied at context creation time.

Note: An alternative approach that was considered was to augment the probablySupportsContext() API by making it check the second argument. That approach is difficult to consolidate with how dictionary argument are meant to work, where unsupported entries are just ignored.

===== ImageBitmap =====

ImageBitmap objects are augmented to have an internal color space attribute of type CanvasColorSpace. The colorSpaceConversion creation attribute is to be augmented with new enum values for coercing conversions to a specific CanvasColorSpace at creation time.

===== ImageData =====

IDL
<code><pre>
typedef (Uint8ClampedArray or Float32Array) ImageDataArray;

[Constructor(unsigned long sw, unsigned long sh, optional CanvasColorSpace colorSpace = "legacy-srgb"),
Constructor(ImageDataArray data, unsigned long sw, optional unsigned long sh, optional CanvasColorSpace colorSpace),
Exposed=(Window,Worker)]
interface ImageData {
readonly attribute unsigned long width;
readonly attribute unsigned long height;
readonly attribute ImageDataArray data;
readonly attribute CanvasColorSpace colorSpace;
};
</pre></code>

* <code>data</code> is a Uint8ClampedArray if colorSpace is "srgb" or "legacy-srgb"
* <code>data</code> is a Float32Array if colorSpace is "linear-rec-2020"
* getImageData() produces an ImageData object in the same color space as the source canvas
* putImageData() performs a color space conversion to the color space of the destination canvas.

==== Limitations ====
* No support for arbitrary color spaces and bit depth. This capability could be added in the future. The current proposal attempts to solve the problem with a minimal API surface, and keeps the implementation scope reasonable. The extensible design will allow us to extend the capabilities in the future if necessary. The rec-2020 space was chosen for its very wide gamut and its non-virtual primary colors, which strikes a balance that is deemed practical.
* toDataURL is lossy when us on a canvas that is in the linear-rec-2020 space. Possible future improvements could solve or mitigate this issue by adding more file formats or adding options to specify the resource color space.
* ImageData uses float32, which is inefficient due to memory consumption and necessary conversion operations. Float32 was chosen because it is convenient for manipulation (e.g. image processing) due to its native support in JavaScript (and current CPUs). A possible extension would be to add and option for rec-2020 content to be encoded as float16s packed into Uint16 values.

==== Security and privacy issues ====
Some current implementations of CanvasRenderingContext2D color correct image resources for the display as they are drawn to the canvas. In other words, the canvas is in output referred color space. This is a known fingerprinting vulnerability since it exposes the user's display's color profile to scripts via getImageData. The current proposal does not solve the fingerprinting issue because it will still exist in legacy-srgb. To solve the problem, implementations must color-correct CSS colors, then by extension, legacy-srgb mode will be in the true sRGB color space by virtue of the color matching rules outlined above. When that becomes the case, images drawn to canvases will be color corrected to sRGB, which solves the problem. There is resistance to adopting this model because going through an sRGB intermediate is lossy compared to directly color correcting images for the display in a single pass (may cause banding and gamut clipping). This feature proposal mitigates the lossiness argument thanks to the linear-rec-2020 option.

==== Implementation notes ====
* Because float16 arithmetic is supported by many GPUs, but not by CPUs, implementations should probably opt to not support rec-2020 on hardware that does not provide any native support.
* When available, the srgb colorspace should use GPU API extension for SRGB support. This will streamline the conversion overhead for performing filtering and compositing in linear space.

==== Adoption ====
Lack of color management and color interoperability is a longstanding complaint about the canvas API.
Authors of games and imaging apps are expected to be enthusiastic adopters.

==== History ====
This proposal was originally incubated in the Khronos 3D Web group, with the participation of engineers from Google, Microsoft, Apple, Nvidia, and others.

[[Category:Proposals]]

CanvasColorSpace

2016-05-07T02:34:16Z

Junov:

:Color managing canvas contents

== Use Case Description ==
* Contents displayed through a canvas element should be color managed in order to minimize differences in appearance across browsers and display devices. Improving color fidelity matters a lot for artistic uses (e.g. photo and paint apps) and for e-commerce (product presentation).
* Canvases should be able to take advantage of the full color gamut of the display device.
* Creative apps that do image manipulation generally prefer compositing, filtering and interpolation calculations to be performed in a linear color space.

=== Current Limitations ===
* The color space of canvases is undefined in the current specification.
* The bit-depth of canvases is currently fixed to 8 bits per component, which is below the capabilities of some monitors. Monitors with higher contrast ratios require more bits per component to avoid banding.

=== Current Usage and Workarounds ===
The lack of color space interoperability is hard to work around. With some browser implementations that color correct images drawn to canvases by applying the display profile, apps that want to use canvases for color corrected image processing are stuck doing convoluted workarounds, such as:
* reverse-engineer the display profile by drawing test pattern images to the canvas and inspecting the color corrected result via getImageData
* bypass CanvasRenderingContext2D.drawImage() and use image decoders implemented in JavaScript to extract raw image data that was not tainted by the browser's color correction behavior.''

An aspect of current implementations that is interoperable is that colors match between CSS/HTML and canvases:
* A color value used as a canvas drawing style will have the same appearance as if the same color value were used as a CSS style
* An image resource drawn to a canvas element will have the same appearance as if it were displayed as the replaced content of an HTML element or used as a CSS style value.

This color matching behavior needs to be preserved to avoid breaking pre-existing content.

Some implementations convert images drawn to canvases to the sRGB color space. This has the advantage of making the color correction behavior device independent, but it clamps the gamuts of the rendered content to the sRGB gamut, which is significantly narrower than the gamuts of some current consumer devices.

=== Requests for this Feature ===

* <cite>[https://github.com/whatwg/html/issues/299]</cite> <blockquote><p>Allow 2dcontexts to use deeper color buffers</p></blockquote>
* <cite>[https://bugs.chromium.org/p/chromium/issues/detail?id=425935]</cite> <blockquote><p>Wrong color profile with 2D canvas</p></blockquote>
* Engineers from the Google Photos, Maps and Sheets teams have expressed a desire for canvases to become color managed. Particularly for the use case of resizing an imaging, using a canvas, prior to uploading it to the server, to save bandwidth. The problem is that the images retrieved from a canvas are in an undefined color space and no color space information is encoded by toDataURL or toBlob.

== Proposed Solutions ==

=== Proposed solution: CanvasColorSpace ===
:Add a canvas color space creation parameter that allows user code to chose between backwards compatible behavior and color managed behaviors The same color space option would exist in the ImageData and ImageBitmap interfaces.

==== Processing Model ====

===== The color-space canvas creation parameter =====

IDL:
<code><pre>
enum CanvasColorSpace {
"legacy-srgb",
"srgb",
"linear-rec-2020"
};

dictionary CanvasRenderingContext2DSettings {
boolean alpha = true;
CanvasColorSpace color-space = "legacy-srgb";
};
</pre></code>

Example:
<code><pre>
canvas.getContext('2d', { color-space: "srgb"})
</pre></code>

===== The legacy-srgb color space =====

* Assures backwards compatible behavior
* Guarantees color matching with CSS and HTML content
* Color management behavior is implementation specific, may not use strict sRGB space, but is expected to be near sRGB. For example, could be display referred color space.
* toDataURL/toBlob produce resources with no color profile (backwards compat)
* Image resources with no color profile are never color corrected (backwards compat). This rule and the previous one allow for lossless toDataURL/drawImage round trips, which is a significant use case.

===== The srgb color space =====

* May break color matching with CSS on implementations that do not color-manage CSS.
* 8 bit unsigned integers per color component.
* All content drawn into the canvas must be color corrected to sRGB
* displayed canvases must be color corrected for the display if a display color profile is available. This color correction happens downstream at the compositing stage, and has no script-visible side-effects.
* Compositing, filtering and interpolation operations must perform all arithmetic in '''linear''' sRGB space.
* toDataURL/toBlob produce resources tagged as being in the sRGB colorspace
* Images with no color profile, when drawn to the canvas, are assumed to already be in the sRGB color space.

===== The linear-rec-2020 color space =====

* Space provide for wide gamut and high dynamic range rendering
* User agents may decide not to support the mode, based on host machine capabilities
* Uses 16-bit floating point representation.
* The color space corresponds to ITU-R Recommendation BT.2020, '''without gamma compression'''.
* toDataURL/toBlob convert image data to sRGB and produce image resources tagged as being in the sRGB color space.
* Image with no color profile, when drawn to the canvas, are assumed to be in the sRGB color space, and are converted to linear-rec-2020 for the purpose of the draw.

===== Feature detection =====

Rendering context objects are to expose a new "settings" attribute, which represents the settings that were successfully applied at context creation time.

Note: An alternative approach that was considered was to augment the probablySupportsContext() API by making it check the second argument. That approach is difficult to consolidate with how dictionary argument are meant to work, where unsupported entries are just ignored.

===== ImageBitmap =====

ImageBitmap objects are augmented to have an internal color space attribute of type CanvasColorSpace. The colorSpaceConversion creation attribute is to be augmented with new enum values for coercing conversions to a specific CanvasColorSpace at creation time.

===== ImageData =====

IDL
<code><pre>
typedef (Uint8ClampedArray or Float32Array) ImageDataArray;

[Constructor(unsigned long sw, unsigned long sh, optional CanvasColorSpace colorSpace = "legacy-srgb"),
Constructor(ImageDataArray data, unsigned long sw, optional unsigned long sh, optional CanvasColorSpace colorSpace),
Exposed=(Window,Worker)]
interface ImageData {
readonly attribute unsigned long width;
readonly attribute unsigned long height;
readonly attribute ImageDataArray data;
readonly attribute CanvasColorSpace colorSpace;
};
</pre></code>

* <code>data</code> is a Uint8ClampedArray if colorSpace is "srgb" or "legacy-srgb"
* <code>data</code> is a Float32Array if colorSpace is "linear-rec-2020"
* getImageData() produces an ImageData object in the same color space as the source canvas
* putImageData() performs a color space conversion to the color space of the destination canvas.

==== Limitations ====
* No support for arbitrary color spaces and bit depth. This capability could be added in the future. The current proposal attempts to solve the problem with a minimal API surface, and keeps the implementation scope reasonable. The extensible design will allow us to extend the capabilities in the future if necessary. The rec-2020 space was chosen for its very wide gamut and its non-virtual primary colors, which strikes a balance that is deemed practical.
* toDataURL is lossy in the linear-rec-2020 space. Possible future improvements could solve or mitigate this issue by adding more file formats or adding options to specify the resource color space.
* ImageData uses float32, which is inefficient due to memory consumption and necessary conversion operations. Float32 was chosen because it is convenient for manipulation (e.g. image processing) due to its native support in JavaScript (and current CPUs). A possible extension would be to add and option for rec-2020 content to be encoded as float16s packed into Uint16 values.

==== Security and privacy issues ====
Some current implementations of CanvasRenderingContext2D color correct image resources for the display as they are drawn to the canvas. In other words, the canvas is in output referred color space. This is a known fingerprinting vulnerability since it exposes the user's display's color profile to scripts via getImageData. The current proposal does not solve the fingerprinting issue because it will still exist in legacy-srgb. To solve the problem, implementations must color-correct CSS colors, then by extension, legacy-srgb mode will be in the true sRGB color space by virtue of the color matching rules outlined above. When that becomes the case, images drawn to canvases will be color corrected to sRGB, which solves the problem. There is resistance to adopting this model because going through an sRGB intermediate is lossy compared to directly color correcting images for the display in a single pass (may cause banding and gamut clipping). This feature proposal mitigates the lossiness argument thanks to the linear-rec-2020 option.

==== Implementation notes ====
* Because float16 arithmetic is supported by many GPUs, but not by CPUs, implementations should probably opt to not support rec-2020 on hardware that does not provide any native support.
* When available, the srgb colorspace should use GPU API extension for SRGB support. This will streamline the conversion overhead for performing filtering and compositing in linear space.

==== Adoption ====
Lack of color management and color interoperability is a longstanding complaint about the canvas API.
Authors of games and imaging apps are expected to be enthusiastic adopters.

==== History ====
This proposal was originally incubated in the Khronos 3D Web group, with the participation of engineers from Google, Microsoft, Apple, Nvidia, and others.

[[Category:Proposals]]

CanvasColorSpace

2016-05-06T18:14:53Z

Junov:

:Color managing canvas contents

Note: This proposal was originally incubated in the Khronos 3D Web group, with the participation of engineers from Google, Microsoft, Apple, Nvidia, and others

== Use Case Description ==
* Contents displayed through a canvas element should be color managed in order to minimize differences in appearance across browsers and display devices. Improving color fidelity matters a lot for artistic uses (e.g. photo and paint apps) and for e-commerce (product presentation).
* Canvases should be able to take advantage of the full color gamut of the display device.
* Creative apps that do image manipulation generally prefer compositing, filtering and interpolation calculations to be performed in a linear color space.

=== Current Limitations ===
* The color space of canvases is undefined in the current specification.
* The bit-depth of canvases is currently fixed to 8 bits per component, which is below the capabilities of some monitors. Monitors with higher contrast ratios require more bits per component to avoid banding.

=== Current Usage and Workarounds ===
The lack of color space interoperability is hard to work around. With some browser implementations that color correct images drawn to canvases by applying the display profile, apps that want to use canvases for color corrected image processing are stuck doing convoluted workarounds, such as:
* reverse-engineer the display profile by drawing test pattern images to the canvas and inspecting the color corrected result via getImageData
* bypass CanvasRenderingContext2D.drawImage() and use image decoders implemented in JavaScript to extract raw image data that was not tainted by the browser's color correction behavior.''

An aspect of current implementations that is interoperable is that colors match between CSS/HTML and canvases:
* A color value used as a canvas drawing style will have the same appearance as if the same color value were used as a CSS style
* An image resource drawn to a canvas element will have the same appearance as if it were displayed as the replaced content of an HTML element or used as a CSS style value.

This color matching behavior needs to be preserved to avoid breaking pre-existing content.

Some implementations convert images drawn to canvases to the sRGB color space. This has the advantage of making the color correction behavior device independent, but it clamps the gamuts of the rendered content to the sRGB gamut, which is significantly narrower than the gamuts of some current consumer devices.

=== Requests for this Feature ===

* <cite>[https://github.com/whatwg/html/issues/299]</cite> <blockquote><p>Allow 2dcontexts to use deeper color buffers</p></blockquote>
* <cite>[https://bugs.chromium.org/p/chromium/issues/detail?id=425935]</cite> <blockquote><p>Wrong color profile with 2D canvas</p></blockquote>
* Engineers from the Google Photos, Maps and Sheets teams have expressed a desire for canvases to become color managed. Particularly for the use case of resizing an imaging, using a canvas, prior to uploading it to the server, to save bandwidth. The problem is that the images retrieved from a canvas are in an undefined color space and no color space information is encoded by toDataURL or toBlob.

== Proposed Solutions ==

=== Proposed solution: CanvasColorSpace ===
:Add a canvas color space creation parameter that allows user code to chose between backwards compatible behavior and color managed behaviors The same color space option would exist in the ImageData and ImageBitmap interfaces.

==== Processing Model ====

===== The color-space canvas creation parameter =====

IDL:
<code><pre>
enum CanvasColorSpace {
"legacy-srgb",
"srgb",
"linear-rec-2020"
};

dictionary CanvasRenderingContext2DSettings {
boolean alpha = true;
CanvasColorSpace color-space = "legacy-srgb";
};
</pre></code>

Example:
<code><pre>
canvas.getContext('2d', { color-space: "srgb"})
</pre></code>

===== The legacy-srgb color space =====

* Assures backwards compatible behavior
* Guarantees color matching with CSS and HTML content
* Color management behavior is implementation specific, may not use strict sRGB space, but is expected to be near sRGB. For example, could be display referred color space.
* toDataURL/toBlob produce resources with no color profile (backwards compat)
* Image resources with no color profile are never color corrected (backwards compat). This rule and the previous one allow for lossless toDataURL/drawImage round trips, which is a significant use case.

===== The srgb color space =====

* May break color matching with CSS on implementations that do not color-manage CSS.
* 8 bit unsigned integers per color component.
* All content drawn into the canvas must be color corrected to sRGB
* displayed canvases must be color corrected for the display if a display color profile is available. This color correction happens downstream at the compositing stage, and has no script-visible side-effects.
* Compositing, filtering and interpolation operations must perform all arithmetic in '''linear''' sRGB space.
* toDataURL/toBlob produce resources tagged as being in the sRGB colorspace
* Images with no color profile, when drawn to the canvas, are assumed to already be in the sRGB color space.

===== The linear-rec-2020 color space =====

* Space provide for wide gamut and high dynamic range rendering
* User agents may decide not to support the mode, based on host machine capabilities
* Uses 16-bit floating point representation.
* The color space corresponds to ITU-R Recommendation BT.2020, '''without gamma compression'''.
* toDataURL/toBlob convert image data to sRGB and produce image resources tagged as being in the sRGB color space.
* Image with no color profile, when drawn to the canvas, are assumed to be in the sRGB color space, and are converted to linear-rec-2020 for the purpose of the draw.

===== Feature detection =====

Rendering context objects are to expose a new "settings" attribute, which represents the settings that were successfully applied at context creation time.

Note: An alternative approach that was considered was to augment the probablySupportsContext() API by making it check the second argument. That approach is difficult to consolidate with how dictionary argument are meant to work, where unsupported entries are just ignored.

===== ImageBitmap =====

ImageBitmap objects are augmented to have an internal color space attribute of type CanvasColorSpace. The colorSpaceConversion creation attribute is to be augmented with new enum values for coercing conversions to a specific CanvasColorSpace at creation time.

===== ImageData =====

IDL
<code><pre>
typedef (Uint8ClampedArray or Float32Array) ImageDataArray;

[Constructor(unsigned long sw, unsigned long sh, optional CanvasColorSpace colorSpace = "legacy-srgb"),
Constructor(ImageDataArray data, unsigned long sw, optional unsigned long sh, optional CanvasColorSpace colorSpace),
Exposed=(Window,Worker)]
interface ImageData {
readonly attribute unsigned long width;
readonly attribute unsigned long height;
readonly attribute ImageDataArray data;
readonly attribute CanvasColorSpace colorSpace;
};
</pre></code>

* <code>data</code> is a Uint8ClampedArray if colorSpace is "srgb" or "linear-srgb"
* <code>data</code> is a Float32Array if colorSpace is "linear-rec-2020"
* getImageData() produces an ImageData object in the same color space as the source canvas
* putImageData() performs a color space conversion to the color space of the destination canvas.

==== Limitations ====
* No support for arbitrary color spaces and bit depth. This capability could be added in the future. The current proposal attempts to solve the problem with a minimal API surface, and keeps the implementation scope reasonable. The extensible design will allow us to extend the capabilities in the future if necessary. The rec-2020 space was chosen for its very wide gamut and its non-virtual primary colors, which strikes a balance that is deemed practical.
* toDataURL is lossy in the linear-rec-2020 space. Possible future improvements could solve or mitigate this issue by adding more file formats or adding options to specify the resource color space.
* ImageData uses float32, which is inefficient due to memory consumption and necessary conversion operations. Float32 was chosen because it is convenient for manipulation (e.g. image processing) due to its native support in JavaScript (and current CPUs). A possible extension would be to add and option for rec-2020 content to be encoded as float16s packed into Uint16 values.

==== Implementation notes ====
* Because float16 arithmetic is supported by many GPUs, but not by CPUs, implementations should probably opt to not support rec-2020 on hardware that does not provide any native support.
* When available, the srgb colorspace should use GPU API extension for SRGB support. This will streamline the conversion overhead for performing filtering and compositing in linear space.

==== Adoption ====
Lack of color management and color interoperability is a longstanding complaint about the canvas API.
Authors of games and imaging apps are expected to be enthusiastic adopters.

[[Category:Proposals]]

CanvasColorSpace

2016-05-06T17:13:17Z

Junov: Created page with ":Color managing canvas contents Note: This proposal was originally incubated in the Khronos 3D Web group, with the participation of engineers from Google, Microsoft, Apple, N..."

:Color managing canvas contents

Note: This proposal was originally incubated in the Khronos 3D Web group, with the participation of engineers from Google, Microsoft, Apple, Nvidia, and others

== Use Case Description ==
* Contents displayed through a canvas element should be color managed in order to minimize differences in appearance across browsers and display devices. Improving color fidelity matters a lot for artistic uses (e.g. photo and paint apps) and for e-commerce (product presentation).
* Canvases should be able to take advantage of the full color gamut of the display device.
* Creative apps that do image manipulation generally prefer compositing, filtering and interpolation calculations to be performed in a linear color space.

=== Current Limitations ===
* The color space of canvases in undefined in the current specification.
* The bit-depth of canvases is currently fixed to 8 bits per component, which is below the capabilities of some monitors. Monitors with higher contrast ratios require more bits per component to avoid banding.

=== Current Usage and Workarounds ===
The lack of color space interoperability is hard to work around. With some browser implementations that color correct images drawn to canvases by applying the display profile, apps that want to use canvases for color corrected image processing are stuck doing convoluted workarounds, such as:
* reverse-engineer the display profile by drawing test pattern images to the canvas and inspecting the color corrected result via getImageData
* bypass CanvasRenderingContext2D.drawImage() and use image decoders implemented in javascript to extract raw image data that was not tainted by the browser's color correction behavior.''

An aspect of current implementations that is interoperable is that colors match between CSS/HTML and canvases:
* A color value used as a canvas drawing style will have the same appearance as if the same color value were used as a CSS style
* An image resource drawn to a canvas element will have the same appearance as if it were displayed as the replaced content of an HTML element or used as a CSS style value.

This color matching behavior needs to be preserved to avoid breaking pre-existing content.

Some implementations convert images drawn to canvases to the sRGB color space. This has the advantage of making the color correction behavior device independent, but it clamps the gamuts of the rendered content to the sRGB gamut, which is significantly narrower than the gamuts of some current consumer devices.

=== Requests for this Feature ===

* <cite>[https://github.com/whatwg/html/issues/299]</cite> <blockquote><p>Allow 2dcontexts to use deeper color buffers</p></blockquote>
* <cite>[https://bugs.chromium.org/p/chromium/issues/detail?id=425935]</cite> <blockquote><p>Wrong color profile with 2D canvas</p></blockquote>
* Engineers from the Google Photos, Maps and Sheets teams have expressed a desire for canvases to become color managed. Particularly for the use case of resizing an imaging, using a canvas, prior to uploading it to the server, to save bandwidth. The problem is that the images retrieved from a canvas are in an undefined color space and no color space information is encoded by toDataURL or toBlob.

== Proposed Solutions ==

=== Proposed solution: CanvasColorSpace ===
:Add a canvas color space creation parameter that allows user code to chose between backwards compatible behavior and color managed behaviors The same color space option would exist in the ImageData and ImageBitmap interfaces.

==== Processing Model ====

===== The color-space canvas creation parameter =====

IDL:
<code><pre>
enum CanvasColorSpace {
"legacy-srgb",
"srgb",
"linear-rec-2020"
};

dictionary CanvasRenderingContext2DSettings {
boolean alpha = true;
CanvasColorSpace color-space = "legacy-srgb";
};
</pre></code>

Example:
<code><pre>
canvas.getContext('2d', { color-space: "srgb"})
</pre></code>

===== The legacy-srgb color space =====

* Assures backwards compatible behavior
* Guarantees color matching with CSS and HTML content
* Color management behavior is implementation specific, may not use strict sRGB space, but is expected to be near sRGB. For example, could be display referred color space.
* toDataURL/toBlob produce resources with no color profile (backwards compat)
* Image resources with no color profile are never color corrected (backwards compat). This rule and the previous one allow for lossless toDataURL/drawImage round trips, which is a significant use case.

===== The srgb color space =====

* May break color matching with CSS on implementations that do not color-manage CSS.
* 8 bit unsigned integers per color component.
* All content drawn into the canvas must be color corrected to sRGB
* displayed canvases must be color corrected for the display if a display color profile is available. This color correction happens downstream at the compositing stage, and has no script-visible side-effects.
* Compositing, filtering and interpolation operations must perform all arithmetic in '''linear''' sRGB space.
* toDataURL/toBlob produce resources tagged as being in the sRGB colorspace
* Images with no color profile, when drawn to the canvas, are assumed to already be in the sRGB color space.

===== The linear-rec-2020 color space =====

* Space provide for wide gamut and high dynamic range rendering
* User agents may decide not to support the mode, based on host machine capabilities
* Uses 16-bit floating point representation.
* The color space corresponds to ITU-R Recommendation BT.2020, '''without gamma compression'''.
* toDataURL/toBlob convert image data to sRGB and produce image resources tagged as being in the sRGB color space.
* Image with no color profile, when drawn to the canvas, are assumed to be in the sRGB color space, and are converted to linear-rec-2020 for the purpose of the draw.

===== Feature detection =====

Rendering context objects are to expose a new "settings" attribute, which represents the settings that were successfully applied at context creation time.

Note: An alternative approach that was considered was to augment the probablySupportsContext() API by making it check the second argument. That approach is difficult to consolidate with how dictionary argument are meant to work, where unsupported entries are just ignored.

===== ImageBitmap =====

ImageBitmap objects are augmented to have an internal color space attribute of type CanvasColorSpace. The colorSpaceConversion creation attribute is to be augmented with new enum values for coercing conversions to a specific CanvasColorSpace at creation time.

===== ImageData =====

IDL
<code><pre>
typedef (Uint8ClampedArray or Float32Array) ImageDataArray;

[Constructor(unsigned long sw, unsigned long sh, optional CanvasColorSpace colorSpace = "legacy-srgb"),
Constructor(ImageDataArray data, unsigned long sw, optional unsigned long sh, optional CanvasColorSpace colorSpace),
Exposed=(Window,Worker)]
interface ImageData {
readonly attribute unsigned long width;
readonly attribute unsigned long height;
readonly attribute ImageDataArray data;
readonly attribute CanvasColorSpace colorSpace;
};
</pre></code>

* <code>data</code> is a Uint8ClampedArray if colorSpace is "srgb" or "linear-srgb"
* <code>data</code> is a Float32Array if colorSpace is "linear-rec-2020"
* getImageData() produces an ImageData object in the same color space as the source canvas
* putImageData() performs a color space conversion to the color space of the destination canvas.

==== Limitations ====
* No support for arbitrary color spaces and bit depth. This capability could be added in the future. The current proposal attempts to solve the problem with a minimal API surface, and keeps the implementation scope reasonable. The extensible design will allow us to extend the capabilities in the future if necessary. The rec-2020 space was chosen for its very wide gamut and its non-virtual primary colors, which strikes a balance that is deemed practical.
* toDataURL is lossy in the linear-rec-2020 space. Possible future improvements could solve or mitigate this issue by adding more file formats or adding options to specify the resource color space.
* ImageData uses float32, which is inefficient due to memory consumption and necessary conversion operations. Float32 was chosen because it is convenient for manipulation (e.g. image processing) due to its native support in JavaScript (and current CPUs). A possible extension would be to add and option for rec-2020 content to be encoded as float16s packed into Uint16 values.

==== Implementation notes ====
* Because float16 arithmetic is supported by many GPUs, but not by CPUs, implementations should probably opt to not support rec-2020 on hardware that does not provide any native support.
* When available, the srgb colorspace should use GPU API extension for SRGB support. This will streamline the conversion overhead for performing filtering and compositing in linear space.

==== Adoption ====
Lack of color management and color interoperability is a longstanding complaint about the canvas API.
Authors of games and imaging apps are expected to be enthusiastic adopters.

[[Category:Proposals]]

OffscreenCanvas

2016-04-08T17:18:21Z

Junov: /* Web IDL */

:''Provides more control over how canvases are rendered. This is a follow-on to the [[WorkerCanvas]] proposal and will be merged once agreement is reached.''

== Use Case Description ==

Feedback from web application authors using canvases have shown the need for the following controls:

* (From ShaderToy, Sketchfab, Verold): need to be able to render to multiple regions on the page efficiently using a single canvas context. 3D model warehouse sites desire to show multiple live interactive models on the page, but creating multiple WebGL contexts per page is too inefficient. A single context should be able to render to multiple regions on the page.
* (From Google Maps): need to be able to render WebGL from a worker, transfer the rendered image to the main thread without making any copy of it, and composite it with other HTML on the page, guaranteeing that the updates are all seen in the same rendered frame.
* (From Mozilla and partners using Emscripten and asm.js): need to be able to render WebGL entirely asynchronously from a worker, displaying the results in a canvas owned by the main thread, without any synchronization with the main thread. In this mode, the entire application runs in the worker. The main thread only receives input events and sends them to the worker for processing.
* (From adopters of the Push API): need to be able to dynamically create images to use as notification icons, such as compositing avatars, or adding an unread count
* (From the Google Docs team): need to be able to layout and render text from a worker using CanvasRenderingContext2D and display those results on the main thread.
* (From the Google Slides team): want to layout and render the slide thumbnails from a worker. During initial load and heavy collaboration these update frequently, and currently cause slowdowns on the main thread.

=== Current Limitations ===

* [https://html.spec.whatwg.org/multipage/scripting.html#proxying-canvases-to-workers CanvasProxy] does not provide sufficient control to allow synchronization between workers' rendering and DOM updates on the main thread. Keeping this rendering in sync is a requirement from Google's Maps team.
* [[CanvasInWorkers]] does not allow a worker to render directly into a canvas on the main thread without running code on the main thread. Allowing completely unsynchronized rendering is a requirement from Mozilla and users of Emscripten such as Epic Games and Unity, in which the desire is to execute all of the game's rendering on a worker thread.
* [[WorkerCanvas]] mostly addresses these two use cases, but some implementers objected to the mechanism for displaying the rendering results in image elements. The specific objection was that image elements already have complex internal state (for example, the management of the image's "loaded" state), and this would make it more complex. It also did not precisely address the use case of producing new frames both on the main thread and in workers.

=== Current Usage and Workarounds ===

[https://blog.mozilla.org/research/2014/07/22/webgl-in-web-workers-today-and-faster-than-expected/ WebGL in Web Workers] details some work attempted in the Emscripten toolchain to address the lack of WebGL in workers. Due to the high volume of calls and large amount of data that is transferred to the graphics card in a typical high-end WebGL application, this approach is not sustainable. It's necessary for workers to be able to call the WebGL API directly, and present those results to the screen in a manner that does not introduce any copies of the rendering results.

=== Benefits ===

Making canvas rendering contexts available to workers will increase parallelism in web applications, leading to increased performance on multi-core systems.

=== Requests for this Feature ===

See the abovementioned use cases:

* Google's Maps team
* Emscripten users such as Epic Games and Unity
* Many others

== Web IDL ==

[Constructor(unsigned long width, unsigned long height),
Exposed=(Window,Worker)]
interface OffscreenCanvas {
attribute unsigned long width;
attribute unsigned long height;
RenderingContext? getContext(DOMString contextId, any... arguments);

// OffscreenCanvas, like HTMLCanvasElement, maintains an origin-clean flag.
// ImageBitmaps created by calling this method also have an
// origin-clean flag which is set to the value of the OffscreenCanvas's
// flag at the time of their construction. Uses of the ImageBitmap
// in other APIs, such as CanvasRenderingContext2D or
// WebGLRenderingContext, propagate this flag like other
// CanvasImageSource types do, such as HTMLImageElement.
ImageBitmap transferToImageBitmap();

// Throws a SecurityError if the OffscreenCanvas's origin-clean flag
// is set to false.
Promise<Blob> toBlob(optional DOMString type, any... arguments);
};

OffscreenCanvas implements Transferable;
ImageBitmap implements Transferable;

// It's crucial that there be a way to explicitly dispose of ImageBitmaps
// since they refer to potentially large graphics resources. Some uses
// of this API proposal will result in repeated allocations of ImageBitmaps,
// and garbage collection will not reliably reclaim them quickly enough.
// Here we reuse close(), which also exists on another Transferable type,
// MessagePort. Potentially, all Transferable types should inherit from a
// new interface type "Closeable".
partial interface ImageBitmap {
// Dispose of all graphical resources associated with this ImageBitmap.
void close();
};

partial interface HTMLCanvasElement {
OffscreenCanvas transferControlToOffscreen();
};

// Note that CanvasRenderingContext2D already has a commit() method
// from the CanvasProxy spec which this proposal obsoletes.
partial interface CanvasRenderingContext2D {
// back-reference to the canvas
readonly attribute (HTMLCanvasElement or OffscreenCanvas) canvas;
};

partial interface WebGLRenderingContextBase {
// back-reference to the canvas
readonly attribute (HTMLCanvasElement or OffscreenCanvas) canvas;

// If this context is associated with an OffscreenCanvas that was
// created by HTMLCanvasElement's transferControlToOffscreen method,
// causes this context's current rendering results to be pushed
// to that canvas element. This has the same effect as returning
// control to the main loop in a single-threaded application. Otherwise,
// this call has no effect.
void commit();
};

// The new ImageBitmapRenderingContext is a canvas rendering context
// which only provides the functionality to replace the canvas's
// contents with the given ImageBitmap. Its context id (the first argument
// to getContext) is "bitmaprenderer".
interface ImageBitmapRenderingContext {
// Displays the given ImageBitmap in the canvas associated with this
// rendering context. Ownership of the ImageBitmap is transferred to
// the canvas. The caller may not use its reference to the ImageBitmap
// after making this call. (This semantic is crucial to enable prompt
// reclamation of expensive graphics resources, rather than relying on
// garbage collection to do so.)
//
// The ImageBitmap conceptually replaces the canvas's bitmap, but
// it does not change the canvas's intrinsic width or height.
//
// The ImageBitmap, when displayed, is clipped to the rectangle
// defined by the canvas's instrinsic width and height. Pixels that
// would be covered by the canvas's bitmap which are not covered by
// the supplied ImageBitmap are rendered transparent black. Any CSS
// styles affecting the display of the canvas are applied as usual.
void transferImageBitmap(ImageBitmap bitmap);
};

== Proposed Solutions ==

=== This Solution ===

This proposed API can be used in several ways to satisfy the use cases described above:

* It supports zero-copy transfer of canvases' rendering results between threads, for example from a worker to the main thread. In this model, the main thread controls when to display new frames produced by the worker, so synchronization with other DOM updates is achieved.

* It supports fully asynchronous rendering by a worker into a canvas displayed on the main thread. This satisfies certain Emscripten developers' full-screen use cases.

* It supports using a single WebGLRenderingContext or Canvas2DRenderingContext to efficiently render into multiple regions on the web page.

* It introduces ImageBitmapRenderingContext, a new canvas context type whose sole purpose is to efficiently display ImageBitmaps. This supersedes the [[WorkerCanvas]] proposal's use of HTMLImageElement for this purpose.

* It supports asynchronous encoding of OffscreenCanvases' rendering results into Blobs which can be consumed by various other web platform APIs.

==== Processing Model ====

This proposal introduces two primary processing models. The first involves ''synchronous'' display of new frames produced by the OffscreenCanvas. The application generates new frames using the RenderingContext obtained from the OffscreenCanvas. When the application is finished rendering each new frame, it calls transferToImageBitmap to "tear off" the most recently rendered image from the OffscreenCanvas -- like a Post-It note. The resulting ImageBitmap can then be used in any API receiving that data type; notably, it can be displayed in a second canvas without introducing a copy. An ImageBitmapRenderingContext is obtained from the second canvas by calling <code>getContext('bitmaprenderer')</code>. Each frame is displayed in the second canvas using the <code>transferImageBitmap</code> method on this rendering context. Note that the threads producing and consuming the frames may be the same, or they may be different. Note also that a single OffscreenCanvas may transfer frames into an arbitrary number of other ImageBitmapRenderingContexts.

The second processing model involves ''asynchronous'' display of new frames produced by the OffscreenCanvas. The main thread instantiates an HTMLCanvasElement and calls <code>transferControlToOffscreeen</code>. <code>getContext</code> is used to obtain a rendering context for that OffscreenCanvas, either on the main thread, or on a worker. The application calls <code>commit</code> against that rendering context in order to push frames to the original HTMLCanvasElement. In this rendering model, it is not defined when those frames become visible in the original canvas element. However, if the following situations apply:

* It is a worker thread which is calling commit(), and
* The worker is calling commit() repeatedly against exactly one rendering context

then it is required that the user agent synchronize the calls to commit() to the vsync interval. Calls to commit() conceptually enqueue frames for display, and after an implementation-defined number of frames have been enqueued, further calls to commit() will block until earlier frames have been presented to the screen. (This requirement allows porting of applications which drive their own main loop rather than using an event-driven loop.)

==== Limitations ====

* A known good way to drive an animation loop from a worker is needed. requestAnimationFrame or a similar API needs to be defined on worker threads.
* Some parts of the CanvasRenderingContext2D interface shall not be supported due OffscreenCanvas objects having no relation to the DOM or Frame: HitRegions, scrollPathIntoView, drawFocusIfNeeded.
* Due to technical challenges, some implementors [https://bugzilla.mozilla.org/show_bug.cgi?id=801176#c29 (Google and Mozilla)] have expressed a desire to ship without initially supporting text rendering in 2D contexts. Open Issue: Should text support be formally excluded from the specification until implementors are prepared to ship it (or until a more feasible API is designed)?

==== Implementation ====

This proposal has been vetted by developers of Apple's Safari, Google's Chrome, Microsoft's Internet Explorer, and Mozilla's Firefox browsers. All vendors agreed upon the basic form of the API, so it is likely it will be implemented widely and compatibly.

==== Adoption ====

Web page authors have demanded increased parallelism support from the web platform for multiple years. If support for multithreaded rendering is added, it is likely it will be rapidly adopted.

==== Example code ====

Jeff Gilbert from Mozilla has crafted some example code utilizing this API:

* [https://github.com/jdashg/snippets/tree/master/webgl-from-worker Rendering WebGL from a worker using the commit() API]
* [https://github.com/jdashg/snippets/blob/master/webgl-one-to-many/index.html Using one WebGL context to render to many Canvas elements]

[[Category:Proposals]]

OffscreenCanvas

2015-10-06T01:49:52Z

Junov: Documented some limitations regarding 2D contexts.

:''Provides more control over how canvases are rendered. This is a follow-on to the [[WorkerCanvas]] proposal and will be merged once agreement is reached.''

== Use Case Description ==

Feedback from web application authors using canvases have shown the need for the following controls:

* (From ShaderToy, Sketchfab, Verold): need to be able to render to multiple regions on the page efficiently using a single canvas context. 3D model warehouse sites desire to show multiple live interactive models on the page, but creating multiple WebGL contexts per page is too inefficient. A single context should be able to render to multiple regions on the page.
* (From Google Maps): need to be able to render WebGL from a worker, transfer the rendered image to the main thread without making any copy of it, and composite it with other HTML on the page, guaranteeing that the updates are all seen in the same rendered frame.
* (From Mozilla and partners using Emscripten and asm.js): need to be able to render WebGL entirely asynchronously from a worker, displaying the results in a canvas owned by the main thread, without any synchronization with the main thread. In this mode, the entire application runs in the worker. The main thread only receives input events and sends them to the worker for processing.
* (From adopters of the Push API): need to be able to dynamically create images to use as notification icons, such as compositing avatars, or adding an unread count
* (From the Google Docs team): need to be able to layout and render text from a worker using CanvasRenderingContext2D and display those results on the main thread.
* (From the Google Slides team): want to layout and render the slide thumbnails from a worker. During initial load and heavy collaboration these update frequently, and currently cause slowdowns on the main thread.

=== Current Limitations ===

* [https://html.spec.whatwg.org/multipage/scripting.html#proxying-canvases-to-workers CanvasProxy] does not provide sufficient control to allow synchronization between workers' rendering and DOM updates on the main thread. Keeping this rendering in sync is a requirement from Google's Maps team.
* [[CanvasInWorkers]] does not allow a worker to render directly into a canvas on the main thread without running code on the main thread. Allowing completely unsynchronized rendering is a requirement from Mozilla and users of Emscripten such as Epic Games and Unity, in which the desire is to execute all of the game's rendering on a worker thread.
* [[WorkerCanvas]] mostly addresses these two use cases, but some implementers objected to the mechanism for displaying the rendering results in image elements. The specific objection was that image elements already have complex internal state (for example, the management of the image's "loaded" state), and this would make it more complex. It also did not precisely address the use case of producing new frames both on the main thread and in workers.

=== Current Usage and Workarounds ===

[https://blog.mozilla.org/research/2014/07/22/webgl-in-web-workers-today-and-faster-than-expected/ WebGL in Web Workers] details some work attempted in the Emscripten toolchain to address the lack of WebGL in workers. Due to the high volume of calls and large amount of data that is transferred to the graphics card in a typical high-end WebGL application, this approach is not sustainable. It's necessary for workers to be able to call the WebGL API directly, and present those results to the screen in a manner that does not introduce any copies of the rendering results.

=== Benefits ===

Making canvas rendering contexts available to workers will increase parallelism in web applications, leading to increased performance on multi-core systems.

=== Requests for this Feature ===

See the abovementioned use cases:

* Google's Maps team
* Emscripten users such as Epic Games and Unity
* Many others

== Web IDL ==

[Constructor(unsigned long width, unsigned long height),
Exposed=(Window,Worker)]
interface OffscreenCanvas {
attribute unsigned long width;
attribute unsigned long height;
RenderingContext? getContext(DOMString contextId, any... arguments);

// OffscreenCanvas, like HTMLCanvasElement, maintains an origin-clean flag.
// ImageBitmaps created by calling this method also have an
// origin-clean flag which is set to the value of the OffscreenCanvas's
// flag at the time of their construction. Uses of the ImageBitmap
// in other APIs, such as CanvasRenderingContext2D or
// WebGLRenderingContext, propagate this flag like other
// CanvasImageSource types do, such as HTMLImageElement.
ImageBitmap transferToImageBitmap();

// Throws a SecurityError if the OffscreenCanvas's origin-clean flag
// is set to false.
Promise<Blob> toBlob(optional DOMString type, any... arguments);
};

OffscreenCanvas implements Transferable;
ImageBitmap implements Transferable;

// It's crucial that there be a way to explicitly dispose of ImageBitmaps
// since they refer to potentially large graphics resources. Some uses
// of this API proposal will result in repeated allocations of ImageBitmaps,
// and garbage collection will not reliably reclaim them quickly enough.
// Here we reuse close(), which also exists on another Transferable type,
// MessagePort. Potentially, all Transferable types should inherit from a
// new interface type "Closeable".
partial interface ImageBitmap {
// Dispose of all graphical resources associated with this ImageBitmap.
void close();
};

partial interface HTMLCanvasElement {
OffscreenCanvas transferControlToOffscreen();
};

// Note that CanvasRenderingContext2D already has a commit() method
// from the CanvasProxy spec which this proposal obsoletes.
partial interface CanvasRenderingContext2D {
// back-reference to the canvas
readonly attribute (HTMLCanvasElement or OffscreenCanvas) canvas;
};

partial interface WebGLRenderingContextBase {
// back-reference to the canvas
readonly attribute (HTMLCanvasElement or OffscreenCanvas) canvas;

// If this context is associated with an OffscreenCanvas that was
// created by HTMLCanvasElement's transferControlToOffscreen method,
// causes this context's current rendering results to be pushed
// to that canvas element. This has the same effect as returning
// control to the main loop in a single-threaded application. Otherwise,
// this call has no effect.
void commit();
};

// The new ImageBitmapRenderingContext is a canvas rendering context
// which only provides the functionality to replace the canvas's
// contents with the given ImageBitmap. Its context id (the first argument
// to getContext) is "bitmaprenderer".
[Exposed=(Window,Worker)]
interface ImageBitmapRenderingContext {
// Displays the given ImageBitmap in the canvas associated with this
// rendering context. Ownership of the ImageBitmap is transferred to
// the canvas. The caller may not use its reference to the ImageBitmap
// after making this call. (This semantic is crucial to enable prompt
// reclamation of expensive graphics resources, rather than relying on
// garbage collection to do so.)
//
// The ImageBitmap conceptually replaces the canvas's bitmap, but
// it does not change the canvas's intrinsic width or height.
//
// The ImageBitmap, when displayed, is clipped to the rectangle
// defined by the canvas's instrinsic width and height. Pixels that
// would be covered by the canvas's bitmap which are not covered by
// the supplied ImageBitmap are rendered transparent black. Any CSS
// styles affecting the display of the canvas are applied as usual.
void transferImageBitmap(ImageBitmap bitmap);
};

== Proposed Solutions ==

=== This Solution ===

This proposed API can be used in several ways to satisfy the use cases described above:

* It supports zero-copy transfer of canvases' rendering results between threads, for example from a worker to the main thread. In this model, the main thread controls when to display new frames produced by the worker, so synchronization with other DOM updates is achieved.

* It supports fully asynchronous rendering by a worker into a canvas displayed on the main thread. This satisfies certain Emscripten developers' full-screen use cases.

* It supports using a single WebGLRenderingContext or Canvas2DRenderingContext to efficiently render into multiple regions on the web page.

* It introduces ImageBitmapRenderingContext, a new canvas context type whose sole purpose is to efficiently display ImageBitmaps. This supersedes the [[WorkerCanvas]] proposal's use of HTMLImageElement for this purpose.

* It supports asynchronous encoding of OffscreenCanvases' rendering results into Blobs which can be consumed by various other web platform APIs.

==== Processing Model ====

This proposal introduces two primary processing models. The first involves ''synchronous'' display of new frames produced by the OffscreenCanvas. The application generates new frames using the RenderingContext obtained from the OffscreenCanvas. When the application is finished rendering each new frame, it calls transferToImageBitmap to "tear off" the most recently rendered image from the OffscreenCanvas -- like a Post-It note. The resulting ImageBitmap can then be used in any API receiving that data type; notably, it can be displayed in a second canvas without introducing a copy. An ImageBitmapRenderingContext is obtained from the second canvas by calling <code>getContext('bitmaprenderer')</code>. Each frame is displayed in the second canvas using the <code>transferImageBitmap</code> method on this rendering context. Note that the threads producing and consuming the frames may be the same, or they may be different. Note also that a single OffscreenCanvas may transfer frames into an arbitrary number of other ImageBitmapRenderingContexts.

The second processing model involves ''asynchronous'' display of new frames produced by the OffscreenCanvas. The main thread instantiates an HTMLCanvasElement and calls <code>transferControlToOffscreeen</code>. <code>getContext</code> is used to obtain a rendering context for that OffscreenCanvas, either on the main thread, or on a worker. The application calls <code>commit</code> against that rendering context in order to push frames to the original HTMLCanvasElement. In this rendering model, it is not defined when those frames become visible in the original canvas element. However, if the following situations apply:

* It is a worker thread which is calling commit(), and
* The worker is calling commit() repeatedly against exactly one rendering context

then it is required that the user agent synchronize the calls to commit() to the vsync interval. Calls to commit() conceptually enqueue frames for display, and after an implementation-defined number of frames have been enqueued, further calls to commit() will block until earlier frames have been presented to the screen. (This requirement allows porting of applications which drive their own main loop rather than using an event-driven loop.)

==== Limitations ====

* A known good way to drive an animation loop from a worker is needed. requestAnimationFrame or a similar API needs to be defined on worker threads.
* Some parts of the CanvasRenderingContext2D interface shall not be supported due OffscreenCanvas objects having no relation to the DOM or Frame: HitRegions, scrollPathIntoView, drawFocusIfNeeded.
* Due to technical challenges, some implementors [https://bugzilla.mozilla.org/show_bug.cgi?id=801176#c29 (Google and Mozilla)] have expressed a desire to ship without initially supporting text rendering in 2D contexts. Open Issue: Should text support be formally excluded from the specification until implementors are prepared to ship it (or until a more feasible API is designed)?

==== Implementation ====

This proposal has been vetted by developers of Apple's Safari, Google's Chrome, Microsoft's Internet Explorer, and Mozilla's Firefox browsers. All vendors agreed upon the basic form of the API, so it is likely it will be implemented widely and compatibly.

==== Adoption ====

Web page authors have demanded increased parallelism support from the web platform for multiple years. If support for multithreaded rendering is added, it is likely it will be rapidly adopted.

==== Example code ====

Jeff Gilbert from Mozilla has crafted some example code utilizing this API:

* [https://github.com/jdashg/snippets/tree/master/webgl-from-worker Rendering WebGL from a worker using the commit() API]
* [https://github.com/jdashg/snippets/blob/master/webgl-one-to-many/index.html Using one WebGL context to render to many Canvas elements]

[[Category:Proposals]]

Changes to ImageBitmap for OffscreenCanvas

2015-04-21T15:09:55Z

Junov:

:'''WORK IN PROGRESS''' ''Amendments to ImageBitmap to provide zero-copy paths for moving pixel data and for working with the OffscreenCanvas proposal''

== Use Case Description ==
A common complaint from web developers writing applications that manipulate large images is that the browser tends to consume large amounts of RAM and GPU memory. Part of this is due to intermediate copies of image data that are made by the browser and transient copies that are kept around, awaiting garbage collection. Another complaint is that manipulation large images on the browser's main thread often makes applications janky.
Specific use cases:
* Generating or processing pixel data in javascript, possibly in a Worker thread, and bringing it to screen.
* Taking a snapshot of canvas-rendered content and uploading it as an image file to a remote server, possibly using XHR2 with a progress updates.
* Using 2D canvas to produce images that are subsequently used as textures in WebGL.
* Transferring a canvas-rendered image to anywhere an image URL can be used (e.g. CSS properties)
* Save locally rendered canvas content to a local disk.
* Stream snapshots of a local canvas to a remote server. ex: Broadcasting a live presentation rendered in WebGL
* Appending a locally rendered canvas image to form data as an image file.
* Save a canvas image locally with the anchor "download" attribute.
* Capturing a snapshot from a video stream (possibly WebRTC), and uploading the snapshot to a remote server.

Note: Many of these use cases were reported by web developpers in support of implementing the HTMLCanvasElement.toBlob interface. Below we counter-propose that putting toBlob on ImageBitmap will serve those use cases better or as well as having toBlob on <canvas>.

=== Current Limitations ===
The current API often forces the web developer to go through a canvas for moving pixel data. Because canvases are mutable, intermediate copies are often unavoidable, and may result in unnecessary multiply and divide by alpha, which is expensive and lossy. ImageBitmap, being an opaque immutable object type would allow for low friction means of moving pixel data, but it is missing some functionality that prevents it from being a universal pixel vehicle. In particular, it is missing transfer methods (required for zero-copy behavior).

=== Current Usage and Workarounds ===
''Some evidence that this feature is desperately needed on the web. You may provide a separate examples page for listing these.''

For sending image data to a remote server, a compressed image file in binary form is the preferred vehicle. This can currently be achieved by using HTMLCanvasElement.toBlob (or equivalent interface) with web browser that provide that API, or with this approach on other browsers: http://stackoverflow.com/questions/4998908/convert-data-uri-to-file-then-append-to-formdata. This method has some disadvantages: because it lives on a DOM interface, it can only be invoked from the main thread; because the canvas backing editable, the implementation is required to make a read-only snapshot for itself; In many use cases, the canvas itself represent an additional copy of the image data. Preventing the proliferation of copies of image data in RAM is very important for applications that manipulate large images (e.g. full res photos, maps), which are vulnerable to malfunctions cause by out-of-memory errors, particularly on mobile.

Experience has shown that garbage collection is often a performance liability when dealing with large temporary objects, particularly those that consume GPU memory. The ability to explicity discard image buffers when they are no longer needed has proven to be widely useful with 2D canvas, which offer this possibility indirectly by setting the intrinsic size to 0. This practice allows resources (RAM, GPU memory) to be freed as early as possible, hence avoiding process bloat, and it helps reduce the frequency of garbage collections, which can cause jank. Using ImageBitmaps as currently spec'ed implies losing this advantage.

=== Benefits ===

The proposed additions to the ImageBitmap interface aim to provide leaner browser memory consumption and fast, smoother performance for the above mentioned use cases.

=== Requests for this Feature ===

* <cite>[http://example.com Source]</cite> <blockquote><p>I would like this feature ...</p></blockquote>

== Proposed Solution ==

<pre>
[Exposed=(Window,Worker)]
interface ImageBitmap implements Transferable {
readonly attribute unsigned long width;
readonly attribute unsigned long width;
void close();
// Throws a SecurityError if the OffscreenCanvas's origin-clean flag
// is set to false.
Promise<Blob> toBlob(optional DOMString type, any... arguments);
}

interface
</pre>

Making ImageBitmap Transferrable allows it to be used as a zero-copy vehicle for passing pixel data between threads, hence reducing peak RAM consumption and avoiding the CPU cost of performing a copy.

Offering toBlob on ImageBitmap, as opposed to HTMLCanvasElement:
* Allows to toBlob be invoked from a worker, which help reduce jank on the main thread (even if the encode is on a separate thread, make a copy and interacting with the blob store can jank the main thread).
* Allows the data flow to bypass the canvas (for cases where the source of the image data is not a canvas), therefore avoiding an alpha multiply+divide in some cases, and avoiding an unnecessary intermediate (the canvas itself)
* Because ImageBitmap is immutable, no safety copy is required for toBlob to operate asynchronously. A counter argument to this is that an implementation of HTMLCanvasElement.toBlob could perform the copy lazily, only when the canvas backing is about to be mutated while a toBlob is in progress, thus performing a copy only in use case that require it. Though this is true, it results in performance characteritcs that may seem idiosyncratic to developers (i.e. the preformance of toBlob degrades if something is done to the canvas after calling toBlob.), as opposed to a flow that guarantees zero copies: OffscreenCanvas.transferToImageBitmap, followed by ImageBitmap.toBlob.

The close() method effectively neuters the ImageBitmap. It is a means of explicitly de-allocating large resources, which avoids waiting for garbage collection, and therefore reduces the frequency of GCs and brings down peak RAM consumption.

==== Processing Model ====
:''Explanation of the changes introduced by this solution. It explains how the document is processed, and how errors are handled. This should be very clear, including things such as event timing if the solution involves events, how to create graphs representing the data in the case of semantic proposals, etc.''

==== Limitations ====
:''Cases not covered by this solution in relation to the problem description; other problems with this solution, if any.''

==== Implementation ====
:''Description of how and why browser vendors would take advantage of this feature.''

==== Adoption ====
:''Reasons why page authors would use this solution.''

[[Category:Proposals]]

Changes to ImageBitmap for OffscreenCanvas

2015-04-17T15:40:16Z

Junov: Created page with ":'''WORK IN PROGRESS''' ''Amendments to ImageBitmap to provide zero-copy paths for moving pixel data and for working with the OffscreenCanvas proposal'' == Use Case Descripti..."

:'''WORK IN PROGRESS''' ''Amendments to ImageBitmap to provide zero-copy paths for moving pixel data and for working with the OffscreenCanvas proposal''

== Use Case Description ==
A common complaint from web developers writing applications that manipulate large images is that the browser tends to consume large amounts of RAM and GPU memory. Part of this is due to intermediate copies of image data that are made by the browser and transient copies that are kept around, awaiting garbage collection. Another complaint is that manipulation large images on the browser's main thread often makes applications janky.
Specific use cases:
* Generating pixel data in javascript, and bringing it to screen.
* Taking a snapshot of canvas-rendered content and uploading it as an image file to a remote server, possibly using XHR2 with a progress updates.
* Using 2D canvas to produce images that are subsequently used as textures in WebGL.
* Transfer a canvas-rendered image to anywhere an image URL can be used (e.g. CSS properties)
* Save locally rendered canvas content to a local disk.
* Stream snapshots of a local canvas to a remote server. ex: Broadcasting a live presentation rendered in WebGL
* Appending a locally rendered canvas image to form data as an image file.
* Save a canvas image locally with the anchor "download" attribute.
* Capturing a snapshot from a video stream (possibly WebRTC), and uploading the snapshot to a remote server.

Note: Many of these use cases were reported by web developpers in support of the HTMLCanvasElement.toBlob interface.

=== Current Limitations ===
The current API often forces the web developer to go through a canvas for moving pixel data. Because canvases are mutable, avoiding intermediate copies is often unavoidable, and may result in unnecessary multiply and divide by alpha, which is expensive and lossy. ImageBitmap, being an opaque immutable object type would allow for low friction means of moving pixel data, but it is missing some functionality that prevents it from being a universal pixel vehicle. In particular, it is missing transfer methods (required for zero-copy behavior).

=== Current Usage and Workarounds ===
''Some evidence that this feature is desperately needed on the web. You may provide a separate examples page for listing these.''

For sending image data to a remote server, a compressed image file in binary form is the preferred vehicle. This can currently be achieved by using HTMLCanvasElement.toBlob (or equivalent interface) with web browser that provide that API, or with this approach on other browsers: http://stackoverflow.com/questions/4998908/convert-data-uri-to-file-then-append-to-formdata. This method has some disadvantages: because it lives on a DOM interface, it can only be invoked from the main thread; because the canvas backing editable, the implementation is required to make a read-only snapshot for itself; In many use cases, the canvas itself represent an additional copy of the image data. Preventing the proliferation of copies of image data in RAM is very important for applications that manipulate large images (e.g. full res photos, maps), which are vulnerable to malfunctions cause by out-of-memory errors, particularly on mobile.

Experience has shown that garbage collection is often a performance liability when dealing with large temporary objects, particularly those that consume GPU memory. The ability to explicity discard image buffers when they are no longer needed has proven to be widely useful with 2D canvas, which offer this possibility indirectly by setting the intrinsic size to 0. This practice allows resources (RAM, GPU memory) to be freed as early as possible, hence avoiding process bloat, and it helps reduce the frequency of garbage collections, which can cause jank. Using ImageBitmaps as currently spec'ed implies losing this advantage.

=== Benefits ===

The proposed additions to the ImageBitmap interface aim to provide leaner browser memory consumption and fast, smoother performance for the above mentioned use cases.

=== Requests for this Feature ===

* <cite>[http://example.com Source]</cite> <blockquote><p>I would like this feature ...</p></blockquote>

== Proposed Solution ==

<pre>
[Exposed=(Window,Worker)]
interface ImageBitmap implements Transferable {
readonly attribute unsigned long width;
readonly attribute unsigned long width;
void close();
// Throws a SecurityError if the OffscreenCanvas's origin-clean flag
// is set to false.
Promise<Blob> toBlob(optional DOMString type, any... arguments);
}

interface
</pre>

Making ImageBitmap Transferrable allows it to be used as a zero-copy vehicle for passing pixel data between threads, hence reducing peak RAM consumption and avoiding the CPU cost of performing a copy.

Offering toBlob on ImageBitmap, as opposed to HTMLCanvasElement:
* Allows to toBlob be invoked from a worker, which help reduce jank on the main thread (even if the encode is on a separate thread, make a copy and interacting with the blob store can jank the main thread).
* Allows the data flow to bypass the canvas (for cases where the source of the image data is not a canvas), therefore avoiding an alpha multiply+divide in some cases, and avoiding an unnecessary intermediate (the canvas itself)
* Because ImageBitmap is immutable, no safety copy is required for toBlob to operate asynchronously. A counter argument to this is that an implementation of HTMLCanvasElement.toBlob could perform the copy lazily, only when the canvas backing is about to be mutated while a toBlob is in progress, thus performing a copy only in use case that require it. Though this is true, it results in performance characteritcs that may seem idiosyncratic to developers (i.e. the preformance of toBlob degrades if something is done to the canvas after calling toBlob.), as opposed to a flow that guarantees zero copies: OffscreenCanvas.transferToImageBitmap, followed by ImageBitmap.toBlob.

The close() method effectively neuters the ImageBitmap. It is a means of explicitly de-allocate large resources, which avoids waiting for garbage collection, and therefore reducing the frequency of GCs and bringing down peak RAM consumption.

==== Processing Model ====
:''Explanation of the changes introduced by this solution. It explains how the document is processed, and how errors are handled. This should be very clear, including things such as event timing if the solution involves events, how to create graphs representing the data in the case of semantic proposals, etc.''

==== Limitations ====
:''Cases not covered by this solution in relation to the problem description; other problems with this solution, if any.''

==== Implementation ====
:''Description of how and why browser vendors would take advantage of this feature.''

==== Adoption ====
:''Reasons why page authors would use this solution.''

[[Category:Proposals]]

CanvasRenderedPixelSize

2015-04-15T18:05:53Z

Junov: /* Issues: */

:''This proposal is to allow Web developers to match canvas backing store resolution to device resolution.''

== Use Case Descriptions ==

Use case #1: a Web application needs to render content to a canvas (using 2D or WebGL contexts) being displayed on a high-DPI screen. For best quality output, the pixels of the canvas backing store need to match 1:1 with screen pixels.

Use case #2: As #1, and the user moves the page from screen to screen resulting in dynamic DPI changes.

Use case #3: As #1, and the canvas is inside a CSS scale transform which may change dynamically.

Use case #4: As #1, and the canvas's CSS content box has arbitrary fractional CSS pixel size and offset from the viewport.

Use case #5: As #3, but the canvas is inside an <code><iframe></code> in a page with a different origin which contains the CSS transform.

=== Current Usage and Workarounds ===

Safari shipped a canvas implementation that automatically resized the backing store but this caused various problems, such as incompatibility with some existing content and a need for new "HD" APIs that authors were not motivated to use. This approach does not work for WebGL in any case.

The current best workaround is to set the canvas backing store size by computing the canvas content box in CSS pixels relative to the viewport, as accurately as possible (e.g. using <code>getBoxQuads</code>), multiplying by <code>window.devicePixelRatio</code>, and using that to set the canvas width and height. This approach does not easily adapt to dynamic changes and does not handle use cases #4 or #5. In use-case #4 the computed device pixel size may be non-integer, in which case the UA is likely to snap the content-box to device pixel edges and the Web developer cannot predict how that snapping will be done. In use-case #5 the developer cannot know about the CSS transform in the foreign-origin content.

== Goals ==

# Leave the application in control of setting the pixel size of the backing store.
# Retain the invariant that 1 unit in canvas coordinate space is 1 backing store pixel.
# Make it easy for applications to set the pixel size of the backing store so that backing store pixels are 1:1 with device pixels, when that's possible.

== Non Goals ==

# Enable high-resolution backing store for existing content.

== Proposed Solutions ==

Hixie proposed a canvas attribute to opt into automatic sizing of the backing store, but this violates goals #1 and #2. Violating goal #1 is a problem because applications often use temporary canvases and other assets which are dependent on the pixel size of the backing store. Goal #1 is also essential for WebGL.

=== Suggested Solution ===

Expose a preferred size for the canvas backing store directly to Web authors. Web authors who want to match screen resolution set the canvas width and height to that size. Expose an event on the canvas that fires when the preferred size changes.

== Suggested IDL ==

Add two new DOM attributes to <code>HTMLCanvasElement</code>:

<pre>partial interface HTMLCanvasElement {
readonly attribute long renderedPixelWidth;
readonly attribute long renderedPixelHeight;
};</pre>

The values of these attributes are not normatively defined. For example, a UA might reduce these values to conserve system resources.
However, under normal steady-state conditions a UA is expected to choose values such that, if the canvas element has a CSS background color exactly filling its content-box, and the UA renders that CSS background color as a rectangle whose edges are aligned to device pixel edges, then <code>renderedPixelWidth</code> and <code>renderedPixelHeight</code> are the device pixel size of that rectangle.

Add a new event <code>renderedsizechange</code> to <code>HTMLCanvasElement</code>. This event does not bubble and is not cancelable. Whenever the value that would be returned by <code>renderedPixelWidth</code> or <code>renderedPixelHeight</code> changes, queue a task to fire <code>renderedsizechange</code> at the <code>HTMLCanvasElement</code> if there is not already a task pending to fire such an event at that element.

== Rationale: ==

This API appears to be the minimal API needed to achieve the goals, is opt-in, and can be used in several different ways depending on how ambitious the app developer is:
* Simplest possible usage: set the canvas size once when loading and don't handle dynamic changes.
<pre> <script>
canvas.width = canvas.renderedPixelWidth; canvas.height = canvas.renderedPixelHeight;
</script></pre>
* Update canvas size for every frame of a game:
<pre> <script>
function renderFrame() {
canvas.width = canvas.renderedPixelWidth; canvas.height = canvas.renderedPixelHeight;
...
window.requestAnimationFrame(renderFrame);
}
</script></pre>
* Update canvas size and re-render only when DPI changes:
<pre> <script>
function render() {
canvas.width = canvas.renderedPixelWidth; canvas.height = canvas.renderedPixelHeight;
...
}
canvas.addEventHandler("renderedsizechange", render, false);
</script></pre>

== Issues: ==

* Hard to determine a rendered size when the canvas is not attached to the DOM. Perhaps in that case the current intrinsic size should be returned? (junov@chromium.org)
* After a layout change that affects rendered pixel size, there is no guarantee that the size change event will be handled before the layout change is propagated to screen, so the content may be temporarily displayed in an inconsistent state. Note: the same issue exists with existing methods that may be based on mutation observers or window.onresize, for example. Though it is not the stated objective of this proposal to solve this problem, there may be an opportunity to do so. (junov)
* Accessing rendered pixel size is layout-inducing. To avoid layout thrashing, we should consider making this an asynchronous getter (e.g. asyncGetBoundignClientRect). This would also prevent renderedsizechanged events from firing from within the evaluation of rendered pixel size, which is weird. (junov, credit: nduca)

[[Category:Proposals]]

CanvasRenderedPixelSize

2015-04-15T16:54:31Z

Junov: /* Issues: */

:''This proposal is to allow Web developers to match canvas backing store resolution to device resolution.''

== Use Case Descriptions ==

Use case #1: a Web application needs to render content to a canvas (using 2D or WebGL contexts) being displayed on a high-DPI screen. For best quality output, the pixels of the canvas backing store need to match 1:1 with screen pixels.

Use case #2: As #1, and the user moves the page from screen to screen resulting in dynamic DPI changes.

Use case #3: As #1, and the canvas is inside a CSS scale transform which may change dynamically.

Use case #4: As #1, and the canvas's CSS content box has arbitrary fractional CSS pixel size and offset from the viewport.

Use case #5: As #3, but the canvas is inside an <code><iframe></code> in a page with a different origin which contains the CSS transform.

=== Current Usage and Workarounds ===

Safari shipped a canvas implementation that automatically resized the backing store but this caused various problems, such as incompatibility with some existing content and a need for new "HD" APIs that authors were not motivated to use. This approach does not work for WebGL in any case.

The current best workaround is to set the canvas backing store size by computing the canvas content box in CSS pixels relative to the viewport, as accurately as possible (e.g. using <code>getBoxQuads</code>), multiplying by <code>window.devicePixelRatio</code>, and using that to set the canvas width and height. This approach does not easily adapt to dynamic changes and does not handle use cases #4 or #5. In use-case #4 the computed device pixel size may be non-integer, in which case the UA is likely to snap the content-box to device pixel edges and the Web developer cannot predict how that snapping will be done. In use-case #5 the developer cannot know about the CSS transform in the foreign-origin content.

== Goals ==

# Leave the application in control of setting the pixel size of the backing store.
# Retain the invariant that 1 unit in canvas coordinate space is 1 backing store pixel.
# Make it easy for applications to set the pixel size of the backing store so that backing store pixels are 1:1 with device pixels, when that's possible.

== Non Goals ==

# Enable high-resolution backing store for existing content.

== Proposed Solutions ==

Hixie proposed a canvas attribute to opt into automatic sizing of the backing store, but this violates goals #1 and #2. Violating goal #1 is a problem because applications often use temporary canvases and other assets which are dependent on the pixel size of the backing store. Goal #1 is also essential for WebGL.

=== Suggested Solution ===

Expose a preferred size for the canvas backing store directly to Web authors. Web authors who want to match screen resolution set the canvas width and height to that size. Expose an event on the canvas that fires when the preferred size changes.

== Suggested IDL ==

Add two new DOM attributes to <code>HTMLCanvasElement</code>:

<pre>partial interface HTMLCanvasElement {
readonly attribute long renderedPixelWidth;
readonly attribute long renderedPixelHeight;
};</pre>

The values of these attributes are not normatively defined. For example, a UA might reduce these values to conserve system resources.
However, under normal steady-state conditions a UA is expected to choose values such that, if the canvas element has a CSS background color exactly filling its content-box, and the UA renders that CSS background color as a rectangle whose edges are aligned to device pixel edges, then <code>renderedPixelWidth</code> and <code>renderedPixelHeight</code> are the device pixel size of that rectangle.

Add a new event <code>renderedsizechange</code> to <code>HTMLCanvasElement</code>. This event does not bubble and is not cancelable. Whenever the value that would be returned by <code>renderedPixelWidth</code> or <code>renderedPixelHeight</code> changes, queue a task to fire <code>renderedsizechange</code> at the <code>HTMLCanvasElement</code> if there is not already a task pending to fire such an event at that element.

== Rationale: ==

This API appears to be the minimal API needed to achieve the goals, is opt-in, and can be used in several different ways depending on how ambitious the app developer is:
* Simplest possible usage: set the canvas size once when loading and don't handle dynamic changes.
<pre> <script>
canvas.width = canvas.renderedPixelWidth; canvas.height = canvas.renderedPixelHeight;
</script></pre>
* Update canvas size for every frame of a game:
<pre> <script>
function renderFrame() {
canvas.width = canvas.renderedPixelWidth; canvas.height = canvas.renderedPixelHeight;
...
window.requestAnimationFrame(renderFrame);
}
</script></pre>
* Update canvas size and re-render only when DPI changes:
<pre> <script>
function render() {
canvas.width = canvas.renderedPixelWidth; canvas.height = canvas.renderedPixelHeight;
...
}
canvas.addEventHandler("renderedsizechange", render, false);
</script></pre>

== Issues: ==

* Hard to determine a rendered size when css size is 'auto', or when the canvas is not attached to the DOM. Perhaps in those cases the current intrinsic size should be returned? (junov@chromium.org)
* Presumably the layout must be updated when the rendered size attributes are evaluated(maybe that should be stated explicitly in the spec?). Assuming that is the case, the layout update may result in a "renderedsizechange" event being fired. It seems weird that this event would be fired as a result of reading the rendered pixel size. Maybe that is fine... (junov@chromium.org)
* After a layout change that affects rendered pixel size, there is no guarantee that the size change event will be handled before the layout change is propagated to screen, so the content may be temporarily displayed in an inconsistent state. Note: the same issue exists with existing methods that may be based on mutation observers or window.onresize, for example. Though it is not the stated objective of this proposal to solve this problem, there may be an opportunity to do so. (junov)

[[Category:Proposals]]

CanvasRenderedPixelSize

2015-04-10T21:33:20Z

Junov: /* Issues: */

:''This proposal is to allow Web developers to match canvas backing store resolution to device resolution.''

== Use Case Descriptions ==

Use case #1: a Web application needs to render content to a canvas (using 2D or WebGL contexts) being displayed on a high-DPI screen. For best quality output, the pixels of the canvas backing store need to match 1:1 with screen pixels.

Use case #2: As #1, and the user moves the page from screen to screen resulting in dynamic DPI changes.

Use case #3: As #1, and the canvas is inside a CSS scale transform which may change dynamically.

Use case #4: As #1, and the canvas's CSS content box has arbitrary fractional CSS pixel size and offset from the viewport.

Use case #5: As #3, but the canvas is inside an <code><iframe></code> in a page with a different origin which contains the CSS transform.

=== Current Usage and Workarounds ===

Safari shipped a canvas implementation that automatically resized the backing store but this caused various problems, such as incompatibility with some existing content and a need for new "HD" APIs that authors were not motivated to use. This approach does not work for WebGL in any case.

The current best workaround is to set the canvas backing store size by computing the canvas content box in CSS pixels relative to the viewport, as accurately as possible (e.g. using <code>getBoxQuads</code>), multiplying by <code>window.devicePixelRatio</code>, and using that to set the canvas width and height. This approach does not easily adapt to dynamic changes and does not handle use cases #4 or #5. In use-case #4 the computed device pixel size may be non-integer, in which case the UA is likely to snap the content-box to device pixel edges and the Web developer cannot predict how that snapping will be done. In use-case #5 the developer cannot know about the CSS transform in the foreign-origin content.

== Goals ==

# Leave the application in control of setting the pixel size of the backing store.
# Retain the invariant that 1 unit in canvas coordinate space is 1 backing store pixel.
# Make it easy for applications to set the pixel size of the backing store so that backing store pixels are 1:1 with device pixels, when that's possible.

== Non Goals ==

# Enable high-resolution backing store for existing content.

== Proposed Solutions ==

Hixie proposed a canvas attribute to opt into automatic sizing of the backing store, but this violates goals #1 and #2. Violating goal #1 is a problem because applications often use temporary canvases and other assets which are dependent on the pixel size of the backing store. Goal #1 is also essential for WebGL.

=== Suggested Solution ===

Expose a preferred size for the canvas backing store directly to Web authors. Web authors who want to match screen resolution set the canvas width and height to that size. Expose an event on the canvas that fires when the preferred size changes.

== Suggested IDL ==

Add two new DOM attributes to <code>HTMLCanvasElement</code>:

<pre>partial interface HTMLCanvasElement {
readonly attribute long renderedPixelWidth;
readonly attribute long renderedPixelHeight;
};</pre>

The values of these attributes are not normatively defined. For example, a UA might reduce these values to conserve system resources.
However, under normal steady-state conditions a UA is expected to choose values such that, if the canvas element has a CSS background color exactly filling its content-box, and the UA renders that CSS background color as a rectangle whose edges are aligned to device pixel edges, then <code>renderedPixelWidth</code> and <code>renderedPixelHeight</code> are the device pixel size of that rectangle.

Add a new event <code>renderedsizechange</code> to <code>HTMLCanvasElement</code>. This event does not bubble and is not cancelable. Whenever the value that would be returned by <code>renderedPixelWidth</code> or <code>renderedPixelHeight</code> changes, queue a task to fire <code>renderedsizechange</code> at the <code>HTMLCanvasElement</code> if there is not already a task pending to fire such an event at that element.

== Rationale: ==

This API appears to be the minimal API needed to achieve the goals, is opt-in, and can be used in several different ways depending on how ambitious the app developer is:
* Simplest possible usage: set the canvas size once when loading and don't handle dynamic changes.
<pre> <script>
canvas.width = canvas.renderedPixelWidth; canvas.height = canvas.renderedPixelHeight;
</script></pre>
* Update canvas size for every frame of a game:
<pre> <script>
function renderFrame() {
canvas.width = canvas.renderedPixelWidth; canvas.height = canvas.renderedPixelHeight;
...
window.requestAnimationFrame(renderFrame);
}
</script></pre>
* Update canvas size and re-render only when DPI changes:
<pre> <script>
function render() {
canvas.width = canvas.renderedPixelWidth; canvas.height = canvas.renderedPixelHeight;
...
}
canvas.addEventHandler("renderedsizechange", render, false);
</script></pre>

== Issues: ==

* Hard to determine a rendered size when css size is 'auto', or when the canvas is not attached to the DOM. Perhaps in those cases the current intrinsic size should be returned? (junov@chromium.org)
* Presumably the layout must be updated when the rendered size attributes are evaluated(maybe that should be stated explicitly in the spec?). Assuming that is the case, the layout update may result in a "renderedsizechange" event being fired. It seems weird that this event would be fired as a result of reading the rendered pixel size. Maybe that is fine... (junov@chromium.org)

[[Category:Proposals]]

CanvasRenderedPixelSize

2015-04-10T20:48:48Z

Junov: /* Issues: */

:''This proposal is to allow Web developers to match canvas backing store resolution to device resolution.''

== Use Case Descriptions ==

Use case #1: a Web application needs to render content to a canvas (using 2D or WebGL contexts) being displayed on a high-DPI screen. For best quality output, the pixels of the canvas backing store need to match 1:1 with screen pixels.

Use case #2: As #1, and the user moves the page from screen to screen resulting in dynamic DPI changes.

Use case #3: As #1, and the canvas is inside a CSS scale transform which may change dynamically.

Use case #4: As #1, and the canvas's CSS content box has arbitrary fractional CSS pixel size and offset from the viewport.

Use case #5: As #3, but the canvas is inside an <code><iframe></code> in a page with a different origin which contains the CSS transform.

=== Current Usage and Workarounds ===

Safari shipped a canvas implementation that automatically resized the backing store but this caused various problems, such as incompatibility with some existing content and a need for new "HD" APIs that authors were not motivated to use. This approach does not work for WebGL in any case.

The current best workaround is to set the canvas backing store size by computing the canvas content box in CSS pixels relative to the viewport, as accurately as possible (e.g. using <code>getBoxQuads</code>), multiplying by <code>window.devicePixelRatio</code>, and using that to set the canvas width and height. This approach does not easily adapt to dynamic changes and does not handle use cases #4 or #5. In use-case #4 the computed device pixel size may be non-integer, in which case the UA is likely to snap the content-box to device pixel edges and the Web developer cannot predict how that snapping will be done. In use-case #5 the developer cannot know about the CSS transform in the foreign-origin content.

== Goals ==

# Leave the application in control of setting the pixel size of the backing store.
# Retain the invariant that 1 unit in canvas coordinate space is 1 backing store pixel.
# Make it easy for applications to set the pixel size of the backing store so that backing store pixels are 1:1 with device pixels, when that's possible.

== Non Goals ==

# Enable high-resolution backing store for existing content.

== Proposed Solutions ==

Hixie proposed a canvas attribute to opt into automatic sizing of the backing store, but this violates goals #1 and #2. Violating goal #1 is a problem because applications often use temporary canvases and other assets which are dependent on the pixel size of the backing store. Goal #1 is also essential for WebGL.

=== Suggested Solution ===

Expose a preferred size for the canvas backing store directly to Web authors. Web authors who want to match screen resolution set the canvas width and height to that size. Expose an event on the canvas that fires when the preferred size changes.

== Suggested IDL ==

Add two new DOM attributes to <code>HTMLCanvasElement</code>:

<pre>partial interface HTMLCanvasElement {
readonly attribute long renderedPixelWidth;
readonly attribute long renderedPixelHeight;
};</pre>

The values of these attributes are not normatively defined. For example, a UA might reduce these values to conserve system resources.
However, under normal steady-state conditions a UA is expected to choose values such that, if the canvas element has a CSS background color exactly filling its content-box, and the UA renders that CSS background color as a rectangle whose edges are aligned to device pixel edges, then <code>renderedPixelWidth</code> and <code>renderedPixelHeight</code> are the device pixel size of that rectangle.

Add a new event <code>renderedsizechange</code> to <code>HTMLCanvasElement</code>. This event does not bubble and is not cancelable. Whenever the value that would be returned by <code>renderedPixelWidth</code> or <code>renderedPixelHeight</code> changes, queue a task to fire <code>renderedsizechange</code> at the <code>HTMLCanvasElement</code> if there is not already a task pending to fire such an event at that element.

== Rationale: ==

This API appears to be the minimal API needed to achieve the goals, is opt-in, and can be used in several different ways depending on how ambitious the app developer is:
* Simplest possible usage: set the canvas size once when loading and don't handle dynamic changes.
<pre> <script>
canvas.width = canvas.renderedPixelWidth; canvas.height = canvas.renderedPixelHeight;
</script></pre>
* Update canvas size for every frame of a game:
<pre> <script>
function renderFrame() {
canvas.width = canvas.renderedPixelWidth; canvas.height = canvas.renderedPixelHeight;
...
window.requestAnimationFrame(renderFrame);
}
</script></pre>
* Update canvas size and re-render only when DPI changes:
<pre> <script>
function render() {
canvas.width = canvas.renderedPixelWidth; canvas.height = canvas.renderedPixelHeight;
...
}
canvas.addEventHandler("renderedsizechange", render, false);
</script></pre>

== Issues: ==

* Hard to determine a rendered size when css size is 'auto', or when the canvas is not attached to the DOM. Perhaps in those cases the current intrinsic size should be returned? (junov@chromium.org)

[[Category:Proposals]]

Talk:WorkerCanvas

2015-03-20T15:36:13Z

Junov: Created page with "Why do ImageBitmaps need to be transferable? The object are immutable, so I think it is Okay for threads to reference the same object without cloning."

Why do ImageBitmaps need to be transferable? The object are immutable, so I think it is Okay for threads to reference the same object without cloning.

Canvas Batch drawImage

2014-08-04T17:19:42Z

Junov:

:Batching drawImage calls to achieve near-native performance for sprite/image based animations and games.

== Use Case Description ==
Many web applications and games use 2D Canvases and the drawImage method to bring sprites to screen. Often, a large number of calls to drawImage occur for each frame presented to screen.

=== Current Limitations ===
When calling drawImage hundreds or thousands of times per animation frame, API bindings overhead and internal bookkeeping costs associated with individual draw calls become a significant performance bottleneck, which prevents web applications from achieving near-native performance levels.

=== Current Workaround ===
With WebGL, batching sprite draws is possible thanks to vertex buffers. WebGL is not as broadly supported as 2D canvas, and is a considerably more complex solution to a problem that could otherwise be solved with a 2D canvas.

=== Benefits ===

Rendering Performance.

=== Requests for this Feature ===

* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/RzHob5VS8Dg]</cite> <blockquote><p>On Nexus 7 device, it can reach 60fps for 1000 sprites drawing at the same time [referring to a native canvas implementation]. However, the fps is very poor (about 9fps) if the demo is running in Chromium 36.</p></blockquote>

== Proposed Solution ==

=== Batch versions of drawImage ===
:New batch variants of drawImage that accept array arguments.

==== Processing Model ====
:We would have new variants of the existing drawImage method where the numerical arguments are are packed into a Float32Array. The image argument may or may not be an array. If it is not an array, the same source image is used for for each draw. When rendering from a single sprite sheet, it would be preferable to not specify the image argument as an array in order to minimize bindings overhead and redundant parameter validation.

==== Suggested IDL ====

<pre>

enum CanvasDrawImageParameterFormat { position, destination-rectangle, source-and-destination-rectangles, source-rectangle-and-transform};

interface CanvasRenderingContext2D {
...
void drawImageBatch(sequence<CanvasImageSource> image, CanvasDrawImageParameterFormat parameterFormat, Float32Array drawParameters);
void drawImageBatch(CanvasImageSource image, ParameterFormat parameterFormat, Float32Array drawParameters);
}
</pre>

:The drawParameters argument is to be interpreted as a table in row-major order. Where each row represents a single draw and each column represents an individual parameter. The mapping of the columns to draw parameters depends on the parameterFormat argument.
:* position: dx, dy
:* destination-rectangle: dx, dy, dw, dh
:* source-and-destination-rectangles: sx, sy, sw, sh, dx, dy, dw, dh
:* source-rectangle-and-transform: sx, sy, sw, sh, a, b, c, d, e, f

:The parameters sx, sy, sw, sh, dx, dy, dw, and dh have the same meaning as with drawImage()
:The parameters a, b, c, d, e, f have the same meaning as with transform()
:With 'source-rectangle-and-transform' the destination rectangle is a unit square defined by the vertices (0, 0), (1, 0), (1, 1), and (0, 1). The destination rectangle is transformed by the transform defined by a, b, c, d, e and f, and the by the canvas's current transform.

Feedback Anne: Perhaps rather than overloading we should introduce new methods? IDL overloading is somewhat costly and not loved much by the JS community. Also, please float this by public-script-coord@w3.org at some point.

==== Exceptions ====

:An INDEX_SIZE_ERR DOM exception is thrown if the size of drawParameters is not a multiple of the number of numeric parameters corresponding to 'parameterFormat'
:An INDEX_SIZE_ERR DOM exception is thrown if 'image' is a sequence and the size of drawParameters is not equal to the number of elements in 'image' multiplied by the number of parameters corresponding to 'parameterFormat'.
:All the same exceptions that apply for calls to drawImage also apply to drawImageBatch. If any individual draw results in an exception being thrown, the entire call to drawImageBatch must abort without drawing anything to the canvas.

==== Limitations ====

:The use of Float32Array is not very programmer friendly. This compromise on usability is justified by performance considerations, and is necessary to achieve near-native or WebGL-like performance.

==== Implementation ====
* The most naive implementation, which would consist in expanding the batch drawImage call into multiple internal drawImage calls would already increase performance by reducing API bindings overhead.
* The use of typed arrays will dramatically reduce the argument type checking burden in the bindings.
* More advanced implementations would carry the batching down to a lower level in the graphics stack. For example, a GPU-accelerated implementation of 2D canvas could use OpenGL vertex buffer objects, or the DirectX sprite interface.
* Some existing implementations of drawImage already detect batching opportunities and will batch consecutive drawImage calls at a lower lever of the graphics stack (for example skia's drawBitmap used in Blink). This auto-detection does improve rasterization performance, but it does not eliminate the bindings overhead and it involves additional overhead to determine whether consecutive calls to drawImage can be grouped together to form a batch. With explicitly batched calls to drawImage, that overhead can be eliminated because the individuals sprite draws would be known to use identical rendering context state, and the calls down to the graphics platform implementation layer would only occur once for the entire batch.

==== Adoption ====
:Game/app devs looking for high frame rates for sprite blitting are likely to adopt enthusiastically.

[[Category:Proposals]]

Canvas Batch drawImage

2014-08-04T17:14:46Z

Junov:

:Batching drawImage calls to achieve near-native performance for sprite/image based animations and games.

== Use Case Description ==
Many web applications and games use 2D Canvases and the drawImage method to bring sprites to screen. Often, a large number of calls to drawImage occur for each frame presented to screen.

=== Current Limitations ===
When calling drawImage hundreds or thousands of times per animation frame, API bindings overhead and internal bookkeeping costs associated with individual draw calls become a significant performance bottleneck, which prevents web applications from achieving near-native performance levels.

=== Current Workaround ===
With WebGL, batching sprite draws is possible thanks to vertex buffers. WebGL is not as broadly supported as 2D canvas, and is a considerably more complex solution to a problem that could otherwise be solved with a 2D canvas.

=== Benefits ===

Rendering Performance.

=== Requests for this Feature ===

* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/RzHob5VS8Dg]</cite> <blockquote><p>On Nexus 7 device, it can reach 60fps for 1000 sprites drawing at the same time [referring to a native canvas implementation]. However, the fps is very poor (about 9fps) if the demo is running in Chromium 36.</p></blockquote>

== Proposed Solution ==

=== Batch versions of drawImage ===
:New batch variants of drawImage that accept array arguments.

==== Processing Model ====
:We would have new variants of the existing drawImage method where the numerical arguments are are packed into a Float32Array. The image argument may or may not be an array. If it is not an array, the same source image is used for for each draw. When rendering from a single sprite sheet, it would be preferable to not specify the image argument as an array in order to minimize bindings overhead and redundant parameter validation.

==== Suggested IDL ====

<pre>

enum CanvasDrawImageParameterFormat { position, destination-rectangle, source-and-destination-rectangles, source-rectangle-and-transform};

interface CanvasRenderingContext2D {
...
void drawImageBatch(sequence<CanvasImageSource> image, CanvasDrawImageParameterFormat parameterFormat, Float32Array drawParameters);
void drawImageBatch(CanvasImageSource image, ParameterFormat parameterFormat, Float32Array drawParameters);
}
</pre>

:The drawParameters argument is to be interpreted as a table in row-major order. Where each row represents a single draw and each column represents an individual parameter. The mapping of the columns to draw parameters depends on the parameterFormat argument.
:* position: dx, dy
:* destination-rectangle: dx, dy, dw, dh
:* source-and-destination-rectangles: sx, sy, sw, sh, dx, dy, dw, dh
:* source-rectangle-and-transform: sx, sy, sw, sh, a, b, c, d, e, f

:The parameters sx, sy, sw, sh, dx, dy, dw, and dh have the same meaning as with drawImage()
:The parameters a, b, c, d, e, f have the same meaning as with transform()
:With 'source-rectangle-and-transform' the destination rectangle is defined by the vertices (0, 0), (sw, 0), (sw, sh), and (0, sh). The destination rectangle is transformed by the transform defined by a, b, c, d, e and f, and the by the canvas's current transform.

Feedback Anne: Perhaps rather than overloading we should introduce new methods? IDL overloading is somewhat costly and not loved much by the JS community. Also, please float this by public-script-coord@w3.org at some point.

==== Exceptions ====

:An INDEX_SIZE_ERR DOM exception is thrown if the size of drawParameters is not a multiple of the number of numeric parameters corresponding to 'parameterFormat'
:An INDEX_SIZE_ERR DOM exception is thrown if 'image' is a sequence and the size of drawParameters is not equal to the number of elements in 'image' multiplied by the number of parameters corresponding to 'parameterFormat'.
:All the same exceptions that apply for calls to drawImage also apply to drawImageBatch. If any individual draw results in an exception being thrown, the entire call to drawImageBatch must abort without drawing anything to the canvas.

==== Limitations ====

:The use of Float32Array is not very programmer friendly. This compromise on usability is justified by performance considerations, and is necessary to achieve near-native or WebGL-like performance.

==== Implementation ====
* The most naive implementation, which would consist in expanding the batch drawImage call into multiple internal drawImage calls would already increase performance by reducing API bindings overhead.
* The use of typed arrays will dramatically reduce the argument type checking burden in the bindings.
* More advanced implementations would carry the batching down to a lower level in the graphics stack. For example, a GPU-accelerated implementation of 2D canvas could use OpenGL vertex buffer objects, or the DirectX sprite interface.
* Some existing implementations of drawImage already detect batching opportunities and will batch consecutive drawImage calls at a lower lever of the graphics stack (for example skia's drawBitmap used in Blink). This auto-detection does improve rasterization performance, but it does not eliminate the bindings overhead and it involves additional overhead to determine whether consecutive calls to drawImage can be grouped together to form a batch. With explicitly batched calls to drawImage, that overhead can be eliminated because the individuals sprite draws would be known to use identical rendering context state, and the calls down to the graphics platform implementation layer would only occur once for the entire batch.

==== Adoption ====
:Game/app devs looking for high frame rates for sprite blitting are likely to adopt enthusiastically.

[[Category:Proposals]]

Canvas Batch drawImage

2014-08-04T17:14:00Z

Junov: /* Proposed Solution */

:Batching drawImage calls to achieve near-native performance for sprite/image based animations and games.

== Use Case Description ==
Many web applications and games use 2D Canvases and the drawImage method to bring sprites to screen. Often, a large number of calls to drawImage occur for each frame presented to screen.

=== Current Limitations ===
When calling drawImage hundreds or thousands of times per animation frame, API bindings overhead and internal bookkeeping costs associated with individual draw calls become a significant performance bottleneck, which prevents web applications from achieving near-native performance levels.

=== Current Workaround ===
With WebGL, batching sprite draws is possible thanks to vertex buffers. WebGL is not as broadly supported as 2D canvas, and is a considerably more complex solution to a problem that could otherwise be solved with a 2D canvas.

=== Benefits ===

Rendering Performance.

=== Requests for this Feature ===

* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/RzHob5VS8Dg]</cite> <blockquote><p>On Nexus 7 device, it can reach 60fps for 1000 sprites drawing at the same time [referring to a native canvas implementation]. However, the fps is very poor (about 9fps) if the demo is running in Chromium 36.</p></blockquote>

== Proposed Solution ==

=== Batch versions of drawImage ===
:New batch variants of drawImage that accept array arguments.

==== Processing Model ====
:We would have new variants of the existing drawImage method where the numerical arguments are are packed into a Float32Array. The image argument may or may not be an array. If it is not an array, the same source image is used for for each draw. When rendering from a single sprite sheet, it would be preferable to not specify the image argument as an array in order to minimize bindings overhead and redundant parameter validation.

==== Suggested IDL ====

<pre>

enum CanvasDrawImageParameterFormat { position, destination-rectangle, source-and-destination-rectangles, source-rectangle-and-transform};

interface CanvasRenderingContext2D {
...
void drawImageBatch(sequence<CanvasImageSource> image, CanvasDrawImageParameterFormat parameterFormat, Float32Array drawParameters);
void drawImageBatch(CanvasImageSource image, ParameterFormat parameterFormat, Float32Array drawParameters);
}
</pre>

:The drawParameters argument is to be interpreted as a table in row-major order. Where each row represents a single draw and each column represents an individual parameter. The mapping of the columns to draw parameters depends on the parameterFormat argument.
:* position: dx, dy
:* destination-rectangle: dx, dy, dw, dh
:* source-and-destination-rectangles: sx, sy, sw, sh, dx, dy, dw, dh
:* source-rectangle-and-transform: sx, sy, sw, sh, a, b, c, d, e, f

:The parameters sx, sy, sw, sh, dx, dy, dw, and dh have the same meaning as with drawImage()
:The parameters a, b, c, d, e, f have the same meaning as with transform()
:With 'source-rectangle-and-transform' the destination rectangle is defined by the vertices (0, 0), (sw, 0), (sw, sh), and (0, sh). The destination rectangle is transformed by the transform defined by a, b, c, d, e and f, and the by the canvas's current transform.

Feedback Anne: Perhaps rather than overloading we should introduce new methods? IDL overloading is somewhat costly and not loved much by the JS community. Also, please float this by public-script-coord@w3.org at some point.

==== Exceptions ====

:An INDEX_SIZE_ERR DOM exception is thrown if the size of drawParameters is not a multiple of the number of numeric parameters corresponding to 'parameterFormat'
:An INDEX_SIZE_ERR DOM exception is thrown if 'image' is a sequence and the size of drawParameters is not equal to the number of elements in 'image' multiplied by the number of parameters corresponding to 'parameterFormat'.
All the same exceptions that apply for calls to drawImage also apply to drawImageBatch. If any individual draw results in an exception being thrown, the entire call to drawImageBatch must abort without drawing anything to the canvas.

==== Limitations ====

:The use of Float32Array is not very programmer friendly. This compromise on usability is justified by performance considerations, and is necessary to achieve near-native or WebGL-like performance.

==== Implementation ====
* The most naive implementation, which would consist in expanding the batch drawImage call into multiple internal drawImage calls would already increase performance by reducing API bindings overhead.
* The use of typed arrays will dramatically reduce the argument type checking burden in the bindings.
* More advanced implementations would carry the batching down to a lower level in the graphics stack. For example, a GPU-accelerated implementation of 2D canvas could use OpenGL vertex buffer objects, or the DirectX sprite interface.
* Some existing implementations of drawImage already detect batching opportunities and will batch consecutive drawImage calls at a lower lever of the graphics stack (for example skia's drawBitmap used in Blink). This auto-detection does improve rasterization performance, but it does not eliminate the bindings overhead and it involves additional overhead to determine whether consecutive calls to drawImage can be grouped together to form a batch. With explicitly batched calls to drawImage, that overhead can be eliminated because the individuals sprite draws would be known to use identical rendering context state, and the calls down to the graphics platform implementation layer would only occur once for the entire batch.

==== Adoption ====
:Game/app devs looking for high frame rates for sprite blitting are likely to adopt enthusiastically.

[[Category:Proposals]]

Canvas Batch drawImage

2014-08-04T17:05:09Z

Junov: /* Use Case Description */

:Batching drawImage calls to achieve near-native performance for sprite/image based animations and games.

== Use Case Description ==
Many web applications and games use 2D Canvases and the drawImage method to bring sprites to screen. Often, a large number of calls to drawImage occur for each frame presented to screen.

=== Current Limitations ===
When calling drawImage hundreds or thousands of times per animation frame, API bindings overhead and internal bookkeeping costs associated with individual draw calls become a significant performance bottleneck, which prevents web applications from achieving near-native performance levels.

=== Current Workaround ===
With WebGL, batching sprite draws is possible thanks to vertex buffers. WebGL is not as broadly supported as 2D canvas, and is a considerably more complex solution to a problem that could otherwise be solved with a 2D canvas.

=== Benefits ===

Rendering Performance.

=== Requests for this Feature ===

* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/RzHob5VS8Dg]</cite> <blockquote><p>On Nexus 7 device, it can reach 60fps for 1000 sprites drawing at the same time [referring to a native canvas implementation]. However, the fps is very poor (about 9fps) if the demo is running in Chromium 36.</p></blockquote>

== Proposed Solution ==

=== Batch versions of drawImage ===
:Additional overloads of drawImage that accept array arguments.

==== Processing Model ====
:We would have new variants of the existing drawImage overloads where the numerical arguments are are packed into a Float32Array. The image argument may or may not be an array. if it is not an array, the same source image is used for for each draw. When rendering from a single sprite sheet, it would be preferable to not specify the image argument as an array in order to minimize bindings overhead.

:An INDEX_SIZE_ERR DOM the Float32Array size is not a multiple of the number of numeric parameter or if it does not match the size of the image argument, if image is an array.

==== Suggested IDL ====

<pre>

enum CanvasDrawImageParameterFormat { position, destination-rectangle, source-and-destination-rectangles, source-rectangle-and-transform};

interface CanvasRenderingContext2D {
...
void drawImageBatch(sequence<CanvasImageSource> image, CanvasDrawImageParameterFormat parameterFormat, Float32Array drawParameters);
void drawImageBatch(CanvasImageSource image, ParameterFormat parameterFormat, Float32Array drawParameters);
}
</pre>

:The drawParameters argument is to be interpreted as a table in row-major order. Where each row represents a single draw and each column represents an individual parameter. The mapping of the columns to draw parameters depends on the parameterFormat argument.
:* position: dx, dy
:* destination-rectangle: dx, dy, dw, dh
:* source-and-destination-rectangles: sx, sy, sw, sh, dx, dy, dw, dh
:* source-rectangle-and-transform: sx, sy, sw, sh, a, b, c, d, e, f

:The parameters sx, sy, sw, sh, dx, dy, dw, and dh have the same meaning as with drawImage()
:The parameters a, b, c, d, e, f have the same meaning as with transform()
:With 'source-rectangle-and-transform' the destination rectangle is defined by the vertices (0, 0), (sw, 0), (sw, sh), and (0, sh). The destination rectangle is transformed by the transform defined by a, b, c, d, e and f, and the by the canvas's current transform.

Feedback Anne: Perhaps rather than overloading we should introduce new methods? IDL overloading is somewhat costly and not loved much by the JS community. Also, please float this by public-script-coord@w3.org at some point.

==== Limitations ====
:The use of Float32Array is not very programmer friendly. This compromise on usability is justified by performance considerations, and is necessary to achieve near-native or WebGL-like performance.

==== Implementation ====
* The most naive implementation, which would consist in expanding the batch drawImage call into multiple internal drawImage calls would already increase performance by reducing API bindings overhead.
* The use of typed arrays will dramatically reduce the argument type checking burden in the bindings.
* More advanced implementations would carry the batching down to a lower level in the graphics stack. For example, a GPU-accelerated implementation of 2D canvas could use OpenGL vertex buffer objects, or the DirectX sprite interface.
* Some existing implementations of drawImage already detect batching opportunities and will batch consecutive drawImage calls at a lower lever of the graphics stack (for example skia's drawBitmap used in Blink). This auto-detection does improve rasterization performance, but it does not eliminate the bindings overhead and it involves additional overhead to determine whether consecutive calls to drawImage can be grouped together to form a batch. With explicitly batched calls to drawImage, that overhead can be eliminated because the individuals sprite draws would be known to use identical rendering context state, and the calls down to the graphics platform implementation layer would only occur once for the entire batch.

==== Adoption ====
:Game/app devs looking for high frame rates for sprite blitting are likely to adopt enthusiastically.

[[Category:Proposals]]

Canvas Batch drawImage

2014-08-04T17:04:10Z

Junov: /* Batch versions of drawImage */

:Batching drawImage calls to achieve near-native performance for sprite/image based animations and games.

== Use Case Description ==
Many web applications and games use 2D Canvases and the drawImage method to bring sprites to screen. Often, a large number of calls to drawImage occur for each frame presented to screen.

=== Current Limitations ===
When calling drawImage hundreds or thousands of times per animation frame, API bindings overhead and internal bookkeeping costs associated with individual draw calls become a significant performance bottleneck, which prevents web application from achieving near-native performance levels.

=== Current Workaround ===
With WebGL, batching sprite draws is possible thanks to vertex buffers. WebGL is not as broadly supported as 2D canvas, and is a considerably more complex solution to a problem that could otherwise be solved with a 2D canvas.

=== Benefits ===

Rendering Performance.

=== Requests for this Feature ===

* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/RzHob5VS8Dg]</cite> <blockquote><p>On Nexus 7 device, it can reach 60fps for 1000 sprites drawing at the same time [referring to a native canvas implementation]. However, the fps is very poor (about 9fps) if the demo is running in Chromium 36.</p></blockquote>

== Proposed Solution ==

=== Batch versions of drawImage ===
:Additional overloads of drawImage that accept array arguments.

==== Processing Model ====
:We would have new variants of the existing drawImage overloads where the numerical arguments are are packed into a Float32Array. The image argument may or may not be an array. if it is not an array, the same source image is used for for each draw. When rendering from a single sprite sheet, it would be preferable to not specify the image argument as an array in order to minimize bindings overhead.

:An INDEX_SIZE_ERR DOM the Float32Array size is not a multiple of the number of numeric parameter or if it does not match the size of the image argument, if image is an array.

==== Suggested IDL ====

<pre>

enum CanvasDrawImageParameterFormat { position, destination-rectangle, source-and-destination-rectangles, source-rectangle-and-transform};

interface CanvasRenderingContext2D {
...
void drawImageBatch(sequence<CanvasImageSource> image, CanvasDrawImageParameterFormat parameterFormat, Float32Array drawParameters);
void drawImageBatch(CanvasImageSource image, ParameterFormat parameterFormat, Float32Array drawParameters);
}
</pre>

:The drawParameters argument is to be interpreted as a table in row-major order. Where each row represents a single draw and each column represents an individual parameter. The mapping of the columns to draw parameters depends on the parameterFormat argument.
:* position: dx, dy
:* destination-rectangle: dx, dy, dw, dh
:* source-and-destination-rectangles: sx, sy, sw, sh, dx, dy, dw, dh
:* source-rectangle-and-transform: sx, sy, sw, sh, a, b, c, d, e, f

:The parameters sx, sy, sw, sh, dx, dy, dw, and dh have the same meaning as with drawImage()
:The parameters a, b, c, d, e, f have the same meaning as with transform()
:With 'source-rectangle-and-transform' the destination rectangle is defined by the vertices (0, 0), (sw, 0), (sw, sh), and (0, sh). The destination rectangle is transformed by the transform defined by a, b, c, d, e and f, and the by the canvas's current transform.

Feedback Anne: Perhaps rather than overloading we should introduce new methods? IDL overloading is somewhat costly and not loved much by the JS community. Also, please float this by public-script-coord@w3.org at some point.

==== Limitations ====
:The use of Float32Array is not very programmer friendly. This compromise on usability is justified by performance considerations, and is necessary to achieve near-native or WebGL-like performance.

==== Implementation ====
* The most naive implementation, which would consist in expanding the batch drawImage call into multiple internal drawImage calls would already increase performance by reducing API bindings overhead.
* The use of typed arrays will dramatically reduce the argument type checking burden in the bindings.
* More advanced implementations would carry the batching down to a lower level in the graphics stack. For example, a GPU-accelerated implementation of 2D canvas could use OpenGL vertex buffer objects, or the DirectX sprite interface.
* Some existing implementations of drawImage already detect batching opportunities and will batch consecutive drawImage calls at a lower lever of the graphics stack (for example skia's drawBitmap used in Blink). This auto-detection does improve rasterization performance, but it does not eliminate the bindings overhead and it involves additional overhead to determine whether consecutive calls to drawImage can be grouped together to form a batch. With explicitly batched calls to drawImage, that overhead can be eliminated because the individuals sprite draws would be known to use identical rendering context state, and the calls down to the graphics platform implementation layer would only occur once for the entire batch.

==== Adoption ====
:Game/app devs looking for high frame rates for sprite blitting are likely to adopt enthusiastically.

[[Category:Proposals]]

Canvas Batch drawImage

2014-07-30T21:25:14Z

Junov:

:Batching drawImage calls to achieve near-native performance for sprite/image based animations and games.

== Use Case Description ==
Many web applications and games use 2D Canvases and the drawImage method to bring sprites to screen. Often, a large number of calls to drawImage occur for each frame presented to screen.

=== Current Limitations ===
When calling drawImage hundreds or thousands of times per animation frame, API bindings overhead and internal bookkeeping costs associated with individual draw calls become a significant performance bottleneck, which prevents web application from achieving near-native performance levels.

=== Current Workaround ===
With WebGL, batching sprite draws is possible thanks to vertex buffers. WebGL is not as broadly supported as 2D canvas, and is a considerably more complex solution to a problem that could otherwise be solved with a 2D canvas.

=== Benefits ===

Rendering Performance.

=== Requests for this Feature ===

* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/RzHob5VS8Dg]</cite> <blockquote><p>On Nexus 7 device, it can reach 60fps for 1000 sprites drawing at the same time [referring to a native canvas implementation]. However, the fps is very poor (about 9fps) if the demo is running in Chromium 36.</p></blockquote>

== Proposed Solution ==

=== Batch versions of drawImage ===
:Additional overloads of drawImage that accept array arguments.

==== Processing Model ====
:We would have new variants of the existing drawImage overloads where all the arguments are arrays (Float32Array for numeric arguments), and other variants where all the arguments except for the image are arrays. The case where the image is not an array would be equivalent to having an array where each element is the same image (convenient for sprite sheets, and more efficient than using an array)

:An INDEX_SIZE_ERR DOM exception would be thrown if not all the array arguments have the same size.

==== Suggested IDL ====

<pre>
interface Canvas2DRenderingContext {
...
// existing variants
void drawImage(CanvasImageSource image, unrestricted double dx, unrestricted double dy);
void drawImage(CanvasImageSource image, unrestricted double dx, unrestricted double dy, unrestricted double dw, unrestricted double dh);
void drawImage(CanvasImageSource image, unrestricted double sx, unrestricted double sy, unrestricted double sw, unrestricted double sh, unrestricted double dx, unrestricted double dy, unrestricted double dw, unrestricted double dh);
// batch variants
void drawImage(CanvasImageSource image[], Float32Array dx, Float32Array dy);
void drawImage(CanvasImageSource image[], Float32Array dx, Float32Array dy, Float32Array dw, Float32Array dh);
void drawImage(CanvasImageSource image[], Float32Array sx, Float32Array sy, Float32Array sw, Float32Array sh, Float32Array dx, Float32Array dy, Float32Array dw, Float32Array dh);
// single image batch variants
void drawImage(CanvasImageSource image, Float32Array dx, Float32Array dy);
void drawImage(CanvasImageSource image, Float32Array dx, Float32Array dy, Float32Array dw, Float32Array dh);
void drawImage(CanvasImageSource image, Float32Array sx, Float32Array sy, Float32Array sw, Float32Array sh, Float32Array dx, Float32Array dy, Float32Array dw, Float32Array dh);
}
</pre>
==== Limitations ====
:Use cases where per-sprite transforms are *required* are not covered. For example, if sprites are individually rotated. If appropriate, this could be resolved by adding more variants to the proposal (a matrix argument?)

==== Implementation ====
* The most naive implementation, which would consist in expanding the batch drawImage call into multiple internal drawImage calls would already increase performance by reducing API bindings overhead.
* The use of typed arrays will dramatically reduce the argument type checking burden in the bindings.
* More advanced implementations would carry the batching down to a lower level in the graphics stack. For example, a GPU-accelerated implementation of 2D canvas could use OpenGL vertex buffer objects, or the DirectX sprite interface.
* Some existing implementations of drawImage already detect batching opportunities and will batch consecutive drawImage calls at a lower lever of the graphics stack (for example skia's drawBitmap used in Blink). This auto-detection does improve rasterization performance, but it does not eliminate the bindings overhead and it involves additional overhead to determine whether consecutive calls to drawImage can be grouped together to form a batch. With explicitly batched calls to drawImage, that overhead can be eliminated because the individuals sprite draws would be known to use identical rendering context state, and the calls down to the graphics platform implementation layer would only occur once for the entire batch.

==== Adoption ====
:Game/app devs looking for high frame rates for sprite blitting are likely to adopt enthusiastically.

[[Category:Proposals]]

Canvas Batch drawImage

2014-07-30T20:58:02Z

Junov: Created page with ":Batching drawImage calls to achieve near-native performance for sprite/image based animations and games. == Use Case Description == Many web applications and games use 2D Ca..."

Canvas Context Loss and Restoration

2014-03-12T21:31:48Z

Junov:

:2D Canvas Rendering contexts are currently required to have persistent backing stores. This proposal aims to relax that requirement by introducing an API that allows canvases to be discarded by the browser and re-drawn by the web application on demand.

== Use Case Description ==
:The 2d canvas backing store persistence requirement often leads to large amounts of RAM (or GPU memory) being consumed by canvas elements that are not actively used because they are off screen or in background tabs or occluded windows. Other types of RAM-greedy HTML elements can release resources in such cases.

:The expectation of canvas content persistence also makes it very difficult--if not impossible--for many web apps to recover from a GPU context reset in web browsers that store 2D canvas contents in GPU memory.

=== Current Limitations ===
:In theory, web apps do have the capability of discarding canvas backing stores (set canvas size to zero) and regenerating canvas content. However, web apps are not and should not be expected to be responsible for resolving resource contention issues. The browser is responsible for monitoring resource usage and availability and is expected to take all necessary and reasonable measures to avoid crashes, hangs, and catastrophic performance degradations that may be caused by resource contention. Under the current specification, browsers have no options for evicting resources held by 2D canvases because there are no means of guaranteeing that the application will redraw the contents when needed.

=== Current Usage and Workarounds ===
:Web apps can track events to detect when the page is no longer visible (http://www.w3.org/TR/page-visibility/) and deallocate backing stores at that time by setting the size of the canvas element to 0. Conversely, they can detect when the page is visible again and reinitialize at that time

:Web apps can track events that are often associated with GPU context losses (e.g. waking-up from hibernation), and conservatively reinitialize the 2D canvas by resetting the context (set canvas width/height) and redrawing, just in case.

=== Benefits ===
:* Empower the browser to monitor resources to decide whether to drop canvas backing stores and in what order (LRU backgroung tabs?) in order to achieve better performance and stability. If web apps must handle resource eviction themselves, they may often free resources when not necessary, which may lead to unnecessary tab switching lag.
:* Make recovery from GPU context losses more robust.
:* Allow GPU-accelerated 2D canvases on platforms that are known to drop graphics contexts often or unpredictably.

=== Requests for this Feature ===
:* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/CQJXpXxO6dk E-mail thread on Chromium graphics-dev mailing list]</cite> <blockquote><p>"Is there any reason why we don't add a similar optional callback to the 2D context? (in reference to WebGL context loss API)" --Rik Cabanier, Adobe</p></blockquote>

== Proposed Solutions ==
The following solutions were pondered in the discussion thread cited above:
:* Generalize the WebGL context lost / context recovered API, so that it applies to all types of canvases.
:* Add a redraw callback on the canvas element

=== Retained Solution : Upstream the context lost/recovered API form the WebGL specification into the parent canvas specification. ===
:General Concept:
:* a contextlost event is fired after the context is lost.
:* a contextrestored event is fired immediately after a previously lost canvas context is brought back to a usable state. The canvas context is returned to its initial state and the canvas's backing store is blank (transparent black) when restored.

==== Processing Model ====
:Rendering context losses may be intended by the user agent (to resolve resource contention), or may be forced by external factors (e.g. a graphics driver reset).

:For convenience, the lost state should be accessible. To do so, the isContextLost method that is defined in the WebGLRenderingContext API should also exist in the CanvasRenderingContext2D API.

:Losing contexts (applies to 2d contexts, some aspects different for WebGL):
:* The UA is only allowed to lose contexts intentionally if the context was opted-in to have discarable storage (see below).
:* The UA is free to use any set of rules to decide which contexts are dropped when and in what order.
:* The return value of isContextLost() may transition from false to true before the contextlost event is dispatched (like webGL).
:* All objects that depend on the content of the canvas (e.g. patterns, imageBitmaps) are neutered when the context is lost. The neutering propagates through creation dependency chains, so a Pattern created from an ImageBitmap created from and ImageBitmap created from a canvas will be neutered if the canvas's context is lost. This rule makes it safe for implementations to optimize their memory consumption by sharing pixel buffers between objects when possible.
:* The contextlost event does not bubble
:* The contextlost event is cancellable. Cancelling the even has the effect of cancelling the future restoration of the context.

:Restoring contexts (applies to 2d contexts, some aspects different for WebGL):
:* Between the time a context is recovered and invoking the listener for the contextrestored event, no other user code can be executed. This is to prevent race conditions that could result from the rendering context validly executing draw commands before the contextrestored event is dispatched. To respect this constraint, a user agent that restores contexts asynchronously would have to behave as if the context was not yet restored between the time the context is restored internally and the time the contextrestored event listener is called.
:* The app is responsible for re-creating the objects that were neutered when the context was lost.
:* There can only be one contextrestored event pending at a time. When there are multiple canvases to be restored, the next canvas to be restored can only be restored after any pending contextrestored events--from previously restored canvases--have been handled.
:* Context restoration can only be initiated by the user agent (can not by triggered by a script action).
:* A lost context can only be restored after its context lost event has been dispatched. This avoids synchronization inconsistencies with isContextLost().
:* The return value of isContextLost() transitions from true to false at the time that contextrestored event is dispatched (like webGL). So the contextrestored event listener is always the first task to be execute after the transition.
:* The contextrestored event does not bubble and is not cancellable.

:Behavior when using a lost context (applies to 2d contexts, WebGL may behave differently):
:* All draw calls exit without drawing.
:* All rendering context API calls will throw the same exceptions as they would if called on a valid context.
:* All calls that read back canvas pixels from either the canvas element or the canvas rendering context (getImageData, toDataURL, toBlob, createPattern, creatImageBitmap, drawImage with lost canvas as source) will behave as they would if the canvas context were valid and blank (all transparent black pixels).
:* Using a Patterns or an ImageBitmap that was neutered because the context of its source canvas was lost will behave as if the Pattern or ImageBitmap were valid and blank (all transparent black pixels).

:Behaviors specific to GPU-accelerated implementation of 2d canvas (non-normative?)
:* If the context was lost due to a GPU-related failure, and the browser is actively restoring GPU functionality, and expects to restore in a timely manner, then context restoration should wait until the browser is ready to resume GPU functionality and the restored canvas should continue to be GPU-accelerated. Conversely, if GPU functionality is permanently disable or if it is unknow whether or how long it may take to resume GPU operation, then the canvas should be restored immediately without GPU-acceleration
:* If an accelerated canvas is resized while the GPU is temporarily unavailable, the creation of the new canvas buffer should be postponed until the GPU functionality is restored.
:* When getContext() is called while GPU functionality is temporarily unavailable:
:: a) If the canvas does not have an associated context, create an new unaccelerated context.
:: b) If the canvas already has an associate context, return the existing context even if it is in a lost state.

:Opting-in to discardable backing stores.
:* A new context creation parameter named 'storage' is to be added. Possible values are 'persistent' and 'discardable'. Usage: canvas.getContext('2d', {storage: 'discardable'});
:* In 'persistent' mode the UA may not elect to evict the canvas's backing store.
:* The default is 'persistent', which provides backwards compatible behavior for application that expect canvas backing store content to never be lost.
:* Note: This parameter is a string rather than a boolean because it is anticipated that additional storage modes may be added in the future.

==== Limitations ====
:Web browsers will not reap the stability and performance rewards associated with this API with Web apps that do not provide at least a handler for the context recovered event. In order for this feature to improve the state of the web, apps will need to opt in to this new API, which unfortunately needs to be optional for backwards compatibility reasons.

==== Implementation ====
:Browser vendors should be highly motivated to implement this API in order to improve platform resilience and performance, especially on mobile platforms where RAM contention is often a critical issue.

==== Adoption ====
:Web App developers should be motivated to adopt this API in order to improve the stability of their products:
:* Avoid triggering out-of-memory crashes in the browser.
:* Avoid browser performance issues associated with having a given app in a background tab
:* Robustly recover from GPU failures.

:Top tier apps (particularly from developers that track usage metrics) would be expected to enthusiastically adopt this API.

==== Specification ====
:WebGL would continue to behave as currently specified (http://www.khronos.org/webgl/wiki/HandlingContextLost), but the wording of the WebGL specification could be modified to refer to the parent canvas specification and would only specify differences in behavior with respect to 2d canvas.

==== Open Issues ====
:* Should there be a base CanvasRenderingContext interface that defines isContextLost and is inherited by both WebGLRenderingContext and CanvasRenderingContext2D?

[[Category:Proposals]]

Canvas Context Loss and Restoration

2014-03-12T15:30:03Z

Junov: /* Processing Model */

:2D Canvas Rendering contexts are currently required to have persistent backing stores. This proposal aims to relax that requirement by introducing an API that allows canvases to be discarded by the browser and re-drawn by the web application on demand.

== Use Case Description ==
:The 2d canvas backing store persistence requirement often leads to large amounts of RAM (or GPU memory) being consumed by canvas elements that are not actively used because they are off screen or in background tabs or occluded windows. Other types of RAM-greedy HTML elements can release resources in such cases.

:The expectation of canvas content persistence also makes it very difficult--if not impossible--for many web apps to recover from a GPU context reset in web browsers that store 2D canvas contents in GPU memory.

=== Current Limitations ===
:In theory, web apps do have the capability of discarding canvas backing stores (set canvas size to zero) and regenerating canvas content. However, web apps are not and should not be expected to be responsible for resolving resource contention issues. The browser is responsible for monitoring resource usage and availability and is expected to take all necessary and reasonable measures to avoid crashes, hangs, and catastrophic performance degradations that may be caused by resource contention. Under the current specification, browsers have no options for evicting resources held by 2D canvases because there are no means of guaranteeing that the application will redraw the contents when needed.

=== Current Usage and Workarounds ===
:Web apps can track events to detect when the page is no longer visible (http://www.w3.org/TR/page-visibility/) and deallocate backing stores at that time by setting the size of the canvas element to 0. Conversely, they can detect when the page is visible again and reinitialize at that time

:Web apps can track events that are often associated with GPU context losses (e.g. waking-up from hibernation), and conservatively reinitialize the 2D canvas by resetting the context (set canvas width/height) and redrawing, just in case.

=== Benefits ===
:* Empower the browser to monitor resources to decide whether to drop canvas backing stores and in what order (LRU backgroung tabs?) in order to achieve better performance and stability. If web apps must handle resource eviction themselves, they may often free resources when not necessary, which may lead to unnecessary tab switching lag.
:* Make recovery from GPU context losses more robust.
:* Allow GPU-accelerated 2D canvases on platforms that are known to drop graphics contexts often or unpredictably.

=== Requests for this Feature ===
:* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/CQJXpXxO6dk E-mail thread on Chromium graphics-dev mailing list]</cite> <blockquote><p>"Is there any reason why we don't add a similar optional callback to the 2D context? (in reference to WebGL context loss API)" --Rik Cabanier, Adobe</p></blockquote>

== Proposed Solutions ==
The following solutions were pondered in the discussion thread cited above:
:* Generalize the WebGL context lost / context recovered API, so that it applies to all types of canvases.
:* Add a redraw callback on the canvas element

=== Retained Solution : Upstream the context lost/recovered API form the WebGL specification into the parent canvas specification. ===
:General Concept:
:* a renderingContextLost event is fired after the context is lost.
:* a renderingContextRestored event is fired immediately after a previously lost canvas context is brought back to a usable state. The canvas context is returned to its initial state and the canvas's backing store is blank (transparent black) when restored.

==== Processing Model ====
:Rendering context losses may be intended by the user agent (to resolve resource contention), or may be forced by external factors (e.g. a graphics driver reset).

:For convenience, the lost state should be accessible. To do so, the isContextLost method that is defined in the WebGLRenderingContext API should also exist in the CanvasRenderingContext2D API.

:Losing contexts (applies to 2d contexts, some aspects different for WebGL):
:* The UA is only allowed to lose contexts intentionally if the context was opted-in to have discarable storage (see below).
:* The UA is free to use any set of rules to decide which contexts are dropped when and in what order.
:* The return value of isContextLost() may transition from false to true before the contextLost event is dispatched (like webGL).
:* All objects that depend on the content of the canvas (e.g. patterns, imageBitmaps) are neutered when the context is lost. The neutering propagates through creation dependency chains, so a Pattern created from an ImageBitmap created from and ImageBitmap created from a canvas will be neutered if the canvas's context is lost. This rule makes it safe for implementations to optimize their memory consumption by sharing pixel buffers between objects when possible.

:Restoring contexts (applies to 2d contexts, some aspects different for WebGL):
:* Between the time a context is recovered and invoking the listener for the contextRestored event, no other user code can be executed. This is to prevent race conditions that could result from the rendering context validly executing draw commands before the contextRestored event is dispatched. To respect this constraint, a user agent that restores contexts asynchronously would have to behave as if the context was not yet restored between the time the context is restored internally and the time the contextRestored event listener is called.
:* The app is responsible for re-creating the objects that were neutered when the context was lost.
:* There can only be one contextRestored event pending at a time. When there are multiple canvases to be restored, the next canvas to be restored can only be restored after any pending contextRestored events--from previously restored canvases--have been handled.
:* Context restoration can only be initiated by the user agent (can not by triggered by a script action).
:* A lost context can only be restored after its context lost event has been dispatched. This avoids synchronization inconsistencies with isContextLost().
:* The return value of isContextLost() transitions from true to false at the time that contextRestored event is dispatched (like webGL). So the contextRestored event listener is always the first task to be execute after the transition.

:Behavior when using a lost context (applies to 2d contexts, WebGL may behave differently):
:* All draw calls exit without drawing.
:* All rendering context API calls will throw the same exceptions as they would if called on a valid context.
:* All calls that read back canvas pixels from either the canvas element or the canvas rendering context (getImageData, toDataURL, toBlob, createPattern, creatImageBitmap, drawImage with lost canvas as source) will behave as they would if the canvas context were valid and blank (all transparent black pixels).
:* Using a Patterns or an ImageBitmap that was neutered because the context of its source canvas was lost will behave as if the Pattern or ImageBitmap were valid and blank (all transparent black pixels).

:Behaviors specific to GPU-accelerated implementation of 2d canvas (non-normative?)
:* If the context was lost due to a GPU-related failure, and the browser is actively restoring GPU functionality, and expects to restore in a timely manner, then context restoration should wait until the browser is ready to resume GPU functionality and the restored canvas should continue to be GPU-accelerated. Conversely, if GPU functionality is permanently disable or if it is unknow whether or how long it may take to resume GPU operation, then the canvas should be restored immediately without GPU-acceleration
:* If an accelerated canvas is resized while the GPU is temporarily unavailable, the creation of the new canvas buffer should be postponed until the GPU functionality is restored.
:* When getContext() is called while GPU functionality is temporarily unavailable:
:: a) If the canvas does not have an associated context, create an new unaccelerated context.
:: b) If the canvas already has an associate context, return the existing context even if it is in a lost state.

:Opting-in to discardable backing stores.
:* A new context creation parameter named 'storage' is to be added. Possible values are 'persistent' and 'discardable'. Usage: canvas.getContext('2d', {storage: 'discardable'});
:* In 'persistent' mode the UA may not elect to evict the canvas's backing store.
:* The default is 'persistent', which provides backwards compatible behavior for application that expect canvas backing store content to never be lost.
:* Note: This parameter is a string rather than a boolean because it is anticipated that additional storage modes may be added in the future.

==== Limitations ====
:Web browsers will not reap the stability and performance rewards associated with this API with Web apps that do not provide at least a handler for the context recovered event. In order for this feature to improve the state of the web, apps will need to opt in to this new API, which unfortunately needs to be optional for backwards compatibility reasons.

==== Implementation ====
:Browser vendors should be highly motivated to implement this API in order to improve platform resilience and performance, especially on mobile platforms where RAM contention is often a critical issue.

==== Adoption ====
:Web App developers should be motivated to adopt this API in order to improve the stability of their products:
:* Avoid triggering out-of-memory crashes in the browser.
:* Avoid browser performance issues associated with having a given app in a background tab
:* Robustly recover from GPU failures.

:Top tier apps (particularly from developers that track usage metrics) would be expected to enthusiastically adopt this API.

==== Specification ====
:WebGL would continue to behave as currently specified (http://www.khronos.org/webgl/wiki/HandlingContextLost), but the wording of the WebGL specification could be modified to refer to the parent canvas specification and would only specify differences in behavior with respect to 2d canvas.

==== Open Issues ====
:* Should there be a base CanvasRenderingContext interface that defines isContextLost and is inherited by both WebGLRenderingContext and CanvasRenderingContext2D?

[[Category:Proposals]]

Canvas Context Loss and Restoration

2014-03-12T15:24:17Z

Junov:

:2D Canvas Rendering contexts are currently required to have persistent backing stores. This proposal aims to relax that requirement by introducing an API that allows canvases to be discarded by the browser and re-drawn by the web application on demand.

== Use Case Description ==
:The 2d canvas backing store persistence requirement often leads to large amounts of RAM (or GPU memory) being consumed by canvas elements that are not actively used because they are off screen or in background tabs or occluded windows. Other types of RAM-greedy HTML elements can release resources in such cases.

:The expectation of canvas content persistence also makes it very difficult--if not impossible--for many web apps to recover from a GPU context reset in web browsers that store 2D canvas contents in GPU memory.

=== Current Limitations ===
:In theory, web apps do have the capability of discarding canvas backing stores (set canvas size to zero) and regenerating canvas content. However, web apps are not and should not be expected to be responsible for resolving resource contention issues. The browser is responsible for monitoring resource usage and availability and is expected to take all necessary and reasonable measures to avoid crashes, hangs, and catastrophic performance degradations that may be caused by resource contention. Under the current specification, browsers have no options for evicting resources held by 2D canvases because there are no means of guaranteeing that the application will redraw the contents when needed.

=== Current Usage and Workarounds ===
:Web apps can track events to detect when the page is no longer visible (http://www.w3.org/TR/page-visibility/) and deallocate backing stores at that time by setting the size of the canvas element to 0. Conversely, they can detect when the page is visible again and reinitialize at that time

:Web apps can track events that are often associated with GPU context losses (e.g. waking-up from hibernation), and conservatively reinitialize the 2D canvas by resetting the context (set canvas width/height) and redrawing, just in case.

=== Benefits ===
:* Empower the browser to monitor resources to decide whether to drop canvas backing stores and in what order (LRU backgroung tabs?) in order to achieve better performance and stability. If web apps must handle resource eviction themselves, they may often free resources when not necessary, which may lead to unnecessary tab switching lag.
:* Make recovery from GPU context losses more robust.
:* Allow GPU-accelerated 2D canvases on platforms that are known to drop graphics contexts often or unpredictably.

=== Requests for this Feature ===
:* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/CQJXpXxO6dk E-mail thread on Chromium graphics-dev mailing list]</cite> <blockquote><p>"Is there any reason why we don't add a similar optional callback to the 2D context? (in reference to WebGL context loss API)" --Rik Cabanier, Adobe</p></blockquote>

== Proposed Solutions ==
The following solutions were pondered in the discussion thread cited above:
:* Generalize the WebGL context lost / context recovered API, so that it applies to all types of canvases.
:* Add a redraw callback on the canvas element

=== Retained Solution : Upstream the context lost/recovered API form the WebGL specification into the parent canvas specification. ===
:General Concept:
:* a renderingContextLost event is fired after the context is lost.
:* a renderingContextRestored event is fired immediately after a previously lost canvas context is brought back to a usable state. The canvas context is returned to its initial state and the canvas's backing store is blank (transparent black) when restored.

==== Processing Model ====
:Rendering context losses may be intended by the user agent (to resolve resource contention), or may be forced by external factors (e.g. a graphics driver reset).

:For convenience, the lost state should be accessible. To do so, the isContextLost method that is defined in the WebGLRenderingContext API should also exist in the CanvasRenderingContext2D API.

:Losing contexts (applies to 2d contexts, some aspects different for WebGL):
:* The UA is only allowed to lose contexts intentionally if the context was opted-in to have discarable storage (see below).
:* The UA is free to use any set of rules to decide which contexts are dropped when and in what order.
:* The return value of isContextLost() may transition from false to true before the contextLost event is dispatched (like webGL).
:* All objects that depend on the content of the canvas (e.g. patterns, imageBitmaps) are neutered when the context is lost. The neutering propagates through creation dependency chains, so a Pattern created from an ImageBitmap created from and ImageBitmap created from a canvas will be neutered if the canvas's context is lost. This rule makes it safe for implementations to optimize their memory consumption by sharing pixel buffers between objects when possible.

:Restoring contexts (applies to 2d contexts, some aspects different for WebGL):
:* Between the time a context is recovered and invoking the listener for the contextRestored event, no other user code can be executed. This is to prevent race conditions that could result from the rendering context validly executing draw commands before the contextRestored event is dispatched. To respect this constraint, a user agent that restores contexts asynchronously would have to behave as if the context was not yet restored between the time the context is restored internally and the time the contextRestored event listener is called.
:* The app is responsible for re-creating the objects that were neutered when the context was lost.
:* There can only be one contextRestored event pending at a time. When there are multiple canvases to be restored, the next canvas to be restored can only be restored after any pending contextRestored events--from previously restored canvases--have been handled.
:* Context restoration can only be initiated by the user agent (can not by triggered by a script action).
:* A lost context can only be restored after its context lost event has been dispatched. This avoids synchronization inconsistencies with isContextLost().
:* The return value of isContextLost() transitions from true to false at the time that contextRestored event is dispatched (like webGL). So the contextRestored event listener is always the first task to be execute after the transition.

:Behavior when using a lost context (applies to 2d contexts, WebGL may behave differently):
:* All draw calls exit without drawing.
:* All rendering context API calls will throw the same exceptions as they would if called on a valid context.
:* All calls that read back canvas pixels from either the canvas element or the canvas rendering context (getImageData, toDataURL, toBlob, createPattern, creatImageBitmap, drawImage with lost canvas as source) will behave as they would if the canvas context were valid and blank (all transparent black pixels).
:* Using a Patterns or an ImageBitmap that was neutered because the context of its source canvas was lost will behave as if the Pattern or ImageBitmap were valid and blank (all transparent black pixels).

:Behaviors specific to GPU-accelerated implementation of 2d canvas (non-normative?)
:* If the context was lost due to a GPU-related failure, and the browser is actively restoring GPU functionality, and expects to restore in a timely manner, then context restoration should wait until the browser is ready to resume GPU functionality and the restored canvas should continue to be GPU-accelerated. Conversely, if GPU functionality is permanently disable or if it is unknow whether or how long it may take to resume GPU operation, then the canvas should be restored immediately without GPU-acceleration
:* If an accelerated canvas is resized while the GPU is temporarily unavailable, the creation of the new canvas buffer should be postponed until the GPU functionality is restored.
:* When getContext() is called while GPU functionality is temporarily unavailable:
:: a) If the canvas does not have an associated context, create an new unaccelerated context.
:: b) If the canvas already has an associate context, return the existing context even if it is in a lost state.

:Opting-in to discardable backing stores.
:* A new context creation parameter named 'storage' is to be added. Possible values are 'persistent' and 'discardable'.
:* In 'persistent' mode the UA may not elect to evict the canvas's backing store.
:* The default is 'persistent', which provides backwards compatible behavior for application that expect canvas backing store content to never be lost.

==== Limitations ====
:Web browsers will not reap the stability and performance rewards associated with this API with Web apps that do not provide at least a handler for the context recovered event. In order for this feature to improve the state of the web, apps will need to opt in to this new API, which unfortunately needs to be optional for backwards compatibility reasons.

==== Implementation ====
:Browser vendors should be highly motivated to implement this API in order to improve platform resilience and performance, especially on mobile platforms where RAM contention is often a critical issue.

==== Adoption ====
:Web App developers should be motivated to adopt this API in order to improve the stability of their products:
:* Avoid triggering out-of-memory crashes in the browser.
:* Avoid browser performance issues associated with having a given app in a background tab
:* Robustly recover from GPU failures.

:Top tier apps (particularly from developers that track usage metrics) would be expected to enthusiastically adopt this API.

==== Specification ====
:WebGL would continue to behave as currently specified (http://www.khronos.org/webgl/wiki/HandlingContextLost), but the wording of the WebGL specification could be modified to refer to the parent canvas specification and would only specify differences in behavior with respect to 2d canvas.

==== Open Issues ====
:* Should there be a base CanvasRenderingContext interface that defines isContextLost and is inherited by both WebGLRenderingContext and CanvasRenderingContext2D?

[[Category:Proposals]]

Canvas Context Loss and Restoration

2014-01-28T16:43:19Z

Junov:

:2D Canvas Rendering contexts are currently required to have persistent backing stores. This proposal aims to relax that requirement by introducing an API that allows canvases to be discarded by the browser and re-drawn by the web application on demand.

== Use Case Description ==
:The 2d canvas backing store persistence requirement often leads to large amounts of RAM (or GPU memory) being consumed by canvas elements that are not actively used because they are off screen or in background tabs or occluded windows. Other types of RAM-greedy HTML elements can release resources in such cases.

:The expectation of canvas content persistence also makes it very difficult--if not impossible--for many web apps to recover from a GPU context reset in web browsers that store 2D canvas contents in GPU memory.

=== Current Limitations ===
:In theory, web apps do have the capability of discarding canvas backing stores (set canvas size to zero) and regenerating canvas content. However, web apps are not and should not be expected to be responsible for resolving resource contention issues. The browser is responsible for monitoring resource usage and availability and is expected to take all necessary and reasonable measures to avoid crashes, hangs, and catastrophic performance degradations that may be caused by resource contention. Under the current specification, browsers have no options for evicting resources held by 2D canvases because there are no means of guaranteeing that the application will redraw the contents when needed.

=== Current Usage and Workarounds ===
:Web apps can track events to detect when the page is no longer visible (http://www.w3.org/TR/page-visibility/) and deallocate backing stores at that time by setting the size of the canvas element to 0. Conversely, they can detect when the page is visible again and reinitialize at that time

:Web apps can track events that are often associated with GPU context losses (e.g. waking-up from hibernation), and conservatively reinitialize the 2D canvas by resetting the context (set canvas width/height) and redrawing, just in case.

=== Benefits ===
:* Empower the browser to monitor resources to decide whether to drop canvas backing stores and in what order (LRU backgroung tabs?) in order to achieve better performance and stability. If web apps must handle resource eviction themselves, they may often free resources when not necessary, which may lead to unnecessary tab switching lag.
:* Make recovery from GPU context losses more robust.
:* Allow GPU-accelerated 2D canvases on platforms that are known to drop graphics contexts often or unpredictably.

=== Requests for this Feature ===
:* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/CQJXpXxO6dk E-mail thread on Chromium graphics-dev mailing list]</cite> <blockquote><p>"Is there any reason why we don't add a similar optional callback to the 2D context? (in reference to WebGL context loss API)" --Rik Cabanier, Adobe</p></blockquote>

== Proposed Solutions ==
The following solutions were pondered in the discussion thread cited above:
:* Generalize the WebGL context lost / context recovered API, so that it applies to all types of canvases.
:* Add a redraw callback on the canvas element

=== Retained Solution : Upstream the context lost/recovered API form the WebGL specification into the parent canvas specification. ===
:General Concept:
:* a renderingContextLost event is fired after the context is lost.
:* a renderingContextRestored event is fired immediately after a previously lost canvas context is brought back to a usable state. The canvas context is returned to its initial state and the canvas's backing store is blank (transparent black) when restored.

==== Processing Model ====
:Rendering context losses may be intended by the user agent (to resolve resource contention), or may be forced by external factors (e.g. a graphics driver reset).

:For convenience, the lost state should be accessible. To do so, the isContextLost method that is defined in the WebGLRenderingContext API should also exist in the CanvasRenderingContext2D API.

:Losing contexts (applies to 2d contexts, some aspects different for WebGL):
:* Intentional losses are only allowed if the contextRestored event is handled on the canvas element associated with the context.
:* The UA is free to use any set of rules to decide which contexts are dropped when and in what order.
:* The return value of isContextLost() may transition from false to true before the contextLost event is dispatched (like webGL).
:* All objects that depend on the content of the canvas (e.g. patterns, imageBitmaps) are neutered when the context is lost. The neutering propagates through creation dependency chains, so a Pattern created from an ImageBitmap created from and ImageBitmap created from a canvas will be neutered if the canvas's context is lost. This rule makes it safe for implementations to optimize their memory consumption by sharing pixel buffers between objects when possible.

:Restoring contexts (applies to 2d contexts, some aspects different for WebGL):
:* Between the time a context is recovered and invoking the listener for the contextRestored event, no other user code can be executed. This is to prevent race conditions that could result from the rendering context validly executing draw commands before the contextRestored event is dispatched. To respect this constraint, a user agent that restores contexts asynchronously would have to behave as if the context was not yet restored between the time the context is restored internally and the time the contextRestored event listener is called.
:* The app is responsible for re-creating the objects that were neutered when the context was lost.
:* There can only be one contextRestored event pending at a time. When there are multiple canvases to be restored, the next canvas to be restored can only be restored after any pending contextRestored events--from previously restored canvases--have been handled.
:* Context restoration can only be initiated by the user agent (can not by triggered by a script action).
:* A lost context can only be restored after its context lost event has been dispatched. This avoids synchronization inconsistencies with isContextLost().
:* The return value of isContextLost() transitions from true to false at the time that contextRestored event is dispatched (like webGL). So the contextRestored event listener is always the first task to be execute after the transition.

:Behavior when using a lost context (applies to 2d contexts, WebGL may behave differently):
:* All draw calls exit without drawing.
:* All rendering context API calls will throw the same exceptions as they would if called on a valid context.
:* All calls that read back canvas pixels from either the canvas element or the canvas rendering context (getImageData, toDataURL, toBlob, createPattern, creatImageBitmap, drawImage with lost canvas as source) will behave as they would if the canvas context were valid and blank (all transparent black pixels).
:* Using a Patterns or an ImageBitmap that was neutered because the context of its source canvas was lost will behave as if the Pattern or ImageBitmap were valid and blank (all transparent black pixels).

:Behaviors specific to GPU-accelerated implementation of 2d canvas (non-normative?)
:* If the context was lost due to a GPU-related failure, and the browser is actively restoring GPU functionality, and expects to restore in a timely manner, then context restoration should wait until the browser is ready to resume GPU functionality and the restored canvas should continue to be GPU-accelerated. Conversely, if GPU functionality is permanently disable or if it is unknow whether or how long it may take to resume GPU operation, then the canvas should be restored immediately without GPU-acceleration
:* If an accelerated canvas is resized while the GPU is temporarily unavailable, the creation of the new canvas buffer should be postponed until the GPU functionality is restored.
:* When getContext() is called while GPU functionality is temporarily unavailable:
:: a) If the canvas does not have an associated context, create an new unaccelerated context.
:: b) If the canvas already has an associate context, return the existing context even if it is in a lost state.

==== Limitations ====
:Web browsers will not reap the stability and performance rewards associated with this API with Web apps that do not provide at least a handler for the context recovered event. In order for this feature to improve the state of the web, apps will need to opt in to this new API, which unfortunately needs to be optional for backwards compatibility reasons.

==== Implementation ====
:Browser vendors should be highly motivated to implement this API in order to improve platform resilience and performance, especially on mobile platforms where RAM contention is often a critical issue.

==== Adoption ====
:Web App developers should be motivated to adopt this API in order to improve the stability of their products:
:* Avoid triggering out-of-memory crashes in the browser.
:* Avoid browser performance issues associated with having a given app in a background tab
:* Robustly recover from GPU failures.

:Top tier apps (particularly from developers that track usage metrics) would be expected to enthusiastically adopt this API.

==== Specification ====
:WebGL would continue to behave as currently specified (http://www.khronos.org/webgl/wiki/HandlingContextLost), but the wording of the WebGL specification could be modified to refer to the parent canvas specification and would only specify differences in behavior with respect to 2d canvas.

==== Open Issues ====
:* Should there be a base CanvasRenderingContext interface that defines isContextLost and is inherited by both WebGLRenderingContext and CanvasRenderingContext2D?

[[Category:Proposals]]

Canvas Context Loss and Restoration

2013-12-04T20:11:25Z

Junov:

:2D Canvas Rendering contexts are currently required to have persistent backing stores. This proposal aims to relax that requirement by introducing an API to allows canvases to be discarded by the browser and re-drawn by the web application on demand.

== Use Case Description ==
:The 2d canvas backing store persistence requirement often leads to large amounts of RAM (or GPU memory) being consumed by canvas elements that are not actively used because they are off screen or in background tabs or occluded windows. Other types of RAM-greedy HTML elements can release resources in such cases.

:The expectation of canvas content persistence also makes it very difficult--if not impossible--for many web apps to recover from a GPU context reset in web browsers that store 2D canvas contents in GPU memory.

=== Current Limitations ===
:In theory, web apps do have the capability of discarding canvas backing stores (set canvas size to zero) and regenerating canvas content. However, web apps are not and should not be expected to be responsible for resolving resource contention issues. The browser is responsible for monitoring resource usage and availability and is expected to take all necessary and reasonable measures to avoid crashes, hangs, and catastrophic performance degradations that may be caused by resource contention. Under the current specification, browsers have no options for evicting resources held by 2D canvases because there are no means of guaranteeing that the application will redraw the contents when needed.

=== Current Usage and Workarounds ===
:Web apps can track events to detect when the page is no longer visible (http://www.w3.org/TR/page-visibility/) and deallocate backing stores at that time by setting the size of the canvas element to 0. Conversely, they can detect when the page is visible again and reinitialize at that time

:Web apps can track events that are often associated with GPU context losses (e.g. waking-up from hibernation), and conservatively reinitialize the 2D canvas by resetting the context (set canvas width/height) and redrawing, just in case.

=== Benefits ===
:* Empower the browser to monitor resources to decide whether to drop canvas backing stores and in what order (LRU backgroung tabs?) in order to achieve better performance and stability. If web apps must handle resource eviction themselves, they may often free resources when not necessary, which may lead to unnecessary tab switching lag.
:* Make recovery from GPU context losses more robust.
:* Allow GPU-accelerated 2D canvases on platforms that are known to drop graphics contexts often or unpredictably.

=== Requests for this Feature ===
:* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/CQJXpXxO6dk E-mail thread on Chromium graphics-dev mailing list]</cite> <blockquote><p>"Is there any reason why we don't add a similar optional callback to the 2D context? (in reference to WebGL context loss API)" --Rik Cabanier, Adobe</p></blockquote>

== Proposed Solutions ==
The following solutions were pondered in the discussion thread cited above:
:* Generalize the WebGL context lost / context recovered API, so that it applies to all types of canvases.
:* Add a redraw callback on the canvas element

=== Retained Solution : Upstream the context lost/recovered API form the WebGL specification into the parent canvas specification. ===
:General Concept:
:* a renderingContextLost event is fired after the context is lost.
:* a renderingContextRestored event is fired immediately after a previously lost canvas context is brought back to a usable state. The canvas context is returned to its initial state and the canvas's backing store is blank (transparent black) when restored.

==== Processing Model ====
:Rendering context losses may be intended by the user agent (to resolve resource contention), or may be forced by external factors (e.g. a graphics driver reset).

:For convenience, the lost state should be accessible. To do so, the isContextLost method that is defined in the WebGLRenderingContext API should also exist in the CanvasRenderingContext2D API.

:Losing contexts (applies to 2d contexts, some aspects different for WebGL):
:* Intentional losses are only allowed if the contextRestored event is handled on the canvas element associated with the context.
:* The UA is free to use any set of rules to decide which contexts are dropped when and in what order.
:* The return value of isContextLost() may transition from false to true before the contextLost event is dispatched (like webGL).
:* All objects that depend on the content of the canvas (e.g. patterns, imageBitmaps) are neutered when the context is lost. The neutering propagates through creation dependency chains, so a Pattern created from an ImageBitmap created from and ImageBitmap created from a canvas will be neutered if the canvas's context is lost. This rule makes it safe for implementations to optimize their memory consumption by sharing pixel buffers between objects when possible.

:Restoring contexts (applies to 2d contexts, some aspects different for WebGL):
:* Between the time a context is recovered and invoking the listener for the contextRestored event, no other user code can be executed. This is to prevent race conditions that could result from the rendering context validly executing draw commands before the contextRestored event is dispatched. To respect this constraint, a user agent that restores contexts asynchronously would have to behave as if the context was not yet restored between the time the context is restored internally and the time the contextRestored event listener is called.
:* The app is responsible for re-creating the objects that were neutered when the context was lost.
:* There can only be one contextRestored event pending at a time. When there are multiple canvases to be restored, the next canvas to be restored can only be restored after any pending contextRestored events--from previously restored canvases--have been handled.
:* Context restoration can only be initiated by the user agent (can not by triggered by a script action).
:* A lost context can only be restored after its context lost event has been dispatched. This avoids synchronization inconsistencies with isContextLost().
:* The return value of isContextLost() transitions from true to false at the time that contextRestored event is dispatched (like webGL). So the contextRestored event listener is always the first task to be execute after the transition.

:Behavior when using a lost context (applies to 2d contexts, WebGL may behave differently):
:* All draw calls exit without drawing.
:* All rendering context API calls will throw the same exceptions as they would if called on a valid context.
:* All calls that read back canvas pixels from either the canvas element or the canvas rendering context (getImageData, toDataURL, toBlob, createPattern, creatImageBitmap, drawImage with lost canvas as source) will behave as they would if the canvas context were valid and blank (all transparent black pixels).
:* Using a Patterns or an ImageBitmap that was neutered because the context of its source canvas was lost will behave as if the Pattern or ImageBitmap were valid and blank (all transparent black pixels).

:Behaviors specific to GPU-accelerated implementation of 2d canvas (non-normative?)
:* If the context was lost due to a GPU-related failure, and the browser is actively restoring GPU functionality, and expects to restore in a timely manner, then context restoration should wait until the browser is ready to resume GPU functionality and the restored canvas should continue to be GPU-accelerated. Conversely, if GPU functionality is permanently disable or if it is unknow whether or how long it may take to resume GPU operation, then the canvas should be restored immediately without GPU-acceleration
:* If an accelerated canvas is resized while the GPU is temporarily unavailable, the creation of the new canvas buffer should be postponed until the GPU functionality is restored.
:* When getContext() is called while GPU functionality is temporarily unavailable:
:: a) If the canvas does not have an associated context, create an new unaccelerated context.
:: b) If the canvas already has an associate context, return the existing context even if it is in a lost state.

==== Limitations ====
:Web browsers will not reap the stability and performance rewards associated with this API with Web apps that do not provide at least a handler for the context recovered event. In order for this feature to improve the state of the web, apps will need to opt in to this new API, which unfortunately needs to be optional for backwards compatibility reasons.

==== Implementation ====
:Browser vendors should be highly motivated to implement this API in order to improve platform resilience and performance, especially on mobile platforms where RAM contention is often a critical issue.

==== Adoption ====
:Web App developers should be motivated to adopt this API in order to improve the stability of their products:
:* Avoid triggering out-of-memory crashes in the browser.
:* Avoid browser performance issues associated with having a given app in a background tab
:* Robustly recover from GPU failures.

:Top tier apps (particularly from developers that track usage metrics) would be expected to enthusiastically adopt this API.

==== Specification ====
:WebGL would continue to behave as currently specified (http://www.khronos.org/webgl/wiki/HandlingContextLost), but the wording of the WebGL specification could be modified to refer to the parent canvas specification and would only specify differences in behavior with respect to 2d canvas.

==== Open Issues ====
:* Should there be a base CanvasRenderingContext interface that defines isContextLost and is inherited by both WebGLRenderingContext and CanvasRenderingContext2D?

[[Category:Proposals]]

Canvas Context Loss and Restoration

2013-10-22T15:07:52Z

Junov:

:2D Canvas Rendering contexts are currently required to have persistent backing stores. This proposal aims to relax that requirement by introducing an API to allows canvases to be discarded by the browser and re-drawn by the web application on demand.

== Use Case Description ==
:The 2d canvas backing store persistence requirement often leads to large amounts of RAM (or GPU memory) being consumed by canvas elements that are not actively used because they are off screen or in background tabs or occluded windows. Other types of RAM-greedy HTML elements can release resources in such cases.

:The expectation of canvas content persistence also makes it very difficult--if not impossible--for many web apps to recover from a GPU context reset in web browsers that store 2D canvas contents in GPU memory.

=== Current Limitations ===
:In theory, web apps do have the capability of discarding canvas backing stores (set canvas size to zero) and regenerating canvas content. However, web apps are not and should not be expected to be responsible for resolving resource contention issues. The browser is responsible for monitoring resource usage and availability and is expected to take all necessary and reasonable measures to avoid crashes, hangs, and catastrophic performance degradations that may be caused by resource contention. Under the current specification, browsers have no options for evicting resources held by 2D canvases because there are no means of guaranteeing that the application will redraw the contents when needed.

=== Current Usage and Workarounds ===
:Web apps can track events to detect when the page is no longer visible (http://www.w3.org/TR/page-visibility/) and deallocate backing stores at that time by setting the size of the canvas element to 0. Conversely, they can detect when the page is visible again and reinitialize at that time

:Web apps can track events that are often associated with GPU context losses (e.g. waking-up from hibernation), and conservatively reinitialize the 2D canvas by resetting the context (set canvas width/height) and redrawing, just in case.

=== Benefits ===
:* Empower the browser to monitor resources to decide whether to drop canvas backing stores and in what order (LRU backgroung tabs?) in order to achieve better performance and stability. If web apps must handle resource eviction themselves, they may often free resources when not necessary, which may lead to unnecessary tab switching lag.
:* Make recovery from GPU context losses more robust.
:* Allow GPU-accelerated 2D canvases on platforms that are known to drop graphics contexts often or unpredictably.

=== Requests for this Feature ===
:* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/CQJXpXxO6dk E-mail thread on Chromium graphics-dev mailing list]</cite> <blockquote><p>"Is there any reason why we don't add a similar optional callback to the 2D context? (in reference to WebGL context loss API)" --Rik Cabanier, Adobe</p></blockquote>

== Proposed Solutions ==
The following solutions were pondered in the discussion thread cited above:
:* Generalize the WebGL context lost / context recovered API, so that it applies to all types of canvases.
:* Add a redraw callback on the canvas element

=== Retained Solution : Upstream the context lost/recovered API form the WebGL specification into the parent canvas specification. ===
:General Concept:
:* a renderingContextLost event is fired after the context is lost.
:* a renderingContextRestored event is fired immediately after a previously lost canvas context is brought back to a usable state. The canvas context is returned to its initial state and the canvas's backing store is blank (transparent black) when restored.

==== Processing Model ====
:Rendering context losses may be intended by the user agent (to resolve resource contention), or may be forced by external factors (e.g. a graphics driver reset).

:For convenience, the lost state should be accessible. To do so, the isContextLost method that is defined in the WebGLRenderingContext API should also exist in the CanvasRenderingContext2D API.

:Losing contexts (applies to 2d contexts, some aspects different for WebGL):
:* Intentional losses are only allowed if the contextRestored event is handled on the canvas element associated with the context.
:* The UA is free to use any set of rules to decide which contexts are dropped when and in what order.
:* The return value of isContextLost() may transition from false to true before the contextLost event is dispatched (like webGL).
:* All objects that depend on the content of the canvas (e.g. patterns, imageBitmaps) are neutered when the context is lost. The neutering propagates through creation dependency chains, so a Pattern created from an ImageBitmap created from and ImageBitmap created from a canvas will be neutered if the canvas's context is lost. This rule makes it safe for implementations to optimize their memory consumption by sharing pixel buffers between objects when possible.

:Restoring contexts (applies to 2d contexts, some aspects different for WebGL):
:* Between the time a context is recovered and invoking the listener for the contextRestored event, no other user code can be executed.
:* The app is responsible for re-creating the objects that were neutered when the context was lost.
:* There can only be one contextRestored event pending at a time. When there are multiple canvases to be restored, the next canvas to be restored can only be restored after any pending contextRestored events--from previously restored canvases--have been handled.
:* Context restoration can only be initiated by the user agent (can not by triggered by a script action).
:* A lost context can only be restored after its context lost event has been dispatched. This avoids synchronization inconsistencies with isContextLost().
:* The return value of isContextLost() transitions from true to false at the time that contextRestored event is dispatched (like webGL). So the contextRestored event listener is always the first task to be execute after the transition.

:Behavior when using a lost context (applies to 2d contexts, WebGL may behave differently):
:* All draw calls exit without drawing.
:* All rendering context API calls will throw the same exceptions as they would if called on a valid context.
:* All calls that read back canvas pixels from either the canvas element or the canvas rendering context (getImageData, toDataURL, toBlob, createPattern, creatImageBitmap, drawImage with lost canvas as source) will behave as they would if the canvas context were valid and blank (all transparent black pixels).
:* Using a Patterns or an ImageBitmap that was neutered because the context of its source canvas was lost will behave as if the Pattern or ImageBitmap were valid and blank (all transparent black pixels).

:Behaviors specific to GPU-accelerated implementation of 2d canvas (non-normative?)
:* If the context was lost due to a GPU-related failure, and the browser is actively restoring GPU functionality, and expects to restore in a timely manner, then context restoration should wait until the browser is ready to resume GPU functionality and the restored canvas should continue to be GPU-accelerated. Conversely, if GPU functionality is permanently disable or if it is unknow whether or how long it may take to resume GPU operation, then the canvas should be restored immediately without GPU-acceleration
:* If an accelerated canvas is resized while the GPU is temporarily unavailable, the creation of the new canvas buffer should be postponed until the GPU functionality is restored.
:* When getContext() is called while GPU functionality is temporarily unavailable:
:: a) If the canvas does not have an associated context, create an new unaccelerated context.
:: b) If the canvas already has an associate context, return the existing context even if it is in a lost state.

==== Limitations ====
:Web browsers will not reap the stability and performance rewards associated with this API with Web apps that do not provide at least a handler for the context recovered event. In order for this feature to improve the state of the web, apps will need to opt in to this new API, which unfortunately needs to be optional for backwards compatibility reasons.

==== Implementation ====
:Browser vendors should be highly motivated to implement this API in order to improve platform resilience and performance, especially on mobile platforms where RAM contention is often a critical issue.

==== Adoption ====
:Web App developers should be motivated to adopt this API in order to improve the stability of their products:
:* Avoid triggering out-of-memory crashes in the browser.
:* Avoid browser performance issues associated with having a given app in a background tab
:* Robustly recover from GPU failures.

:Top tier apps (particularly from developers that track usage metrics) would be expected to enthusiastically adopt this API.

==== Specification ====
:WebGL would continue to behave as currently specified (http://www.khronos.org/webgl/wiki/HandlingContextLost), but the wording of the WebGL specification could be modified to refer to the parent canvas specification and would only specify differences in behavior with respect to 2d canvas.

==== Open Issues ====
:* Should there be a base CanvasRenderingContext interface that defines isContextLost and is inherited by both WebGLRenderingContext and CanvasRenderingContext2D?

[[Category:Proposals]]

Canvas Context Loss and Restoration

2013-10-21T15:26:49Z

Junov:

:2D Canvas Rendering contexts are currently required to have persistent backing stores. This proposal aims to relax that requirement by introducing an API to allows canvases to be discarded by the browser and re-drawn by the web application on demand.

== Use Case Description ==
:The 2d canvas backing store persistence requirement often leads to large amounts of RAM (or GPU memory) being consumed by canvas elements that are not actively used because they are off screen or in background tabs or occluded windows. Other types of RAM-greedy HTML elements can release resources in such cases.

:The expectation of canvas content persistence also makes it very difficult--if not impossible--for many web apps to recover from a GPU context reset in web browsers that store 2D canvas contents in GPU memory.

=== Current Limitations ===
:In theory, web apps do have the capability of discarding canvas backing stores (set canvas size to zero) and regenerating canvas content. However, web apps are not and should not be expected to be responsible for resolving resource contention issues. The browser is responsible for monitoring resource usage and availability and is expected to take all necessary and reasonable measures to avoid crashes, hangs, and catastrophic performance degradations that may be caused by resource contention. Under the current specification, browsers have no options for evicting resources held by 2D canvases because there are no means of guaranteeing that the application will redraw the contents when needed.

=== Current Usage and Workarounds ===
:Web apps can track events to detect when the page is no longer visible (http://www.w3.org/TR/page-visibility/) and deallocate backing stores at that time by setting the size of the canvas element to 0. Conversely, they can detect when the page is visible again and reinitialize at that time

:Web apps can track events that are often associated with GPU context losses (e.g. waking-up from hibernation), and conservatively reinitialize the 2D canvas by resetting the context (set canvas width/height) and redrawing, just in case.

=== Benefits ===
:* Empower the browser to monitor resources to decide whether to drop canvas backing stores and in what order (LRU backgroung tabs?) in order to achieve better performance and stability. If web apps must handle resource eviction themselves, they may often free resources when not necessary, which may lead to unnecessary tab switching lag.
:* Make recovery from GPU context losses more robust.
:* Allow GPU-accelerated 2D canvases on platforms that are known to drop graphics contexts often or unpredictably.

=== Requests for this Feature ===
:* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/CQJXpXxO6dk E-mail thread on Chromium graphics-dev mailing list]</cite> <blockquote><p>"Is there any reason why we don't add a similar optional callback to the 2D context? (in reference to WebGL context loss API)" --Rik Cabanier, Adobe</p></blockquote>

== Proposed Solutions ==
The following solutions were pondered in the discussion thread cited above:
:* Generalize the WebGL context lost / context recovered API, so that it applies to all types of canvases.
:* Add a redraw callback on the canvas element

=== Retained Solution : Upstream the context lost/recovered API form the WebGL specification into the parent canvas specification. ===
:General Concept:
:* a renderingContextLost event is fired after the context is lost.
:* a renderingContextRestored event is fired immediately after a previously lost canvas context is brought back to a usable state. The canvas context is returned to its initial state and the canvas's backing store is blank (transparent black) when restored.

==== Processing Model ====
:Rendering context losses may be intended by the user agent (to resolve resource contention), or may be forced by external factors (e.g. a graphics driver reset).

:For convenience, the lost state should be accessible. To do so, the isContextLost method that is defined in the WebGLRenderingContext API should also exist in the CanvasRenderingContext2D API.

:Losing contexts (applies to 2d contexts, some aspects different for WebGL):
:* Intentional losses are only allowed if the contextRestored event is handled on the canvas element associated with the context.
:* The UA is free to use any set of rules to decide which contexts are dropped when and in what order.
:* The return value of isContextLost() may transition from false to true before the contextLost event is dispatched (like webGL).
:* All objects that depend on the content of the canvas (e.g. patterns, imageBitmaps) are neutered when the context is lost. The neutering propagates through creation dependency chains, so a Pattern created from an ImageBitmap created from and ImageBitmap created from a canvas will be neutered if the canvas's context is lost. This rule makes it safe for implementations to optimize their memory consumption by sharing pixel buffers between objects when possible.

:Restoring contexts (applies to 2d contexts, some aspects different for WebGL):
:* Between the time a context is recovered and invoking the listener for the contextRestored event, no other user code can be executed.
:* The app is responsible for re-creating the objects that were neutered when the context was lost.
:* There can only be one contextRestored event pending at a time. When there are multiple canvases to be restored, the next canvas to be restored can only be restored after any pending contextRestored events--from previously restored canvases--have been handled.
:* Context restoration can only be initiated by the user agent (can not by triggered by a script action).
:* A lost context can only be restored after its context lost event has been dispatched. This avoids synchronization inconsistencies with isContextLost().
:* The return value of isContextLost() transitions from true to false at the time that contextRestored event is dispatched (like webGL). So the contextRestored event listener is always the first task to be execute after the transition.

:Behavior when using a lost context (applies to 2d contexts, WebGL may behave differently):
:* All draw calls exit without drawing.
:* All rendering context API calls will throw the same exceptions as they would if called on a valid context.
:* All calls that read back canvas pixels from either the canvas element or the canvas rendering context (getImageData, toDataURL, toBlob, createPattern, creatImageBitmap, drawImage with lost canvas as source) will behave as they would if the canvas context were valid and blank (all transparent black pixels).
:* Using a Patterns or an ImageBitmap that was neutered because the context of its source canvas was lost will behave as if the Pattern or ImageBitmap were valid and blank (all transparent black pixels).

:Behaviors specific to GPU-accelerated implementation of 2d canvas (non-normative?)
:* If the context was lost due to a GPU-related failure, and the browser is actively restoring GPU functionality, and expects to restore in a timely manner, then context restoration should wait until the browser is ready to resume GPU functionality and the restored canvas should continue to be GPU-accelerated. Conversely, if GPU functionality is permanently disable or if it is unknow whether or how long it may take to resume GPU operation, then the canvas should be restored immediately without GPU-acceleration
:* If an accelerated canvas is resized while the GPU is temporarily unavailable, the creation of the new canvas buffer should be postponed until the GPU functionality is restored.
:* When getContext() is called while GPU functionality is temporarily unavailable:
:: a) If the canvas does not have an associated context, create an new unaccelerated context.
:: b) If the canvas already has an associate context, return the existing context even if it is in a lost state.

==== Limitations ====
:Web browsers will not reap the stability and performance rewards associated with this API with Web apps that do not provide at least a handler for the context recovered event. In order for this feature to improve the state of the web, apps will need to opt in to this new API, which unfortunately needs to be optional for backwards compatibility reasons.

==== Implementation ====
:Browser vendors should be highly motivated to implement this API in order to improve platform resilience and performance, especially on mobile platforms where RAM contention is often a critical issue.

==== Adoption ====
:Web App developers should be motivated to adopt this API in order to improve the stability of their products:
:* Avoid triggering out-of-memory crashes in the browser.
:* Avoid browser performance issues associated with having a given app in a background tab
:* Robustly recover from GPU failures.

:Top tier apps (particularly from developers that track usage metrics) would be expected to enthusiastically adopt this API.

==== Specification ====
:WebGL would continue to behave as currently specified (http://www.khronos.org/webgl/wiki/HandlingContextLost), but the wording of the WebGL specification could be modified to refer to the parent canvas specification and would only specify differences in behavior with respect to 2d canvas.

==== Open Issues ====
:* Should there be a base CanvasRenderingContext interface that defines isContextLost and is inherited by both WebGLRenderingContext and CanvasRenderingContext2D?
:* Should there be loseContext/restoreContext methods. Could be very useful for testing purposes, or for app-initiated resource management.

[[Category:Proposals]]

Canvas Context Loss and Restoration

2013-10-21T15:23:20Z

Junov:

:2D Canvas Rendering contexts are currently required to have persistent backing stores. This proposal aims to relax that requirement by introducing an API to allows canvases to be discarded by the browser and re-drawn by the web application on demand.

== Use Case Description ==
:The 2d canvas backing store persistence requirement often leads to large amounts of RAM (or GPU memory) being consumed by canvas elements that are not actively used because they are off screen or in background tabs or occluded windows. Other types of RAM-greedy HTML elements can release resources in such cases.

:The expectation of canvas content persistence also makes it very difficult--if not impossible--for many web apps to recover from a GPU context reset in web browsers that store 2D canvas contents in GPU memory.

=== Current Limitations ===
:In theory, web apps do have the capability of discarding canvas backing stores (set canvas size to zero) and regenerating canvas content. However, web apps are not and should not be expected to be responsible for resolving resource contention issues. The browser is responsible for monitoring resource usage and availability and is expected to take all necessary and reasonable measures to avoid crashes, hangs, and catastrophic performance degradations that may be caused by resource contention. Under the current specification, browsers have no options for evicting resources held by 2D canvases because there are no means of guaranteeing that the application will redraw the contents when needed.

=== Current Usage and Workarounds ===
:Web apps can track events to detect when the page is no longer visible (http://www.w3.org/TR/page-visibility/) and deallocate backing stores at that time by setting the size of the canvas element to 0. Conversely, they can detect when the page is visible again and reinitialize at that time

:Web apps can track events that are often associated with GPU context losses (e.g. waking-up from hibernation), and conservatively reinitialize the 2D canvas by resetting the context (set canvas width/height) and redrawing, just in case.

=== Benefits ===

:* Empower the browser to monitor resources to decide whether to drop canvas backing stores and in what order (LRU backgroung tabs?) in order to achieve better performance and stability. If web apps must handle resource eviction themselves, they may often free resources when not necessary, which may lead to unnecessary tab switching lag.
:* Make recovery from GPU context losses more robust.
:* Allow GPU-accelerated 2D canvases on platforms that are known to drop graphics contexts often or unpredictably.

=== Requests for this Feature ===

:* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/CQJXpXxO6dk E-mail thread on Chromium graphics-dev mailing list]</cite> <blockquote><p>"Is there any reason why we don't add a similar optional callback to the 2D context? (in reference to WebGL context loss API)" --Rik Cabanier, Adobe</p></blockquote>

== Proposed Solutions ==

The following solutions were pondered in the discussion thread cited above:
:* Generalize the WebGL context lost / context recovered API, so that it applies to all types of canvases.
:* Add a redraw callback on the canvas element

=== Retained Solution : Upstream the context lost/recovered API form the WebGL specification into the parent canvas specification. ===

:General Concept:
:* a renderingContextLost event is fired after the context is lost.
:* a renderingContextRestored event is fired immediately after a previously lost canvas context is brought back to a usable state. The canvas context is returned to its initial state and the canvas's backing store is blank (transparent black) when restored.

==== Processing Model ====
:Rendering context losses may be intended by the user agent (to resolve resource contention), or may be forced by external factors (e.g. a graphics driver reset).

:For convenience, the lost state should be accessible. To do so, the isContextLost method that is defined in the WebGLRenderingContext API should also exist in the CanvasRenderingContext2D API.

:Losing contexts (applies to 2d contexts, some aspects different for WebGL):
:* Intentional losses are only allowed if the contextRestored event is handled on the canvas element associated with the context.
:* The UA is free to use any set of rules to decide which contexts are dropped when and in what order.
:* The return value of isContextLost() may transition from false to true before the contextLost event is dispatched (like webGL).
:* All objects that depend on the content of the canvas (e.g. patterns, imageBitmaps) are neutered when the context is lost. The neutering propagates through creation dependency chains, so a Pattern created from an ImageBitmap created from and ImageBitmap created from a canvas will be neutered if the canvas's context is lost. This rule makes it safe for implementations to optimize their memory consumption by sharing pixel buffers between objects when possible.

:Restoring contexts (applies to 2d contexts, some aspects different for WebGL):
:* Between the time a context is recovered and invoking the listener for the contextRestored event, no other user code can be executed.
:* The app is responsible for re-creating the objects that were neutered when the context was lost.
:* There can only be one contextRestored event pending at a time. When there are multiple canvases to be restored, the next canvas to be restored can only be restored after any pending contextRestored events--from previously restored canvases--have been handled.
:* Context restoration can only be initiated by the user agent (can not by triggered by a script action).
:* A lost context can only be restored after its context lost event has been dispatched. This avoids synchronization inconsistencies with isContextLost().
:* The return value of isContextLost() transitions from true to false at the time that contextRestored event is dispatched (like webGL). So the contextRestored event listener is always the first task to be execute after the transition.

:Behavior when using a lost context (applies to 2d contexts, WebGL may behave differently):
:* All draw calls exit without drawing.
:* All rendering context API calls will throw the same exceptions as they would if called on a valid context.
:* All calls that read back canvas pixels from either the canvas element or the canvas rendering context (getImageData, toDataURL, toBlob, createPattern, creatImageBitmap, drawImage with lost canvas as source) will behave as they would if the canvas context were valid and blank (all transparent black pixels).
:* Using a Patterns or an ImageBitmap that was neutered because the context of its source canvas was lost will behave as if the Pattern or ImageBitmap were valid and blank (all transparent black pixels).

:Behaviors specific to GPU-accelerated implementation of 2d canvas (non-normative?)
:* If the context was lost due to a GPU-related failure, and the browser is actively restoring GPU functionality, and expects to restore in a timely manner, then context restoration should wait until the browser is ready to resume GPU functionality and the restored canvas should continue to be GPU-accelerated. Conversely, if GPU functionality is permanently disable or if it is unknow whether or how long it may take to resume GPU operation, then the canvas should be restored immediately without GPU-acceleration
:* If an accelerated canvas is resized while the GPU is temporarily unavailable, the creation of the new canvas buffer should be postponed until the GPU functionality is restored.
:* When getContext() is called while GPU functionality is temporarily unavailable:
:: a) If the canvas does not have an associated context, create an new unaccelerated context.
:: b) If the canvas already has an associate context, return the existing context even if it is in a lost state.

==== Limitations ====
:Web browsers will not reap the stability and performance rewards associated with this API with Web apps that do not provide at least a handler for the context recovered event. In order for this feature to improve the state of the web, apps will need to opt in to this new API, which unfortunately needs to be optional for backwards compatibility reasons.

==== Implementation ====
:Browser vendors should be highly motivated to implement this API in order to improve platform resilience and performance, especially on mobile platforms where RAM contention is often a critical issue.

==== Adoption ====
:Web App developers should be motivated to adopt this API in order to improve the stability of their products:
:* Avoid triggering out-of-memory crashes in the browser.
:* Avoid browser performance issues associated with having a given app in a background tab
:* Robustly recover from GPU failures.

Top tier apps (particularly from developers that track usage metrics) would be expected to enthusiastically adopt this API.

==== Specification ====
: WebGL would continue to behave as currently specified (http://www.khronos.org/webgl/wiki/HandlingContextLost), but the wording of the WebGL specification could be modified to refer to the parent canvas specification and would only specify differences in behavior with respect to 2d canvas.

==== Open Issues ====
:* Should there be a base CanvasRenderingContext interface that defines isContextLost and is inherited by both WebGLRenderingContext and CanvasRenderingContext2D?
:* Should there be loseContext/restoreContext methods. Could be very useful for testing purposes, or for app-initiated resource management.

[[Category:Proposals]]

Canvas Context Loss and Restoration

2013-10-18T18:18:39Z

Junov:

:2D Canvas Rendering contexts are currently required to have persistent backing stores. This proposal aims to relax that requirement by introducing an API to allows canvases to be discarded by the browser and re-drawn by the web application on demand.

== Use Case Description ==
:The 2d canvas backing store persistence requirement often leads to large amounts of RAM (or GPU memory) being consumed by canvas elements that are not actively used because they are off screen or in background tabs or occluded windows. Other types of RAM-greedy HTML elements can release resources in such cases.

:The expectation of canvas content persistence also makes it very difficult--if not impossible--for many web apps to recover from a GPU context reset in web browsers that store 2D canvas contents in GPU memory.

=== Current Limitations ===
:In theory, web apps do have the capability of discarding canvas backing stores (set canvas size to zero) and regenerating canvas content. However, web apps are not and should not be expected to be responsible for resolving resource contention issues. The browser is responsible for monitoring resource usage and availability and is expected to take all necessary and reasonable measures to avoid crashes, hangs, and catastrophic performance degradations that may be caused by resource contention. Under the current specification, browsers have no options for evicting resources held by 2D canvases because there are no means of guaranteeing that the application will redraw the contents when needed.

=== Current Usage and Workarounds ===
:Web apps can track events to detect when the page is no longer visible (http://www.w3.org/TR/page-visibility/) and deallocate backing stores at that time by setting the size of the canvas element to 0. Conversely, they can detect when the page is visible again and reinitialize at that time

:Web apps can track events that are often associated with GPU context losses (e.g. waking-up from hibernation), and conservatively reinitialize the 2D canvas by resetting the context (set canvas width/height) and redrawing, just in case.

=== Benefits ===

:* Empower the browser monitor resources to decide whether to drop canvas backing stores and in what order (LRU backgroung tabs?) in order to achieve better performance and stability. If web apps must handle resource eviction themselves, they may often free resources when not necessary, which may lead to unnecessary tab switching lag.
:* Make recovery from GPU context losses more robust.
:* Allow GPU-accelerated 2D canvases on platforms that are known to drop graphics contexts often or unpredictably.

=== Requests for this Feature ===

:* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/CQJXpXxO6dk E-mail thread on Chromium graphics-dev mailing list]</cite> <blockquote><p>"Is there any reason why we don't add a similar optional callback to the 2D context? (in reference to WebGL context loss API)" --Rik Cabanier, Adobe</p></blockquote>

== Proposed Solutions ==

The following solutions were pondered in the discussion thread cited above:
:* Generalize the WebGL context lost / context recovered API, so that it applies to all types of canvases.
:* Add a redraw callback on the canvas element

=== Retained Solution : Upstream the context lost/recovered API form the WebGL specification into the parent canvas specification. ===

:General Concept:
:* a renderingContextLost event is fired after the context is lost.
:* a renderingContextRestored event is fired immediately after a previously lost canvas context is brought back to a usable state. The canvas context is returned to its initial state and the canvas's backing store is blank (transparent black) when restored.

==== Processing Model ====
:Rendering context losses may be intended by the user agent (to resolve resource contention), or may be forced by external factors (e.g. a graphics driver reset).

:For convenience, the lost state should be accessible. To do so, the isContextLost method that is defined in the WebGLRenderingContext API should also exist in the CanvasRenderingContext2D API.

:Losing contexts (applies to 2d contexts, some aspects different for WebGL):
:* Intentional losses are only allowed if the contextRestored event is handled on the canvas element associated with the context.
:* The UA is free to use any set of rules to decide which contexts are dropped when and in what order.
:* The return value of isContextLost() may transition from false to true before the contextLost event is dispatched (like webGL).
:* All objects that depend on the content of the canvas (e.g. patterns, imageBitmaps) are neutered when the context is lost. The neutering propagates through creation dependency chains, so a Pattern created from an ImageBitmap created from and ImageBitmap created from a canvas will be neutered if the canvas's context is lost. This rule makes it safe for implementations to optimize their memory consumption by sharing pixel buffers between objects when possible.

:Restoring contexts (applies to 2d contexts, some aspects different for WebGL):
:* Between the time a context is recovered and invoking the listener for the contextRestored event, no other user code can be executed.
:* The app is responsible for re-creating the objects that were neutered when the context was lost.
:* There can only be one contextRestored event pending at a time. When there are multiple canvases to be restored, the next canvas to be restored can only be restored after any pending contextRestored events--from previously restored canvases--have been handled.
:* Context restoration can only be initiated by the user agent (can not by triggered by a script action).
:* A lost context can only be restored after its context lost event has been dispatched. This avoids synchronization inconsistencies with isContextLost().
:* The return value of isContextLost() transitions from true to false at the time that contextRestored event is dispatched (like webGL). So the contextRestored event listener is always the first task to be execute after the transition.

:Behavior when using a lost context (applies to 2d contexts, WebGL may behave differently):
:* All draw calls exit without drawing.
:* All rendering context API calls will throw the same exceptions as they would if called on a valid context.
:* All calls that read back canvas pixels from either the canvas element or the canvas rendering context (getImageData, toDataURL, toBlob, createPattern, creatImageBitmap, drawImage with lost canvas as source) will behave as they would if the canvas context were valid and blank (all transparent black pixels).
:* Using a Patterns or an ImageBitmap that was neutered because the context of its source canvas was lost will behave as if the Pattern or ImageBitmap were valid and blank (all transparent black pixels).

:Behaviors specific to GPU-accelerated implementation of 2d canvas (non-normative?)
:* If the context was lost due to a GPU-related failure, and the browser is actively restoring GPU functionality, and expects to restore in a timely manner, then context restoration should wait until the browser is ready to resume GPU functionality and the restored canvas should continue to be GPU-accelerated. Conversely, if GPU functionality is permanently disable or if it is unknow whether or how long it may take to resume GPU operation, then the canvas should be restored immediately without GPU-acceleration
:* If an accelerated canvas is resized while the GPU is temporarily unavailable, the creation of the new canvas buffer should be postponed until the GPU functionality is restored.
:* When getContext() is called while GPU functionality is temporarily unavailable:
:: a) If the canvas does not have an associated context, create an new unaccelerated context.
:: b) If the canvas already has an associate context, return the existing context even if it is in a lost state.

==== Limitations ====
:Web browsers will not reap the stability and performance rewards associated with this API with Web apps that do not provide at least a handler for the context recovered event. In order for this feature to improve the state of the web, apps will need to opt in to this new API, which unfortunately needs to be optional for backwards compatibility reasons.

==== Implementation ====
:Browser vendors should be highly motivated to implement this API in order to improve platform resilience and performance, especially on mobile platforms where RAM contention is often a critical issue.

==== Adoption ====
:Web App developers should be motivated to adopt this API in order to improve the stability of their products:
:* Avoid triggering out-of-memory crashes in the browser.
:* Avoid browser performance issues associated with having a given app in a background tab
:* Robustly recover from GPU failures.

Top tier apps (particularly from developers that track usage metrics) would be expected to enthusiastically adopt this API.

==== Specification ====
: WebGL would continue to behave as currently specified (http://www.khronos.org/webgl/wiki/HandlingContextLost), but the wording of the WebGL specification could be modified to refer to the parent canvas specification and would only specify differences in behavior with respect to 2d canvas.

==== Open Issues ====
:* Should there be a base CanvasRenderingContext interface that defines isContextLost and is inherited by both WebGLRenderingContext and CanvasRenderingContext2D?
:* Should there be loseContext/restoreContext methods. Could be very useful for testing purposes, or for app-initiated resource management.

[[Category:Proposals]]

Canvas Context Loss and Restoration

2013-10-18T17:59:51Z

Junov:

:2D Canvas Rendering contexts are currently required to have persistent backing stores. This proposal aims to relax that requirement by introducing an API to allows canvases to be discarded by the browser and re-drawn by the web application on demand.

== Use Case Description ==
:The 2d canvas backing store persistence requirement often leads to large amounts of RAM (or GPU memory) being consumed by canvas elements that are not actively used because they are off screen or in background tabs or occluded windows. Other types of RAM-greedy HTML elements can release resources in such cases.

:The expectation of canvas content persistence also makes it very difficult--if not impossible--for many web apps to recover from a GPU context reset in web browsers that store 2D canvas contents in GPU memory.

=== Current Limitations ===
:In theory, web apps do have the capability of discarding canvas backing stores (set canvas size to zero) and regenerating canvas the content. However, web apps are not and should not be expected to be responsible for resolving resource contention issues. The browser is responsible for monitoring resource usage and availability and is expected to take all necessary and reasonable measures to avoid crashes, hangs, and catastrophic performance degradations that may be caused by resource contention. Under the current specification, browsers have no options for evicting resources held by 2D canvases because there are no means of guaranteeing that the application will redraw the contents when needed.

=== Current Usage and Workarounds ===
:Web apps can track events to detect when the page is no longer visible (http://www.w3.org/TR/page-visibility/) and deallocate backing stores at that time by setting the size of the canvas element to 0. Conversely, they can detect when the page is visible again and reinitialize at that time

:Web apps can track events that are often associated with GPU context losses (e.g. waking-up from hibernation), and conservatively reinitialize the 2D canvas by resetting the context (set canvas width/height) and redrawing, just in case.

=== Benefits ===

:* Empower the browser monitor resources to decide whether to drop canvas backing stores and in what order (LRU backgroung tabs?) in order to achieve better performance and stability. If web apps must handle resource eviction themselves, they may often free resources when not necessary, which may lead to unnecessary tab switching lag.
:* Make recovery from GPU context losses more robust.
:* Allow GPU-accelerated 2D canvases on platforms that are known to drop graphics contexts often or unpredictably.

=== Requests for this Feature ===

:* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/CQJXpXxO6dk E-mail thread on Chromium graphics-dev mailing list]</cite> <blockquote><p>"Is there any reason why we don't add a similar optional callback to the 2D context? (in reference to WebGL context loss API)" --Rik Cabanier, Adobe</p></blockquote>

== Proposed Solutions ==

The following solutions were pondered in the discussion thread cited above:
:* Generalize the WebGL context lost / context recovered API, so that it applies to all types of canvases.
:* Add a redraw callback on the canvas element

=== Retained Solution : Upstream the context lost/recovered API form the WebGL specification into the parent canvas specification. ===

:General Concept:
:* a renderingContextLost event is fired after the context is lost.
:* a renderingContextRestored event is fired immediately after a previously lost canvas context is brought back to a usable state. The canvas context is returned to its initial state and the canvas's backing store is blank (transparent black) when restored.

==== Processing Model ====
:Rendering context losses may be intended by the user agent (to resolve resource contention), or may be forced by external factors (e.g. a graphics driver reset).

:For convenience, the lost state should be accessible. To do so, the isContextLost method that is defined in the WebGLRenderingContext API should also exist in the CanvasRenderingContext2D API.

:Losing contexts (applies to 2d contexts, some aspects different for WebGL):
:* Intentional losses are only allowed if the contextRestored event is handled on the canvas element associated with the context.
:* The UA is free to use any set of rules to decide which contexts are dropped when and in what order.
:* The return value of isContextLost() may transition from false to true before the contextLost event is dispatched (like webGL).
:* All objects that depend on the content of the canvas (e.g. patterns, imageBitmaps) are neutered when the context is lost. The neutering propagates through creation dependency chains, so a Pattern created from an ImageBitmap created from and ImageBitmap created from a canvas will be neutered if the canvas's context is lost. This rule makes it safe for implementations to optimize their memory consumption by sharing pixel buffers between objects when possible.

:Restoring contexts (applies to 2d contexts, some aspects different for WebGL):
:* Between the time a context is recovered and invoking the listener for the contextRestored event, no other user code can be executed.
:* The app is responsible for re-creating the objects that were neutered when the context was lost.
:* There can only be one contextRestored event pending at a time. When there are multiple canvases to be restored, the next canvas to be restored can only be restored after any pending contextRestored events--from previously restored canvases--have been handled.
:* Context restoration can only be initiated by the user agent (can not by triggered by a script action).
:* A lost context can only be restored after its context lost event has been dispatched. This avoids synchronization inconsistencies with isContextLost().
:* The return value of isContextLost() transitions from true to false at the time that contextRestored event is dispatched (like webGL). So the contextRestored event listener is always the first task to be execute after the transition.

:Behavior when using a lost context (applies to 2d contexts, WebGL may behave differently):
:* All draw calls exit without drawing.
:* All rendering context API calls will throw the same exceptions as they would if called on a valid context.
:* All calls that read back canvas pixels from either the canvas element or the canvas rendering context (getImageData, toDataURL, toBlob, createPattern, creatImageBitmap, drawImage with lost canvas as source) will behave as they would if the canvas context were valid and blank (all transparent black pixels).
:* Using a Patterns or an ImageBitmap that was neutered because the context of its source canvas was lost will behave as if the Pattern or ImageBitmap were valid and blank (all transparent black pixels).

:Behaviors specific to GPU-accelerated implementation of 2d canvas (non-normative?)
:* If the context was lost due to a GPU-related failure, and the browser is actively restoring GPU functionality, and expects to restore in a timely manner, then context restoration should wait until the browser is ready to resume GPU functionality and the restored canvas should continue to be GPU-accelerated. Conversely, if GPU functionality is permanently disable or if it is unknow whether or how long it may take to resume GPU operation, then the canvas should be restored immediately without GPU-acceleration
:* If an accelerated canvas is resized while the GPU is temporarily unavailable, the creation of the new canvas buffer should be postponed until the GPU functionality is restored.
:* When getContext() is called while GPU functionality is temporarily unavailable:
:: a) If the canvas does not have an associated context, create an new unaccelerated context.
:: b) If the canvas already has an associate context, return the existing context even if it is in a lost state.

==== Limitations ====
:Web browsers will not reap the stability and performance rewards associated with this API with Web apps that do not provide at least a handler for the context recovered event. In order for this feature to improve the state of the web, apps will need to opt in to this new API, which unfortunately needs to be optional for backwards compatibility reasons.

==== Implementation ====
:Browser vendors should be highly motivated to implement this API in order to improve platform resilience and performance, especially on mobile platforms where RAM contention is often a critical issue.

==== Adoption ====
:Web App developers should be motivated to adopt this API in order to improve the stability of their products:
:* Avoid triggering out-of-memory crashes in the browser.
:* Avoid browser performance issues associated with having a given app in a background tab
:* Robustly recover from GPU failures.

Top tier apps (particularly from developers that track usage metrics) would be expected to enthusiastically adopt this API.

==== Specification ====
: WebGL would continue to behave as currently specified (http://www.khronos.org/webgl/wiki/HandlingContextLost), but the wording of the WebGL specification could be modified to refer to the parent canvas specification and would only specify differences in behavior with respect to 2d canvas.

==== Open Issues ====
:* Should there be a base CanvasRenderingContext interface that defines isContextLost and is inherited by both WebGLRenderingContext and CanvasRenderingContext2D?
:* Should there be loseContext/restoreContext methods. Could be very useful for testing purposes, or for app-initiated resource management.

[[Category:Proposals]]

Canvas Context Loss and Restoration

2013-10-18T17:57:25Z

Junov: Created page with ":2D Canvas Rendering contexts are currently required to have persistent backing stores. This proposal aims to relax that requirement by introducing an API to allows canvases t..."

:2D Canvas Rendering contexts are currently required to have persistent backing stores. This proposal aims to relax that requirement by introducing an API to allows canvases to be discarded by the browser and re-drawn by the web application on demand.

== Use Case Description ==
:The 2d canvas backing store persistence requirement often leads to large amounts of RAM (or GPU memory) being consumed by canvas elements that are not actively used because they are off screen or in background tabs or occluded windows. Other types of RAM-greedy HTML elements can release resources in such cases.

:The expectation of canvas content persistence also makes it very difficult--if not impossible--for many web apps to recover from a GPU context reset in web browsers that store 2D canvas contents in GPU memory.

=== Current Limitations ===
:In theory, web apps do have the capability of discarding canvas backing stores (set canvas size to zero) and regenerating canvas the content. However, web apps are not and should not be expected to be responsible for resolving resource contention issues. The browser is responsible for monitoring resource usage and availability and is expected to take all necessary and reasonable measures to avoid crashes, hangs, and catastrophic performance degradations that may be caused by resource contention. Under the current specification, browsers have no options for evicting resources held by 2D canvases because there are no means of guaranteeing that the application will redraw the contents when needed.

=== Current Usage and Workarounds ===
:Web apps can track events to detect when the page is no longer visible (http://www.w3.org/TR/page-visibility/) and deallocate backing stores at that time by setting the size of the canvas element to 0. Conversely, they can detect when the page is visible again and reinitialize at that time

:Web apps can track events that are often associated with GPU context losses (e.g. waking-up from hibernation), and conservatively reinitialize the 2D canvas by resetting the context (set canvas width/height) and redrawing, just in case.

=== Benefits ===

:* Empower the browser monitor resources to decide whether to drop canvas backing stores and in what order (LRU backgroung tabs?) in order to achieve better performance and stability. If web apps must handle resource eviction themselves, they may often free resources when not necessary, which may lead to unnecessary tab switching lag.
:* Make recovery from GPU context losses more robust.
:* Allow GPU-accelerated 2D canvases on platforms that are known to drop graphics contexts often or unpredictably.

=== Requests for this Feature ===

:* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/CQJXpXxO6dk E-mail thread on Chromium graphics-dev mailing list]</cite> <blockquote><p>"Is there any reason why we don't add a similar optional callback to the 2D context? (in reference to WebGL context loss API)" --Rik Cabanier, Adobe</p></blockquote>

== Proposed Solutions ==

The following solutions were pondered in the discussion thread cited above:
:* Generalize the WebGL context lost / context recovered API, so that it applies to all types of canvases.
:* Add a redraw callback on the canvas element

=== Retained Solution : Upstream the context lost/recovered API form the WebGL specification into the parent canvas specification. ===

:General Concept:
:* a renderingContextLost event is fired after the context is lost.
:* a renderingContextRestored event is fired immediately after a previously lost canvas context is brought back to a usable state. The canvas context is returned to its initial state and the canvas's backing store is blank (transparent black) when restored.

==== Processing Model ====
:Rendering context losses may be intended by the user agent (to resolve resource contention), or may be forced by external factors (e.g. a graphics driver reset).

:For convenience, the lost state should be accessible. To do so, the isContextLost method that is defined in the WebGLRenderingContext API should also exist in the CanvasRenderingContext2D API.

:Losing contexts (applies to 2d contexts, some aspects different for WebGL):
:* Intentional losses are only allowed if the contextRestored event is handled on the canvas element associated with the context.
:* The UA is free to use any set of rules to decide which contexts are dropped when and in what order.
:* The return value of isContextLost() may transition from false to true before the contextLost event is dispatched (like webGL).
:* All objects that depend on the content of the canvas (e.g. patterns, imageBitmaps) are neutered when the context is lost. The neutering propagates through creation dependency chains, so a Pattern created from an ImageBitmap created from and ImageBitmap created from a canvas will be neutered if the canvas's context is lost. This rule makes it safe for implementations to optimize their memory consumption by sharing pixel buffers between objects when possible.

:Restoring contexts (applies to 2d contexts, some aspects different for WebGL):
:* Between the time a context is recovered and invoking the listener for the contextRestored event, no other user code can be executed.
:* The app is responsible for re-creating the objects that were neutered when the context was lost.
:* There can only be one contextRestored event pending at a time. When there are multiple canvases to be restored, the next canvas to be restored can only be restored after any pending contextRestored events--from previously restored canvases--have been handled.
:* Context restoration can only be initiated by the user agent (can not by triggered by a script action).
:* A lost context can only be restored after its context lost event has been dispatched. This avoids synchronization inconsistencies with isContextLost().
:* The return value of isContextLost() transitions from true to false at the time that contextRestored event is dispatched (like webGL). So the contextRestored event listener is always the first task to be execute after the transition.

:Behavior when using a lost context (applies to 2d contexts, WebGL may behave differently):
:* All draw calls exit without drawing.
:* All rendering context API calls will throw the same exceptions as they would if called on a valid context.
:* All calls that read back canvas pixels from either the canvas element or the canvas rendering context (getImageData, toDataURL, toBlob, createPattern, creatImageBitmap, drawImage with lost canvas as source) will behave as they would if the canvas context were valid and blank (all transparent black pixels).
:* Using a Patterns or an ImageBitmap that was neutered because the context of its source canvas was lost will behave as if the Pattern or ImageBitmap were valid and blank (all transparent black pixels).

:Behaviors specific to GPU-accelerated implementation of 2d canvas (non-normative?)
:* If the context was lost due to a GPU-related failure, and the browser is actively restoring GPU functionality, and expects to restore in a timely manner, then context restoration should wait until the browser is ready to resume GPU functionality and the restored canvas should continue to be GPU-accelerated. Conversely, if GPU functionality is permanently disable or if it is unknow whether or how long it may take to resume GPU operation, then the canvas should be restored immediately without GPU-acceleration
:* If an accelerated canvas is resized while the GPU is temporarily unavailable, the creation of the new canvas buffer should be postponed until the GPU functionality is restored.
:* When getContext() is called while GPU functionality is temporarily unavailable:
:: a) If the canvas does not have an associated context, create an new unaccelerated context.
:: b) If the canvas already has an associate context, return the existing context even if it is in a lost state.

==== Limitations ====
:Web browsers will not reap the stability and performance rewards associated with this API with Web apps that do not provide at least a handler for the context recovered event. In order for this feature to improve the state of the web, apps will need to opt in to this new API, which unfortunately needs to be optional for backwards compatibility reasons.

==== Implementation ====
:Browser vendors should be highly motivated to implement this API in order to improve platform resilience and performance, especially on mobile platforms where RAM contention is often a critical issue.

==== Adoption ====
:Web App developers should be motivated to adopt this API in order to improve the stability of their products:
:* Avoid triggering out-of-memory crashes in the browser.
:* Avoid browser performance issues associated with having a given app in a background tab
:* Robustly recover from GPU failures.

Top tier apps from developers that track usage metrics would be expected to enthusiastically adopt this API.

==== Specification ====
: WebGL would continue to behave as currently specified (http://www.khronos.org/webgl/wiki/HandlingContextLost), but the wording of the WebGL specification could be modified to refer to the parent canvas specification and would only specify differences in behavior with respect to 2d canvas.

==== Open Issues ====
:* Should there be a base CanvasRenderingContext interface that defines isContextLost and is inherited by both WebGLRenderingContext and CanvasRenderingContext2D?
:* Should there be loseContext/restoreContext methods. Could be very useful for testing purposes, or for app-initiated resource management.

[[Category:Proposals]]