Changes to ImageBitmap for OffscreenCanvas

WORK IN PROGRESS Amendments to ImageBitmap to provide zero-copy paths for moving pixel data and for working with the OffscreenCanvas proposal

Use Case Description

A common complaint from web developers writing applications that manipulate large images is that the browser tends to consume large amounts of RAM and GPU memory. Part of this is due to intermediate copies of image data that are made by the browser and transient copies that are kept around, awaiting garbage collection. Another complaint is that manipulation large images on the browser's main thread often makes applications janky. Specific use cases:

• Generating or processing pixel data in javascript, possibly in a Worker thread, and bringing it to screen.
• Taking a snapshot of canvas-rendered content and uploading it as an image file to a remote server, possibly using XHR2 with a progress updates.
• Using 2D canvas to produce images that are subsequently used as textures in WebGL.
• Transferring a canvas-rendered image to anywhere an image URL can be used (e.g. CSS properties)
• Save locally rendered canvas content to a local disk.
• Stream snapshots of a local canvas to a remote server. ex: Broadcasting a live presentation rendered in WebGL
• Appending a locally rendered canvas image to form data as an image file.
• Save a canvas image locally with the anchor "download" attribute.
• Capturing a snapshot from a video stream (possibly WebRTC), and uploading the snapshot to a remote server.

Note: Many of these use cases were reported by web developpers in support of implementing the HTMLCanvasElement.toBlob interface. Below we counter-propose that putting toBlob on ImageBitmap will serve those use cases better or as well as having toBlob on <canvas>.

Current Limitations

The current API often forces the web developer to go through a canvas for moving pixel data. Because canvases are mutable, intermediate copies are often unavoidable, and may result in unnecessary multiply and divide by alpha, which is expensive and lossy. ImageBitmap, being an opaque immutable object type would allow for low friction means of moving pixel data, but it is missing some functionality that prevents it from being a universal pixel vehicle. In particular, it is missing transfer methods (required for zero-copy behavior).

Current Usage and Workarounds

Some evidence that this feature is desperately needed on the web. You may provide a separate examples page for listing these.

For sending image data to a remote server, a compressed image file in binary form is the preferred vehicle. This can currently be achieved by using HTMLCanvasElement.toBlob (or equivalent interface) with web browser that provide that API, or with this approach on other browsers: http://stackoverflow.com/questions/4998908/convert-data-uri-to-file-then-append-to-formdata. This method has some disadvantages: because it lives on a DOM interface, it can only be invoked from the main thread; because the canvas backing editable, the implementation is required to make a read-only snapshot for itself; In many use cases, the canvas itself represent an additional copy of the image data. Preventing the proliferation of copies of image data in RAM is very important for applications that manipulate large images (e.g. full res photos, maps), which are vulnerable to malfunctions cause by out-of-memory errors, particularly on mobile.

Experience has shown that garbage collection is often a performance liability when dealing with large temporary objects, particularly those that consume GPU memory. The ability to explicity discard image buffers when they are no longer needed has proven to be widely useful with 2D canvas, which offer this possibility indirectly by setting the intrinsic size to 0. This practice allows resources (RAM, GPU memory) to be freed as early as possible, hence avoiding process bloat, and it helps reduce the frequency of garbage collections, which can cause jank. Using ImageBitmaps as currently spec'ed implies losing this advantage.

Benefits

The proposed additions to the ImageBitmap interface aim to provide leaner browser memory consumption and fast, smoother performance for the above mentioned use cases.

Requests for this Feature

• Source

  I would like this feature ...

Proposed Solution

[Exposed=(Window,Worker)]
interface ImageBitmap implements Transferable {
  readonly attribute unsigned long width;
  readonly attribute unsigned long width;
  void close();
  // Throws a SecurityError if the OffscreenCanvas's origin-clean flag
  // is set to false.
  Promise<Blob> toBlob(optional DOMString type, any... arguments);
}

interface

Making ImageBitmap Transferrable allows it to be used as a zero-copy vehicle for passing pixel data between threads, hence reducing peak RAM consumption and avoiding the CPU cost of performing a copy.

Offering toBlob on ImageBitmap, as opposed to HTMLCanvasElement:

• Allows to toBlob be invoked from a worker, which help reduce jank on the main thread (even if the encode is on a separate thread, make a copy and interacting with the blob store can jank the main thread).
• Allows the data flow to bypass the canvas (for cases where the source of the image data is not a canvas), therefore avoiding an alpha multiply+divide in some cases, and avoiding an unnecessary intermediate (the canvas itself)
• Because ImageBitmap is immutable, no safety copy is required for toBlob to operate asynchronously. A counter argument to this is that an implementation of HTMLCanvasElement.toBlob could perform the copy lazily, only when the canvas backing is about to be mutated while a toBlob is in progress, thus performing a copy only in use case that require it. Though this is true, it results in performance characteritcs that may seem idiosyncratic to developers (i.e. the preformance of toBlob degrades if something is done to the canvas after calling toBlob.), as opposed to a flow that guarantees zero copies: OffscreenCanvas.transferToImageBitmap, followed by ImageBitmap.toBlob.

The close() method effectively neuters the ImageBitmap. It is a means of explicitly de-allocating large resources, which avoids waiting for garbage collection, and therefore reduces the frequency of GCs and brings down peak RAM consumption.

Processing Model

Explanation of the changes introduced by this solution. It explains how the document is processed, and how errors are handled. This should be very clear, including things such as event timing if the solution involves events, how to create graphs representing the data in the case of semantic proposals, etc.

Limitations

Cases not covered by this solution in relation to the problem description; other problems with this solution, if any.

Implementation

Description of how and why browser vendors would take advantage of this feature.

Adoption

Reasons why page authors would use this solution.