ImageBitmap Options

This page is a Work in Progress

Problem to be solved: Loading and preparing image resources for rendering to 2D canvas contexts and WebGL contexts is often inefficient using existing APIs.

Use Case Description

WebGL applications often use image media for transferring non-image data (illumination maps, vector maps, etc.) For such applications the image data must usually be decoded without colorspace conversions and without alpha premultiplications.

Current Usage and Workarounds

Currently, there are several ways to apply arbitrary transformations on the client side, many of which involve intermediate copies of the image data. Some solutions involve DataURLs, temporary canvases, or direct manipulation of image data arrays, which are inefficient. Preparing the resources asynchronously also presents an additional challenge to the web developer.

WebGL applications can request that colorspace conversions be skipped at texture upload time by specifying pixel storage parameters, but this currently causes a synchronous image decode, because it is not known in advance that the image will need to be decoded in a special way. This latency is a significant problem for applications that upload many images to the GPU, such as Google Maps.

ImageBitmap offers an opportunity to provide exactly the desired image decoding options when the image is fetched, so that when the ImageBitmap is ready, it is in exactly the desired form for upload to the GPU.

Benefits

With the right API support, image resources can be prepared asynchronously and/or in advance to avoid rendering latency issues. Furthermore, the preparation steps could be performed by native code, and in many case they can be integrated directly into the image decoding step without any additional memory consumption and little or no additional computational burden.

Requests for this Feature

• June 2013 whatwg thread
• July 2013 whatwg thread

Proposed Solutions

ImageBitmap Construction Options

Add an optional options argument to createImageBitmap to specify image treatments to be applied when creating the ImageBitmap object.

Strongly desired options

• enum imageOrientation, default = 'none'

  (Higher priority) 'none': Do not change orientation.

  (Higher priority) 'flipY': Flip image vertically, disregarding image orientation meta data (EXIF or other), if any.

  (Lower priority) 'topLeft': identify visual orientation from image media meta data and re-order data so top-down, left-to-right processing for display results in correct visual orientation. If the image source is an ImageBitmap, the image data will be flipped vertically if and only if the source was created with bottomLeft orientation. If the image source holds no orientation information, the image orientation is not changed.

  (Lower priority) 'bottomLeft': identify visual orientation from image media meta data and re-order data so bottom-up, left-to-right processing for display results in correct visual orientation. If the image source is an ImageBitmap, the image data will be flipped vertically if and only if the source was created with topLeft orientation. If the image source holds no orientation information, the image is flipped vertically.

• enum colorspaceConversion, default = 'default'

  'none': The ImageBitmap must be created by decoding the source image media without performing any colorspace conversions. This means that the image decoding algorithm must ignore color profile meta-data embedded in the image source as well as the display device color profile. colorspaceConversion only applies when creating ImageBitmaps from bitmap media data (i.e. blobs and image elements with non-vector images)

  'default': Implementation-specific behavior, possibly optimized for the implementation's graphics framework.

• enum premultiplyAlpha, default = 'default'

  'none': Image sources that are premultiplied by alpha will have their color components divided by alpha and image sources that are not premultiplied will be left untouched.

  'default': Implementation-specific behavior, possibly optimized for the implementation's graphics framework.

  When premultiplyAlpha is set to 'none', the division by alpha may result in loss of precision, which must be avoided whenever possible. For example, if the image source is a blob or an image element, the decoded image may already be available in premultipled form. Rather than divide the already decoded data, the user agent must re-decode the image without premultiplying, thus ensuring that the ImageBitmap is delivered with the highest quality data.

  Image sources that do not have an alpha channel are not affected by this option.

  Caution: Canvases are assumed to be premultiplied image sources, which may not always be accurate.

Other potentially useful options

• DOMString? crossOrigin: The crossOrigin option is a [CORS settings attribute]. It is consulted when constructing an ImageBitmap from the URL of an image. Because ImageBitmaps can only be constructed from same-origin resources, in order to construct an ImageBitmap from the URL of a cross-origin image resource, the crossOrigin option must be specified, and the server must grant the request. Otherwise the Promise returned from createImageBitmap will be rejected.
• resizedWidth, resizedHeight: dimension to which the image should be resized. Could also be expressed using scale factors
• resizeFilter: filtering method used for scaling the image (pixelated, low quality, high quality, etc.)
• frameIndex: for creating an image bitmap from multi-frame image media
• flipX: like flipY but horizontal
• colorspace: to decode images to a colorspace other than the device colorspace (sRGB, Adobe RGB, YUV, YCbCr, etc.)
• custom color tranforms: color matrix, 3D Look up tables, per component look up tables, bias, gain, ...

Processing Model

These options affect the internal color representation used to store the ImageBitmap's bitmap data. The only observable effects would be on color fidelity. There would be no re-interpretation of color values. This is important because treating unpremultiplied colors as premultiplied is evil. For example, in blink's implementation, setting both options to 'none' would have no side-effects on the graphical results of CanvasRenderingContext2D.drawImage(). The only difference would be that the color-space conversion and the multiplication by alpha would be deferred to drawImage(), instead of being applied during ImageBitmap creation. The usefulness of 'none' is strictly for preventing losses in precision and gamut that may result from dividing by alpha and/or reversing the default color-space conversion when using APIs that would require it (WebGL's UNPACK_PREMULTIPLY_ALPHA_WEBGL and UNPACK_COLORSPACE_CONVERSION_WEBGL pixel storage parameters)

Limitations

TBD Cases not covered by this solution in relation to the problem description; other problems with this solution, if any.

Implementation

TBD Description of how and why browser vendors would take advantage of this feature.

Adoption

• Easier to develop asychronous image resource loading and preparation
• Better performance that what can be implemented using current APIs
• Applications can rely on the browser resource management facilities for caching decoded image data with the desired options baked-in and for evicting pre-decoded resources in low memory conditions to avoid out-of-memory crashes.

See Also

Related is the ImageBitmap Constructor taking URL.