CanvasColorSpace

Use Case Description

• Contents displayed through a canvas element should be color managed in order to minimize differences in appearance across browsers and display devices. Improving color fidelity matters a lot for artistic uses (e.g. photo and paint apps) and for e-commerce (product presentation).
• Canvases should be able to take advantage of the full color gamut of the display device.
• Creative apps that do image manipulation generally prefer compositing, filtering and interpolation calculations to be performed in a linear color space.

Current Limitations

• The color space of canvases is undefined in the current specification.
• The bit-depth of canvases is currently fixed to 8 bits per component, which is below the capabilities of some monitors. Monitors with higher contrast ratios require more bits per component to avoid banding.

Current Usage and Workarounds

The lack of color space interoperability is hard to work around. With some browser implementations that color correct images drawn to canvases by applying the display profile, apps that want to use canvases for color corrected image processing are stuck doing convoluted workarounds, such as:

• reverse-engineer the display profile by drawing test pattern images to the canvas and inspecting the color corrected result via getImageData
• bypass CanvasRenderingContext2D.drawImage() and use image decoders implemented in JavaScript to extract raw image data that was not tainted by the browser's color correction behavior.

An aspect of current implementations that is interoperable is that colors match between CSS/HTML and canvases:

• A color value used as a canvas drawing style will have the same appearance as if the same color value were used as a CSS style
• An image resource drawn to a canvas element will have the same appearance as if it were displayed as the replaced content of an HTML element or used as a CSS style value.

This color matching behavior needs to be preserved to avoid breaking pre-existing content.

Some implementations convert images drawn to canvases to the sRGB color space. This has the advantage of making the color correction behavior device independent, but it clamps the gamuts of the rendered content to the sRGB gamut, which is significantly narrower than the gamuts of some current consumer devices.

Requests for this Feature

• [1]

  Allow 2dcontexts to use deeper color buffers

• [2]

  Wrong color profile with 2D canvas

• Engineers from the Google Photos, Maps and Sheets teams have expressed a desire for canvases to become color managed. Particularly for the use case of resizing an imaging, using a canvas, prior to uploading it to the server, to save bandwidth. The problem is that the images retrieved from a canvas are in an undefined color space and no color space information is encoded by toDataURL or toBlob.

Proposed Solutions

Proposed solution: CanvasColorSpace

Add a canvas color space creation parameter that allows user code to chose between backwards compatible behavior and color managed behaviors The same color space option would exist in the ImageData and ImageBitmap interfaces.

Processing Model

The color-space canvas creation parameter

IDL:

enum CanvasColorSpace {
  "legacy-srgb",
  "srgb",
  "linear-rec-2020",
  "optimal"
};

dictionary CanvasRenderingContext2DSettings {
  boolean alpha = true;
  CanvasColorSpace color-space = "legacy-srgb";
};

Example:

canvas.getContext('2d', { color-space: "srgb"})

The legacy-srgb color space

• Assures backwards compatible behavior
• Guarantees color matching with CSS and HTML content
• Color management behavior is implementation specific, may not use strict sRGB space, but is expected to be near sRGB. For example, could be display referred color space.
• toDataURL/toBlob produce resources with no color profile (backwards compat)
• Image resources with no color profile are never color corrected (backwards compat). This rule and the previous one allow for lossless toDataURL/drawImage round trips, which is a significant use case.

The srgb color space

• May break color matching with CSS on implementations that do not color-manage CSS.
• 8 bit unsigned integers per color component.
• All content drawn into the canvas must be color corrected to sRGB
• displayed canvases must be color corrected for the display if a display color profile is available. This color correction happens downstream at the compositing stage, and has no script-visible side-effects.
• Compositing, filtering and interpolation operations must perform all arithmetic in linear sRGB space.
• toDataURL/toBlob produce resources tagged as being in the sRGB colorspace
• Images with no color profile, when drawn to the canvas, are assumed to already be in the sRGB color space.

The rec-2020 color space

• Color space provided for wide gamut support without increasing memory cost.
• 8 bit unsigned integers per color component.
• All content drawn into the canvas must be color corrected to the rec-2020 color space.
• Displayed canvases must be color corrected for the display if a display color profile is available. This color correction happens downstream at the compositing stage, and has no script-visible side-effects. If there is no display color profile, the user agent should assume the display uses sRGB.
• Compositing, filtering and interpolation operations must perform all arithmetic in linear rec-2020 space.
• toDataURL/toBlob produce image resources in the rec-2020 colorspace.
• Images with no color profile, when drawn to the canvas, are assumed to be in the sRGB color space, and must therefore be converted from sRGB to rec-2020.

The linear-rec-2020 color space

• Color space provided for wide gamut and high dynamic range rendering.
• User agents may decide not to support the mode, based on host machine capabilities
• Uses 16-bit floating point representation.
• The color space corresponds to ITU-R Recommendation BT.2020, without gamma compression.
• toDataURL/toBlob convert image data to the rec-2020 color space (with gamma), and produce image resources with at least 12 bits per color component, if the format supports it. Thus, in the case of the png format, which supports 8 or 16 bits per component, 16bpc would be used.
• Image with no color profile, when drawn to the canvas, are assumed to be in the sRGB color space, and are converted to linear-rec-2020 for the purpose of the draw.

The optimal color space

The "optimal" option lets the user agent decide which space is optimal for the current display device based on the device's capabilities and color profile characteristics.

• This option may not select "legacy-srgb"
• Graphics devices with color gamuts that extend significantly beyond the sRGB color space should cause the UA to favor rec-2020; and linear linear-rec-2020 should be used for displays with high dynamic ranges.
• Graphics devices that would not produce noticeably higher quality visual results in rec-2020 or linear-rec-2020 should cause the UA to favor "srgb".

Feature detection

Rendering context objects are to expose a new "settings" attribute, which represents the settings that were successfully applied at context creation time.

Note: An alternative approach that was considered was to augment the probablySupportsContext() API by making it check the second argument. That approach is difficult to consolidate with how dictionary argument are meant to work, where unsupported entries are just ignored.

ImageBitmap

ImageBitmap objects are augmented to have an internal color space attribute of type CanvasColorSpace. The colorSpaceConversion creation attribute is to be augmented with new enum values for coercing conversions to a specific CanvasColorSpace at creation time.

ImageData

IDL

typedef (Uint8ClampedArray or Float32Array) ImageDataArray;

[Constructor(unsigned long sw, unsigned long sh, optional CanvasColorSpace colorSpace = "legacy-srgb"),
 Constructor(ImageDataArray data, unsigned long sw, optional unsigned long sh, optional CanvasColorSpace colorSpace),
 Exposed=(Window,Worker)]
interface ImageData {
  readonly attribute unsigned long width;
  readonly attribute unsigned long height;
  readonly attribute ImageDataArray data;
  readonly attribute CanvasColorSpace colorSpace;
};

• data is a Uint8ClampedArray if colorSpace is "srgb" or "legacy-srgb"
• data is a Float32Array if colorSpace is "linear-rec-2020"
• getImageData() produces an ImageData object in the same color space as the source canvas
• putImageData() performs a color space conversion to the color space of the destination canvas.

Limitations

• No support for arbitrary color spaces and bit depth. This capability could be added in the future. The current proposal attempts to solve the problem with a minimal API surface, and keeps the implementation scope reasonable. The extensible design will allow us to extend the capabilities in the future if necessary. The rec-2020 space was chosen for its very wide gamut and its non-virtual primary colors, which strikes a balance that is deemed practical.
• toDataURL is lossy when us on a canvas that is in the linear-rec-2020 space. Possible future improvements could solve or mitigate this issue by adding more file formats or adding options to specify the resource color space.
• ImageData uses float32, which is inefficient due to memory consumption and necessary conversion operations. Float32 was chosen because it is convenient for manipulation (e.g. image processing) due to its native support in JavaScript (and current CPUs). A possible extension would be to add and option for rec-2020 content to be encoded as float16s packed into Uint16 values.

Security and privacy issues

Some current implementations of CanvasRenderingContext2D color correct image resources for the display as they are drawn to the canvas. In other words, the canvas is in output referred color space. This is a known fingerprinting vulnerability since it exposes the user's display's color profile to scripts via getImageData. The current proposal does not solve the fingerprinting issue because it will still exist in legacy-srgb. To solve the problem, implementations must color-correct CSS colors, then by extension, legacy-srgb mode will be in the true sRGB color space by virtue of the color matching rules outlined above. When that becomes the case, images drawn to canvases will be color corrected to sRGB, which solves the problem. There is resistance to adopting this model because going through an sRGB intermediate is lossy compared to directly color correcting images for the display in a single pass (may cause banding and gamut clipping). This feature proposal mitigates the lossiness argument thanks to the linear-rec-2020 option.

Implementation notes

• Because float16 arithmetic is supported by many GPUs, but not by CPUs, implementations should probably opt to not support rec-2020 on hardware that does not provide any native support.
• When available, the srgb colorspace should use GPU API extension for SRGB support. This will streamline the conversion overhead for performing filtering and compositing in linear space.

Adoption

Lack of color management and color interoperability is a longstanding complaint about the canvas API. Authors of games and imaging apps are expected to be enthusiastic adopters.

History

This proposal was originally incubated in the Khronos 3D Web group, with the participation of engineers from Google, Microsoft, Apple, Nvidia, and others.