A user account is required in order to edit this wiki, but we've had to disable public user registrations due to spam.

To request an account, ask an autoconfirmed user on Chat (such as one of these permanent autoconfirmed members).

Canvas Batch drawImage: Difference between revisions

From WHATWG Wiki
Jump to navigation Jump to search
(Created page with ":Batching drawImage calls to achieve near-native performance for sprite/image based animations and games. == Use Case Description == Many web applications and games use 2D Ca...")
 
 
(7 intermediate revisions by 2 users not shown)
Line 5: Line 5:


=== Current Limitations ===
=== Current Limitations ===
When calling drawImage hundreds or thousands of times per animation frame, API bindings overhead and internal bookkeeping costs associated with individual draw calls become a significant performance bottleneck, which prevents web application from achieving near-native performance levels.
When calling drawImage hundreds or thousands of times per animation frame, API bindings overhead and internal bookkeeping costs associated with individual draw calls become a significant performance bottleneck, which prevents web applications from achieving near-native performance levels.


=== Current Workaround ===
=== Current Workaround ===
Line 21: Line 21:


=== Batch versions of drawImage ===
=== Batch versions of drawImage ===
:Additional overloads of drawImage that accept array arguments.
:New batch variants of drawImage that accept array arguments.


==== Processing Model ====
==== Processing Model ====
:We would have new variants of the existing drawImage overloads where all the arguments are arrays (Float32Array for numeric arguments), and other variants where all the arguments except for the image are arrays. The case where the image is not an array would be equivalent to having an array where each element is the same image (convenient for sprite sheets, and more efficient than using an array)
:We would have new variants of the existing drawImage method where the numerical arguments are are packed into a Float32Array. The image argument may or may not be an array. If it is not an array, the same source image is used for for each draw. When rendering from a single sprite sheet, it would be preferable to not specify the image argument as an array in order to minimize bindings overhead and redundant parameter validation.


:An INDEX_SIZE_ERR DOM exception would be thrown if not all the array arguments have the same size.
==== Suggested IDL ====
 
<pre>
 
enum CanvasDrawImageParameterFormat { position, destination-rectangle, source-and-destination-rectangles, source-rectangle-and-transform};
 
interface CanvasRenderingContext2D {
    ...
    void drawImageBatch(sequence&lt;CanvasImageSource> image, CanvasDrawImageParameterFormat parameterFormat, Float32Array drawParameters);
    void drawImageBatch(CanvasImageSource image, ParameterFormat parameterFormat, Float32Array drawParameters);
}
</pre>
 
:The drawParameters argument is to be interpreted as a table in row-major order. Where each row represents a single draw and each column represents an individual parameter. The mapping of the columns to draw parameters depends on the parameterFormat argument.
:* position: dx, dy
:* destination-rectangle: dx, dy, dw, dh
:* source-and-destination-rectangles: sx, sy, sw, sh, dx, dy, dw, dh
:* source-rectangle-and-transform: sx, sy, sw, sh, a, b, c, d, e, f
 
:The parameters sx, sy, sw, sh, dx, dy, dw, and dh have the same meaning as with drawImage()
:The parameters a, b, c, d, e, f have the same meaning as with transform()
:With 'source-rectangle-and-transform' the destination rectangle is a unit square defined by the vertices (0, 0), (1, 0), (1, 1), and (0, 1). The destination rectangle is transformed by the transform defined by a, b, c, d, e and f, and the by the canvas's current transform.
 
Feedback Anne: Perhaps rather than overloading we should introduce new methods? IDL overloading is somewhat costly and not loved much by the JS community. Also, please float this by [email protected] at some point.
 
Feedback Jonas: Yeah, something like drawImagePositionBatch/drawImageDestRectBatch/drawImageSrcAndDestRectBatch/etc will make feature detection of future expansions easier. See [http://robert.ocallahan.org/2012/05/canvas-getcontext-mistake.html]
 
==== Exceptions ====
 
:An INDEX_SIZE_ERR DOM exception is thrown if the size of drawParameters is not a multiple of the number of numeric parameters corresponding to 'parameterFormat'
:An INDEX_SIZE_ERR DOM exception is thrown if 'image' is a sequence and the size of drawParameters is not equal to the number of elements in 'image' multiplied by the number of parameters corresponding to 'parameterFormat'.
:All the same exceptions that apply for calls to drawImage also apply to drawImageBatch. If any individual draw results in an exception being thrown, the entire call to drawImageBatch must abort without drawing anything to the canvas.


==== Limitations ====  
==== Limitations ====  
:Use cases where per-sprite transforms are *required* are not covered. For example, if sprites are individually rotated. If appropriate, this could be resolved by adding more variants to the proposal (a matrix argument?)
 
:The use of Float32Array is not very programmer friendly. This compromise on usability is justified by performance considerations, and is necessary to achieve near-native or WebGL-like performance.


==== Implementation ====  
==== Implementation ====  

Latest revision as of 19:53, 4 August 2014

Batching drawImage calls to achieve near-native performance for sprite/image based animations and games.

Use Case Description

Many web applications and games use 2D Canvases and the drawImage method to bring sprites to screen. Often, a large number of calls to drawImage occur for each frame presented to screen.

Current Limitations

When calling drawImage hundreds or thousands of times per animation frame, API bindings overhead and internal bookkeeping costs associated with individual draw calls become a significant performance bottleneck, which prevents web applications from achieving near-native performance levels.

Current Workaround

With WebGL, batching sprite draws is possible thanks to vertex buffers. WebGL is not as broadly supported as 2D canvas, and is a considerably more complex solution to a problem that could otherwise be solved with a 2D canvas.

Benefits

Rendering Performance.

Requests for this Feature

  • [1]

    On Nexus 7 device, it can reach 60fps for 1000 sprites drawing at the same time [referring to a native canvas implementation]. However, the fps is very poor (about 9fps) if the demo is running in Chromium 36.

Proposed Solution

Batch versions of drawImage

New batch variants of drawImage that accept array arguments.

Processing Model

We would have new variants of the existing drawImage method where the numerical arguments are are packed into a Float32Array. The image argument may or may not be an array. If it is not an array, the same source image is used for for each draw. When rendering from a single sprite sheet, it would be preferable to not specify the image argument as an array in order to minimize bindings overhead and redundant parameter validation.

Suggested IDL


enum CanvasDrawImageParameterFormat { position, destination-rectangle, source-and-destination-rectangles, source-rectangle-and-transform};

interface CanvasRenderingContext2D {
    ...
    void drawImageBatch(sequence<CanvasImageSource> image, CanvasDrawImageParameterFormat parameterFormat, Float32Array drawParameters);
    void drawImageBatch(CanvasImageSource image, ParameterFormat parameterFormat, Float32Array drawParameters);
}
The drawParameters argument is to be interpreted as a table in row-major order. Where each row represents a single draw and each column represents an individual parameter. The mapping of the columns to draw parameters depends on the parameterFormat argument.
  • position: dx, dy
  • destination-rectangle: dx, dy, dw, dh
  • source-and-destination-rectangles: sx, sy, sw, sh, dx, dy, dw, dh
  • source-rectangle-and-transform: sx, sy, sw, sh, a, b, c, d, e, f
The parameters sx, sy, sw, sh, dx, dy, dw, and dh have the same meaning as with drawImage()
The parameters a, b, c, d, e, f have the same meaning as with transform()
With 'source-rectangle-and-transform' the destination rectangle is a unit square defined by the vertices (0, 0), (1, 0), (1, 1), and (0, 1). The destination rectangle is transformed by the transform defined by a, b, c, d, e and f, and the by the canvas's current transform.

Feedback Anne: Perhaps rather than overloading we should introduce new methods? IDL overloading is somewhat costly and not loved much by the JS community. Also, please float this by [email protected] at some point.

Feedback Jonas: Yeah, something like drawImagePositionBatch/drawImageDestRectBatch/drawImageSrcAndDestRectBatch/etc will make feature detection of future expansions easier. See [2]

Exceptions

An INDEX_SIZE_ERR DOM exception is thrown if the size of drawParameters is not a multiple of the number of numeric parameters corresponding to 'parameterFormat'
An INDEX_SIZE_ERR DOM exception is thrown if 'image' is a sequence and the size of drawParameters is not equal to the number of elements in 'image' multiplied by the number of parameters corresponding to 'parameterFormat'.
All the same exceptions that apply for calls to drawImage also apply to drawImageBatch. If any individual draw results in an exception being thrown, the entire call to drawImageBatch must abort without drawing anything to the canvas.

Limitations

The use of Float32Array is not very programmer friendly. This compromise on usability is justified by performance considerations, and is necessary to achieve near-native or WebGL-like performance.

Implementation

  • The most naive implementation, which would consist in expanding the batch drawImage call into multiple internal drawImage calls would already increase performance by reducing API bindings overhead.
  • The use of typed arrays will dramatically reduce the argument type checking burden in the bindings.
  • More advanced implementations would carry the batching down to a lower level in the graphics stack. For example, a GPU-accelerated implementation of 2D canvas could use OpenGL vertex buffer objects, or the DirectX sprite interface.
  • Some existing implementations of drawImage already detect batching opportunities and will batch consecutive drawImage calls at a lower lever of the graphics stack (for example skia's drawBitmap used in Blink). This auto-detection does improve rasterization performance, but it does not eliminate the bindings overhead and it involves additional overhead to determine whether consecutive calls to drawImage can be grouped together to form a batch. With explicitly batched calls to drawImage, that overhead can be eliminated because the individuals sprite draws would be known to use identical rendering context state, and the calls down to the graphics platform implementation layer would only occur once for the entire batch.

Adoption

Game/app devs looking for high frame rates for sprite blitting are likely to adopt enthusiastically.