Canvas Batch drawImage - Revision history

Sicking: /* Suggested IDL */

2014-08-04T19:53:08Z

Suggested IDL

← Older revision		Revision as of 19:53, 4 August 2014
Line 50:		Line 50:

	Feedback Anne: Perhaps rather than overloading we should introduce new methods? IDL overloading is somewhat costly and not loved much by the JS community. Also, please float this by public-script-coord@w3.org at some point.		Feedback Anne: Perhaps rather than overloading we should introduce new methods? IDL overloading is somewhat costly and not loved much by the JS community. Also, please float this by public-script-coord@w3.org at some point.

			Feedback Jonas: Yeah, something like drawImagePositionBatch/drawImageDestRectBatch/drawImageSrcAndDestRectBatch/etc will make feature detection of future expansions easier. See [http://robert.ocallahan.org/2012/05/canvas-getcontext-mistake.html]

	==== Exceptions ====		==== Exceptions ====

Junov at 17:19, 4 August 2014

2014-08-04T17:19:42Z

← Older revision		Revision as of 17:19, 4 August 2014
Line 47:		Line 47:
	:The parameters sx, sy, sw, sh, dx, dy, dw, and dh have the same meaning as with drawImage()		:The parameters sx, sy, sw, sh, dx, dy, dw, and dh have the same meaning as with drawImage()
	:The parameters a, b, c, d, e, f have the same meaning as with transform()		:The parameters a, b, c, d, e, f have the same meaning as with transform()
	:With 'source-rectangle-and-transform' the destination rectangle is defined by the vertices (0, 0), (sw, 0), (sw, sh), and (0, sh). The destination rectangle is transformed by the transform defined by a, b, c, d, e and f, and the by the canvas's current transform.		:With 'source-rectangle-and-transform' the destination rectangle is a unit square defined by the vertices (0, 0), (1, 0), (1, 1), and (0, 1). The destination rectangle is transformed by the transform defined by a, b, c, d, e and f, and the by the canvas's current transform.

	Feedback Anne: Perhaps rather than overloading we should introduce new methods? IDL overloading is somewhat costly and not loved much by the JS community. Also, please float this by public-script-coord@w3.org at some point.		Feedback Anne: Perhaps rather than overloading we should introduce new methods? IDL overloading is somewhat costly and not loved much by the JS community. Also, please float this by public-script-coord@w3.org at some point.

Junov at 17:14, 4 August 2014

2014-08-04T17:14:46Z

← Older revision		Revision as of 17:14, 4 August 2014
Line 55:		Line 55:
	:An INDEX_SIZE_ERR DOM exception is thrown if the size of drawParameters is not a multiple of the number of numeric parameters corresponding to 'parameterFormat'		:An INDEX_SIZE_ERR DOM exception is thrown if the size of drawParameters is not a multiple of the number of numeric parameters corresponding to 'parameterFormat'
	:An INDEX_SIZE_ERR DOM exception is thrown if 'image' is a sequence and the size of drawParameters is not equal to the number of elements in 'image' multiplied by the number of parameters corresponding to 'parameterFormat'.		:An INDEX_SIZE_ERR DOM exception is thrown if 'image' is a sequence and the size of drawParameters is not equal to the number of elements in 'image' multiplied by the number of parameters corresponding to 'parameterFormat'.
	All the same exceptions that apply for calls to drawImage also apply to drawImageBatch. If any individual draw results in an exception being thrown, the entire call to drawImageBatch must abort without drawing anything to the canvas.		:All the same exceptions that apply for calls to drawImage also apply to drawImageBatch. If any individual draw results in an exception being thrown, the entire call to drawImageBatch must abort without drawing anything to the canvas.

	==== Limitations ====		==== Limitations ====

Junov: /* Proposed Solution */

2014-08-04T17:14:00Z

Proposed Solution

← Older revision		Revision as of 17:14, 4 August 2014
Line 21:		Line 21:

	=== Batch versions of drawImage ===		=== Batch versions of drawImage ===
	:~~Additional overloads~~ of drawImage that accept array arguments.		:New batch variants of drawImage that accept array arguments.

	==== Processing Model ====		==== Processing Model ====
	:We would have new variants of the existing drawImage ~~overloads~~ where the numerical arguments are are packed into a Float32Array. The image argument may or may not be an array. if it is not an array, the same source image is used for for each draw. When rendering from a single sprite sheet, it would be preferable to not specify the image argument as an array in order to minimize bindings overhead.		:We would have new variants of the existing drawImage method where the numerical arguments are are packed into a Float32Array. The image argument may or may not be an array. If it is not an array, the same source image is used for for each draw. When rendering from a single sprite sheet, it would be preferable to not specify the image argument as an array in order to minimize bindings overhead and redundant parameter validation.

	~~:An INDEX_SIZE_ERR DOM the Float32Array size is not a multiple of the number of numeric~~ parameter ~~or if it does not match the size of the image argument, if image is an array~~.

	==== Suggested IDL ====		==== Suggested IDL ====
Line 52:		Line 50:

	Feedback Anne: Perhaps rather than overloading we should introduce new methods? IDL overloading is somewhat costly and not loved much by the JS community. Also, please float this by public-script-coord@w3.org at some point.		Feedback Anne: Perhaps rather than overloading we should introduce new methods? IDL overloading is somewhat costly and not loved much by the JS community. Also, please float this by public-script-coord@w3.org at some point.

			==== Exceptions ====

			:An INDEX_SIZE_ERR DOM exception is thrown if the size of drawParameters is not a multiple of the number of numeric parameters corresponding to 'parameterFormat'
			:An INDEX_SIZE_ERR DOM exception is thrown if 'image' is a sequence and the size of drawParameters is not equal to the number of elements in 'image' multiplied by the number of parameters corresponding to 'parameterFormat'.
			All the same exceptions that apply for calls to drawImage also apply to drawImageBatch. If any individual draw results in an exception being thrown, the entire call to drawImageBatch must abort without drawing anything to the canvas.

	==== Limitations ====		==== Limitations ====

	:The use of Float32Array is not very programmer friendly. This compromise on usability is justified by performance considerations, and is necessary to achieve near-native or WebGL-like performance.		:The use of Float32Array is not very programmer friendly. This compromise on usability is justified by performance considerations, and is necessary to achieve near-native or WebGL-like performance.

Junov: /* Use Case Description */

2014-08-04T17:05:09Z

Use Case Description

← Older revision		Revision as of 17:05, 4 August 2014
Line 5:		Line 5:

	=== Current Limitations ===		=== Current Limitations ===
	When calling drawImage hundreds or thousands of times per animation frame, API bindings overhead and internal bookkeeping costs associated with individual draw calls become a significant performance bottleneck, which prevents web ~~application~~ from achieving near-native performance levels.		When calling drawImage hundreds or thousands of times per animation frame, API bindings overhead and internal bookkeeping costs associated with individual draw calls become a significant performance bottleneck, which prevents web applications from achieving near-native performance levels.

	=== Current Workaround ===		=== Current Workaround ===

Junov: /* Batch versions of drawImage */

2014-08-04T17:04:10Z

Batch versions of drawImage

← Older revision		Revision as of 17:04, 4 August 2014
Line 24:		Line 24:

	==== Processing Model ====		==== Processing Model ====
	:We would have new variants of the existing drawImage overloads where ~~all~~ the arguments are ~~arrays (~~Float32Array ~~for numeric arguments), and other variants where all the arguments except for the image are arrays~~. The ~~case where the~~ image is not an array ~~would be equivalent to having~~ an array ~~where each element is~~ the same image ~~(convenient~~ for sprite ~~sheets~~, ~~and more efficient than using~~ an array)		:We would have new variants of the existing drawImage overloads where the numerical arguments are are packed into a Float32Array. The image argument may or may not be an array. if it is not an array, the same source image is used for for each draw. When rendering from a single sprite sheet, it would be preferable to not specify the image argument as an array in order to minimize bindings overhead.

	:An INDEX_SIZE_ERR DOM ~~exception would be thrown~~ if not ~~all~~ the array ~~arguments have the same size~~.		:An INDEX_SIZE_ERR DOM the Float32Array size is not a multiple of the number of numeric parameter or if it does not match the size of the image argument, if image is an array.

	==== Suggested IDL ====		==== Suggested IDL ====

	<pre>		<pre>
	interface ~~Canvas2DRenderingContext~~ {
			enum CanvasDrawImageParameterFormat { position, destination-rectangle, source-and-destination-rectangles, source-rectangle-and-transform};

			interface CanvasRenderingContext2D {
	...		...
	~~// existing variants~~		void drawImageBatch(sequence<CanvasImageSource> image, CanvasDrawImageParameterFormat parameterFormat, Float32Array drawParameters);
	~~void drawImage(CanvasImageSource image, unrestricted double dx, unrestricted double dy);~~		void drawImageBatch(CanvasImageSource image, ParameterFormat parameterFormat, Float32Array drawParameters);
	void ~~drawImage(CanvasImageSource image, unrestricted double dx, unrestricted double dy, unrestricted double dw, unrestricted double dh);~~
	void drawImage(CanvasImageSource image, unrestricted double sx, unrestricted double sy, unrestricted double sw, unrestricted double sh, unrestricted double dx, unrestricted double dy, unrestricted double dw, unrestricted double dh);
	~~// batch variants~~
	~~void drawImage(sequence<CanvasImageSource> image, Float32Array dx, Float32Array dy);~~
	~~void drawImage~~(sequence<CanvasImageSource> image, ~~Float32Array dx~~, Float32Array ~~dy, Float32Array dw, Float32Array dh~~);
	void ~~drawImage~~(~~sequence<~~CanvasImageSource> image, ~~Float32Array sx, Float32Array sy, Float32Array sw, Float32Array sh, Float32Array dx, Float32Array dy, Float32Array dw, Float32Array dh);~~
	~~// single image batch variants~~
	~~void drawImage(CanvasImageSource image, Float32Array dx, Float32Array dy);~~
	~~void drawImage(CanvasImageSource image, Float32Array dx, Float32Array dy, Float32Array dw, Float32Array dh);~~
	~~void drawImage(CanvasImageSource image, Float32Array sx, Float32Array sy, Float32Array sw, Float32Array sh, Float32Array dx, Float32Array dy, Float32Array dw~~, Float32Array dh);
	}		}
	</pre>		</pre>

			:The drawParameters argument is to be interpreted as a table in row-major order. Where each row represents a single draw and each column represents an individual parameter. The mapping of the columns to draw parameters depends on the parameterFormat argument.
			:* position: dx, dy
			:* destination-rectangle: dx, dy, dw, dh
			:* source-and-destination-rectangles: sx, sy, sw, sh, dx, dy, dw, dh
			:* source-rectangle-and-transform: sx, sy, sw, sh, a, b, c, d, e, f

			:The parameters sx, sy, sw, sh, dx, dy, dw, and dh have the same meaning as with drawImage()
			:The parameters a, b, c, d, e, f have the same meaning as with transform()
			:With 'source-rectangle-and-transform' the destination rectangle is defined by the vertices (0, 0), (sw, 0), (sw, sh), and (0, sh). The destination rectangle is transformed by the transform defined by a, b, c, d, e and f, and the by the canvas's current transform.

	Feedback Anne: Perhaps rather than overloading we should introduce new methods? IDL overloading is somewhat costly and not loved much by the JS community. Also, please float this by public-script-coord@w3.org at some point.		Feedback Anne: Perhaps rather than overloading we should introduce new methods? IDL overloading is somewhat costly and not loved much by the JS community. Also, please float this by public-script-coord@w3.org at some point.

	==== Limitations ====		==== Limitations ====
	:~~Use cases where per-sprite transforms are required are~~ not ~~covered~~. ~~For example~~, ~~if sprites are individually rotated~~. ~~If appropriate, this could be resolved by adding more variants to the proposal (a matrix argument?)~~		:The use of Float32Array is not very programmer friendly. This compromise on usability is justified by performance considerations, and is necessary to achieve near-native or WebGL-like performance.

	==== Implementation ====		==== Implementation ====

Annevk: /* Suggested IDL */ add feedback, [] -> sequence<>

2014-08-04T12:39:45Z

Suggested IDL: add feedback, [] -> sequence<>

← Older revision		Revision as of 12:39, 4 August 2014
Line 38:		Line 38:
	void drawImage(CanvasImageSource image, unrestricted double sx, unrestricted double sy, unrestricted double sw, unrestricted double sh, unrestricted double dx, unrestricted double dy, unrestricted double dw, unrestricted double dh);		void drawImage(CanvasImageSource image, unrestricted double sx, unrestricted double sy, unrestricted double sw, unrestricted double sh, unrestricted double dx, unrestricted double dy, unrestricted double dw, unrestricted double dh);
	// batch variants		// batch variants
	void drawImage(CanvasImageSource image[], Float32Array dx, Float32Array dy);		void drawImage(sequence<CanvasImageSource> image, Float32Array dx, Float32Array dy);
	void drawImage(CanvasImageSource image[], Float32Array dx, Float32Array dy, Float32Array dw, Float32Array dh);		void drawImage(sequence<CanvasImageSource> image, Float32Array dx, Float32Array dy, Float32Array dw, Float32Array dh);
	void drawImage(CanvasImageSource image[], Float32Array sx, Float32Array sy, Float32Array sw, Float32Array sh, Float32Array dx, Float32Array dy, Float32Array dw, Float32Array dh);		void drawImage(sequence<CanvasImageSource> image, Float32Array sx, Float32Array sy, Float32Array sw, Float32Array sh, Float32Array dx, Float32Array dy, Float32Array dw, Float32Array dh);
	// single image batch variants		// single image batch variants
	void drawImage(CanvasImageSource image, Float32Array dx, Float32Array dy);		void drawImage(CanvasImageSource image, Float32Array dx, Float32Array dy);
Line 47:		Line 47:
	}		}
	</pre>		</pre>

			Feedback Anne: Perhaps rather than overloading we should introduce new methods? IDL overloading is somewhat costly and not loved much by the JS community. Also, please float this by public-script-coord@w3.org at some point.

	==== Limitations ====		==== Limitations ====
	:Use cases where per-sprite transforms are required are not covered. For example, if sprites are individually rotated. If appropriate, this could be resolved by adding more variants to the proposal (a matrix argument?)		:Use cases where per-sprite transforms are required are not covered. For example, if sprites are individually rotated. If appropriate, this could be resolved by adding more variants to the proposal (a matrix argument?)

Junov at 21:25, 30 July 2014

2014-07-30T21:25:14Z

@@ Line 28: / Line 28: @@
 :An INDEX_SIZE_ERR DOM exception would be thrown if not all the array arguments have the same size.
 ==== Limitations ====
 :Use cases where per-sprite transforms are *required* are not covered. For example, if sprites are individually rotated. If appropriate, this could be resolved by adding more variants to the proposal (a matrix argument?)

Junov: Created page with ":Batching drawImage calls to achieve near-native performance for sprite/image based animations and games. == Use Case Description == Many web applications and games use 2D Ca..."

2014-07-30T20:58:02Z

Created page with ":Batching drawImage calls to achieve near-native performance for sprite/image based animations and games. == Use Case Description == Many web applications and games use 2D Ca..."

New page

:Batching drawImage calls to achieve near-native performance for sprite/image based animations and games.

== Use Case Description ==
Many web applications and games use 2D Canvases and the drawImage method to bring sprites to screen. Often, a large number of calls to drawImage occur for each frame presented to screen.

=== Current Limitations ===
When calling drawImage hundreds or thousands of times per animation frame, API bindings overhead and internal bookkeeping costs associated with individual draw calls become a significant performance bottleneck, which prevents web application from achieving near-native performance levels.

=== Current Workaround ===
With WebGL, batching sprite draws is possible thanks to vertex buffers. WebGL is not as broadly supported as 2D canvas, and is a considerably more complex solution to a problem that could otherwise be solved with a 2D canvas.

=== Benefits ===

Rendering Performance.

=== Requests for this Feature ===

* <cite>[https://groups.google.com/a/chromium.org/forum/#!topic/graphics-dev/RzHob5VS8Dg]</cite> <blockquote><p>On Nexus 7 device, it can reach 60fps for 1000 sprites drawing at the same time [referring to a native canvas implementation]. However, the fps is very poor (about 9fps) if the demo is running in Chromium 36.</p></blockquote>

== Proposed Solution ==

=== Batch versions of drawImage ===
:Additional overloads of drawImage that accept array arguments.

==== Processing Model ====
:We would have new variants of the existing drawImage overloads where all the arguments are arrays (Float32Array for numeric arguments), and other variants where all the arguments except for the image are arrays. The case where the image is not an array would be equivalent to having an array where each element is the same image (convenient for sprite sheets, and more efficient than using an array)

:An INDEX_SIZE_ERR DOM exception would be thrown if not all the array arguments have the same size.

==== Limitations ====
:Use cases where per-sprite transforms are *required* are not covered. For example, if sprites are individually rotated. If appropriate, this could be resolved by adding more variants to the proposal (a matrix argument?)

==== Implementation ====
* The most naive implementation, which would consist in expanding the batch drawImage call into multiple internal drawImage calls would already increase performance by reducing API bindings overhead.
* The use of typed arrays will dramatically reduce the argument type checking burden in the bindings.
* More advanced implementations would carry the batching down to a lower level in the graphics stack. For example, a GPU-accelerated implementation of 2D canvas could use OpenGL vertex buffer objects, or the DirectX sprite interface.
* Some existing implementations of drawImage already detect batching opportunities and will batch consecutive drawImage calls at a lower lever of the graphics stack (for example skia's drawBitmap used in Blink). This auto-detection does improve rasterization performance, but it does not eliminate the bindings overhead and it involves additional overhead to determine whether consecutive calls to drawImage can be grouped together to form a batch. With explicitly batched calls to drawImage, that overhead can be eliminated because the individuals sprite draws would be known to use identical rendering context state, and the calls down to the graphics platform implementation layer would only occur once for the entire batch.

==== Adoption ====
:Game/app devs looking for high frame rates for sprite blitting are likely to adopt enthusiastically.

[[Category:Proposals]]