WHATWG Wiki - User contributions [en]

StringEncoding

2012-10-19T16:21:04Z

Jsbell: Obsoleted

Proposed Text Encoding Web API for Typed Arrays

= This Document Is Obsolete =

See [http://encoding.spec.whatwg.org/#api Encoding Standard - API] instead.

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Should the <var>encoding</var> attribute return the <var>Name</var> of the encoding or the name that was passed in?

== Notes to Implementers ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.

<dt>Should legacy encodings be supported?
<dd>Resolution: Consensus on the [http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-August/036825.html WHATWG mailing list] - support legacy encodings for decode, and only 'utf-8', 'utf-16' and 'utf-16be' for encode.

<dt>What do to on encoding errors? If non-UTF encodings are supported then we may want to allow substitution (e.g. ASCII '?') or a script callback (for arbitrary escaping).
<dd>Resolution: not for "v1" (see above)

<dt>How are byte order marks handled?
<dd>BOM is respected if and only if the requested coding is a case-insensitive match for "<code>utf-16</code>"

</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
Uint8Array encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://encoding.spec.whatwg.org/ Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, if the <var>Name</var> of the returned encoding is not one of "<code>utf-8</code>", "<code>utf-16</code>", or "<code>utf-16be</code>" throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://encoding.spec.whatwg.org/ Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

:''NOTE: Because only UTF encodings are supported, and because of the use of the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] and thus Unicode code points, no input can cause the encoding process to emit an encoder error.''

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, a <var>streaming</var> flag which is initially unset, an <var>offset</var> pointer which is initially <code>0</code>, and a <var>useBOM</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# If <var>label</var> is a case-insensitive match for "<code>utf-16</code>" then set the internal <var>useBOM</var> flag.
# Run the '''steps to get an encoding''' from [http://encoding.spec.whatwg.org/ Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://encoding.spec.whatwg.org/ Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If <var>view</var> is not specified, let <var>view</var> be a <code>Uint8Array</code> of length <code>0</code>.
# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var> and set <var>offset</var> to <code>0</code>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set. Otherwise the internal <var>streaming</var> flag is cleared.
# Run or resume the decoder algorithm of the ''decoder object's'' encoder, with the following additions:
#* The input to the algorithm is a <var>byte stream</var>.
#** If <var>offset</var> is greater than <code>0</code>, the bytes in <var>byte stream</var> for positions less than <var>offset</var> are provided by buffer(s) passed in to previous calls to <code>decode()</code>.
#** The bytes in <var>byte stream</var> for positions <var>offset</var> through <code><var>offset</var> + <var>view.byteLength</var> - 1</code> are provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>.
#** When accessing the byte in <var>byte stream</var> at position <code><var>offset</var> + <var>view.byteLength</var></code>:
#*** If the internal <var>streaming</var> flag is not set, then yield the '''EOF byte'''.
#*** Otherwise, set <var>offset</var> to <code><var>offset</var> + <var>view.byteLength</var></code> and suspend the steps of the decoder algorithm until a subsequent call to <code>decode()</code>
#* If encoding is one of "<code>utf-8</code>", "<code>utf-16</code>" or "<code>utf-16be</code>" and <var>offset</var> is <code>0</code>, then prior to running the steps of the decoder algorithm:
#** If the internal <var>useBOM</var> flag is set, then:
#*** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
#*** If the next two bytes of the stream are '''0xFF 0xFE''' then set <var>offset</var> to <code>2</code>, and clear the <var>utf-16be</var> flag of the decoder state.
#*** If the next two bytes of the stream are '''0xFE 0xFF''' then set <var>offset</var> to <code>2</code>, and set the <var>utf-16be</var> flag of the decoder state.
#** If encoding is "<code>utf-8</code>", then:
#*** If less than three bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
#*** If the next three bytes of the stream are '''0xEF 0xBB 0xBF''' then set <var>offset</var> to <code>3</code>.
#** If encoding is "<code>utf-16</code>", then:
#*** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
#*** If the next two bytes of the stream are '''0xFF 0xFE''' then set <var>offset</var> to <code>2</code>.
#** If encoding is "<code>utf-16be</code>", then:
#*** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
#*** If the next two bytes of the stream are '''0xFE 0xFF''' then set <var>offset</var> to <code>2</code>.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a <var>sequence of emitted code points</var>.
# Return an IDL <code>DOMString</code> value that represents the sequence of code units resulting from encoding the <var>sequence of emitted code points</var> as UTF-16.

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://encoding.spec.whatwg.org/ Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://encoding.spec.whatwg.org/ Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://encoding.spec.whatwg.org/ Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://encoding.spec.whatwg.org/ Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://encoding.spec.whatwg.org/

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren
* Cameron McCormack

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-09-21T20:27:32Z

Jsbell: Encoding standard URL update

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Should the <var>encoding</var> attribute return the <var>Name</var> of the encoding or the name that was passed in?

== Notes to Implementers ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.

<dt>Should legacy encodings be supported?
<dd>Resolution: Consensus on the [http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-August/036825.html WHATWG mailing list] - support legacy encodings for decode, and only 'utf-8', 'utf-16' and 'utf-16be' for encode.

<dt>What do to on encoding errors? If non-UTF encodings are supported then we may want to allow substitution (e.g. ASCII '?') or a script callback (for arbitrary escaping).
<dd>Resolution: not for "v1" (see above)

<dt>How are byte order marks handled?
<dd>BOM is respected if and only if the requested coding is a case-insensitive match for "<code>utf-16</code>"

</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
Uint8Array encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://encoding.spec.whatwg.org/ Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, if the <var>Name</var> of the returned encoding is not one of "<code>utf-8</code>", "<code>utf-16</code>", or "<code>utf-16be</code>" throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://encoding.spec.whatwg.org/ Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

:''NOTE: Because only UTF encodings are supported, and because of the use of the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] and thus Unicode code points, no input can cause the encoding process to emit an encoder error.''

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, a <var>streaming</var> flag which is initially unset, an <var>offset</var> pointer which is initially <code>0</code>, and a <var>useBOM</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# If <var>label</var> is a case-insensitive match for "<code>utf-16</code>" then set the internal <var>useBOM</var> flag.
# Run the '''steps to get an encoding''' from [http://encoding.spec.whatwg.org/ Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://encoding.spec.whatwg.org/ Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If <var>view</var> is not specified, let <var>view</var> be a <code>Uint8Array</code> of length <code>0</code>.
# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var> and set <var>offset</var> to <code>0</code>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set. Otherwise the internal <var>streaming</var> flag is cleared.
# Run or resume the decoder algorithm of the ''decoder object's'' encoder, with the following additions:
#* The input to the algorithm is a <var>byte stream</var>.
#** If <var>offset</var> is greater than <code>0</code>, the bytes in <var>byte stream</var> for positions less than <var>offset</var> are provided by buffer(s) passed in to previous calls to <code>decode()</code>.
#** The bytes in <var>byte stream</var> for positions <var>offset</var> through <code><var>offset</var> + <var>view.byteLength</var> - 1</code> are provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>.
#** When accessing the byte in <var>byte stream</var> at position <code><var>offset</var> + <var>view.byteLength</var></code>:
#*** If the internal <var>streaming</var> flag is not set, then yield the '''EOF byte'''.
#*** Otherwise, set <var>offset</var> to <code><var>offset</var> + <var>view.byteLength</var></code> and suspend the steps of the decoder algorithm until a subsequent call to <code>decode()</code>
#* If encoding is one of "<code>utf-8</code>", "<code>utf-16</code>" or "<code>utf-16be</code>" and <var>offset</var> is <code>0</code>, then prior to running the steps of the decoder algorithm:
#** If the internal <var>useBOM</var> flag is set, then:
#*** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
#*** If the next two bytes of the stream are '''0xFF 0xFE''' then set <var>offset</var> to <code>2</code>, and clear the <var>utf-16be</var> flag of the decoder state.
#*** If the next two bytes of the stream are '''0xFE 0xFF''' then set <var>offset</var> to <code>2</code>, and set the <var>utf-16be</var> flag of the decoder state.
#** If encoding is "<code>utf-8</code>", then:
#*** If less than three bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
#*** If the next three bytes of the stream are '''0xEF 0xBB 0xBF''' then set <var>offset</var> to <code>3</code>.
#** If encoding is "<code>utf-16</code>", then:
#*** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
#*** If the next two bytes of the stream are '''0xFF 0xFE''' then set <var>offset</var> to <code>2</code>.
#** If encoding is "<code>utf-16be</code>", then:
#*** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
#*** If the next two bytes of the stream are '''0xFE 0xFF''' then set <var>offset</var> to <code>2</code>.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a <var>sequence of emitted code points</var>.
# Return an IDL <code>DOMString</code> value that represents the sequence of code units resulting from encoding the <var>sequence of emitted code points</var> as UTF-16.

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://encoding.spec.whatwg.org/ Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://encoding.spec.whatwg.org/ Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://encoding.spec.whatwg.org/ Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://encoding.spec.whatwg.org/ Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://encoding.spec.whatwg.org/

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren
* Cameron McCormack

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-09-17T21:06:11Z

Jsbell: HTML typo

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Should the <var>encoding</var> attribute return the <var>Name</var> of the encoding or the name that was passed in?

== Notes to Implementers ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.

<dt>Should legacy encodings be supported?
<dd>Resolution: Consensus on the [http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-August/036825.html WHATWG mailing list] - support legacy encodings for decode, and only 'utf-8', 'utf-16' and 'utf-16be' for encode.

<dt>What do to on encoding errors? If non-UTF encodings are supported then we may want to allow substitution (e.g. ASCII '?') or a script callback (for arbitrary escaping).
<dd>Resolution: not for "v1" (see above)

<dt>How are byte order marks handled?
<dd>BOM is respected if and only if the requested coding is a case-insensitive match for "<code>utf-16</code>"

</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
Uint8Array encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, if the <var>Name</var> of the returned encoding is not one of "<code>utf-8</code>", "<code>utf-16</code>", or "<code>utf-16be</code>" throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

:''NOTE: Because only UTF encodings are supported, and because of the use of the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] and thus Unicode code points, no input can cause the encoding process to emit an encoder error.''

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, a <var>streaming</var> flag which is initially unset, an <var>offset</var> pointer which is initially <code>0</code>, and a <var>useBOM</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# If <var>label</var> is a case-insensitive match for "<code>utf-16</code>" then set the internal <var>useBOM</var> flag.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If <var>view</var> is not specified, let <var>view</var> be a <code>Uint8Array</code> of length <code>0</code>.
# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var> and set <var>offset</var> to <code>0</code>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set. Otherwise the internal <var>streaming</var> flag is cleared.
# Run or resume the decoder algorithm of the ''decoder object's'' encoder, with the following additions:
#* The input to the algorithm is a <var>byte stream</var>.
#** If <var>offset</var> is greater than <code>0</code>, the bytes in <var>byte stream</var> for positions less than <var>offset</var> are provided by buffer(s) passed in to previous calls to <code>decode()</code>.
#** The bytes in <var>byte stream</var> for positions <var>offset</var> through <code><var>offset</var> + <var>view.byteLength</var> - 1</code> are provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>.
#** When accessing the byte in <var>byte stream</var> at position <code><var>offset</var> + <var>view.byteLength</var></code>:
#*** If the internal <var>streaming</var> flag is not set, then yield the '''EOF byte'''.
#*** Otherwise, set <var>offset</var> to <code><var>offset</var> + <var>view.byteLength</var></code> and suspend the steps of the decoder algorithm until a subsequent call to <code>decode()</code>
#* If encoding is one of "<code>utf-8</code>", "<code>utf-16</code>" or "<code>utf-16be</code>" and <var>offset</var> is <code>0</code>, then prior to running the steps of the decoder algorithm:
#** If the internal <var>useBOM</var> flag is set, then:
#*** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
#*** If the next two bytes of the stream are '''0xFF 0xFE''' then set <var>offset</var> to <code>2</code>, and clear the <var>utf-16be</var> flag of the decoder state.
#*** If the next two bytes of the stream are '''0xFE 0xFF''' then set <var>offset</var> to <code>2</code>, and set the <var>utf-16be</var> flag of the decoder state.
#** If encoding is "<code>utf-8</code>", then:
#*** If less than three bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
#*** If the next three bytes of the stream are '''0xEF 0xBB 0xBF''' then set <var>offset</var> to <code>3</code>.
#** If encoding is "<code>utf-16</code>", then:
#*** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
#*** If the next two bytes of the stream are '''0xFF 0xFE''' then set <var>offset</var> to <code>2</code>.
#** If encoding is "<code>utf-16be</code>", then:
#*** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
#*** If the next two bytes of the stream are '''0xFE 0xFF''' then set <var>offset</var> to <code>2</code>.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a <var>sequence of emitted code points</var>.
# Return an IDL <code>DOMString</code> value that represents the sequence of code units resulting from encoding the <var>sequence of emitted code points</var> as UTF-16.

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren
* Cameron McCormack

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-09-17T21:05:24Z

Jsbell: Make BOM handling algorithmic

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Should the <var>encoding</var> attribute return the <var>Name</var> of the encoding or the name that was passed in?

== Notes to Implementers ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.

<dt>Should legacy encodings be supported?
<dd>Resolution: Consensus on the [http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-August/036825.html WHATWG mailing list] - support legacy encodings for decode, and only 'utf-8', 'utf-16' and 'utf-16be' for encode.

<dt>What do to on encoding errors? If non-UTF encodings are supported then we may want to allow substitution (e.g. ASCII '?') or a script callback (for arbitrary escaping).
<dd>Resolution: not for "v1" (see above)

<dt>How are byte order marks handled?
<dd>BOM is respected if and only if the requested coding is a case-insensitive match for "<code>utf-16</code>"

</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
Uint8Array encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, if the <var>Name</var> of the returned encoding is not one of "<code>utf-8</code>", "<code>utf-16</code>", or "<code>utf-16be</code>" throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

:''NOTE: Because only UTF encodings are supported, and because of the use of the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] and thus Unicode code points, no input can cause the encoding process to emit an encoder error.''

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, a <var>streaming</var> flag which is initially unset, an <var>offset</var> pointer which is initially <code>0</code>, and a <var>useBOM</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# If <var>label</var> is a case-insensitive match for "<code>utf-16</code>" then set the internal <var>useBOM</var> flag.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If <var>view</var> is not specified, let <var>view</var> be a <code>Uint8Array</code> of length <code>0</code>.
# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var> and set <var>offset</var> to <code>0</code>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set. Otherwise the internal <var>streaming</var> flag is cleared.
# Run or resume the decoder algorithm of the ''decoder object's'' encoder, with the following additions:
#* The input to the algorithm is a <var>byte stream</var>.
#** If <var>offset</var> is greater than <code>0</code>, the bytes in <var>byte stream</var> for positions less than <var>offset</var> are provided by buffer(s) passed in to previous calls to <code>decode()</code>.
#** The bytes in <var>byte stream</var> for positions <var>offset</var> through <code><var>offset</var> + <var>view.byteLength</var> - 1</code> are provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>.
#** When accessing the byte in <var>byte stream</var> at position code><var>offset</var> + <var>view.byteLength</var></code>:
#*** If the internal <var>streaming</var> flag is not set, then yield the '''EOF byte'''.
#*** Otherwise, set <var>offset</var> to <code><var>offset</var> + <var>view.byteLength</var></code> and suspend the steps of the decoder algorithm until a subsequent call to <code>decode()</code>
#* If encoding is one of "<code>utf-8</code>", "<code>utf-16</code>" or "<code>utf-16be</code>" and <var>offset</var> is <code>0</code>, then prior to running the steps of the decoder algorithm:
#** If the internal <var>useBOM</var> flag is set, then:
#*** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
#*** If the next two bytes of the stream are '''0xFF 0xFE''' then set <var>offset</var> to <code>2</code>, and clear the <var>utf-16be</var> flag of the decoder state.
#*** If the next two bytes of the stream are '''0xFE 0xFF''' then set <var>offset</var> to <code>2</code>, and set the <var>utf-16be</var> flag of the decoder state.
#** If encoding is "<code>utf-8</code>", then:
#*** If less than three bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
#*** If the next three bytes of the stream are '''0xEF 0xBB 0xBF''' then set <var>offset</var> to <code>3</code>.
#** If encoding is "<code>utf-16</code>", then:
#*** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
#*** If the next two bytes of the stream are '''0xFF 0xFE''' then set <var>offset</var> to <code>2</code>.
#** If encoding is "<code>utf-16be</code>", then:
#*** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
#*** If the next two bytes of the stream are '''0xFE 0xFF''' then set <var>offset</var> to <code>2</code>.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a <var>sequence of emitted code points</var>.
# Return an IDL <code>DOMString</code> value that represents the sequence of code units resulting from encoding the <var>sequence of emitted code points</var> as UTF-16.

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren
* Cameron McCormack

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-09-13T23:09:13Z

Jsbell: /* TextDecoder */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Should the <var>encoding</var> attribute return the <var>Name</var> of the encoding or the name that was passed in?

== Notes to Implementers ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.

<dt>Should legacy encodings be supported?
<dd>Resolution: Consensus on the [http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-August/036825.html WHATWG mailing list] - support legacy encodings for decode, and only 'utf-8', 'utf-16' and 'utf-16be' for encode.

<dt>What do to on encoding errors? If non-UTF encodings are supported then we may want to allow substitution (e.g. ASCII '?') or a script callback (for arbitrary escaping).
<dd>Resolution: not for "v1" (see above)

<dt>How are byte order marks handled?
<dd>BOM is respected if and only if the requested coding is a case-insensitive match for "<code>utf-16</code>"

</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
Uint8Array encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, if the <var>Name</var> of the returned encoding is not one of "<code>utf-8</code>", "<code>utf-16</code>", or "<code>utf-16be</code>" throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

:''NOTE: Because only UTF encodings are supported, and because of the use of the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] and thus Unicode code points, no input can cause the encoding process to emit an encoder error.''

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, a <var>streaming</var> flag which is initially unset, an <var>offset</var> pointer which is initially <code>0</code>, and a <var>useBOM</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# If <var>label</var> is a case-insensitive match for "<code>utf-16</code>" then set the internal <var>useBOM</var> flag.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If <var>view</var> is not specified, let <var>view</var> be a <code>Uint8Array</code> of length <code>0</code>.
# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var> and set <var>offset</var> to <code>0</code>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set. Otherwise the internal <var>streaming</var> flag is cleared.
# Run or resume the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>.
#** If <var>offset</var> is greater than <code>0</code>, the bytes in <var>byte stream</var> for positions less than <var>offset</var> are provided by buffer(s) passed in to previous calls to <code>decode()</code>.
#** The bytes in <var>byte stream</var> for positions <var>offset</var> through <code><var>offset</var> + <var>view.byteLength</var> - 1</code> are provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>.
#** When accessing the byte in <var>byte stream</var> at position code><var>offset</var> + <var>view.byteLength</var></code>:
#*** If the internal <var>streaming</var> flag is not set, then yield the '''EOF byte'''.
#*** Otherwise, set <var>offset</var> to <code><var>offset</var> + <var>view.byteLength</var></code> and suspend the steps of the decoder algorithm until a subsequent call to <code>decode()</code>
#* If the internal <var>useBOM</var> flag is set and <var>offset</var> is <code>0</code> then:
#** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
#** If the next two bytes of the stream are '''0xFF 0xFE''' then set <var>offset</var> to <code>2</code>, and clear the <var>utf-16be</var> flag of the decoder state.
#** If the next two bytes of the stream are '''0xFE 0xFF''' then set <var>offset</var> to <code>2</code>, and set the <var>utf-16be</var> flag of the decoder state.
#* Subsequent to the above <var>useBOM</var> test and prior to executing the decoder algorithm, if encoding is one of "<code>utf-8</code>", "<code>utf-16</code>" or "<code>utf-16be</code>" and <var>offset</var> is <code>0</code>, then if a matching BOM is present then <var>offset</var> should be advanced past it. If not enough bytes are present in the stream for a match and the internal <var>streaming</var> flag is set suspend this algorithm and return the empty string.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a <var>sequence of emitted code points</var>.
# Return an IDL <code>DOMString</code> value that represents the sequence of code units resulting from encoding the <var>sequence of emitted code points</var> as UTF-16.

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren
* Cameron McCormack

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-09-13T23:01:22Z

Jsbell:

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Should the <var>encoding</var> attribute return the <var>Name</var> of the encoding or the name that was passed in?

== Notes to Implementers ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.

<dt>Should legacy encodings be supported?
<dd>Resolution: Consensus on the [http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-August/036825.html WHATWG mailing list] - support legacy encodings for decode, and only 'utf-8', 'utf-16' and 'utf-16be' for encode.

<dt>What do to on encoding errors? If non-UTF encodings are supported then we may want to allow substitution (e.g. ASCII '?') or a script callback (for arbitrary escaping).
<dd>Resolution: not for "v1" (see above)

<dt>How are byte order marks handled?
<dd>BOM is respected if and only if the requested coding is a case-insensitive match for "<code>utf-16</code>"

</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
Uint8Array encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, if the <var>Name</var> of the returned encoding is not one of "<code>utf-8</code>", "<code>utf-16</code>", or "<code>utf-16be</code>" throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

:''NOTE: Because only UTF encodings are supported, and because of the use of the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] and thus Unicode code points, no input can cause the encoding process to emit an encoder error.''

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, a <var>streaming</var> flag which is initially unset, an <var>offset</var> pointer which is initially <code>0</code>, and a <var>useBOM</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# If <var>label</var> is a case-insensitive match for "<code>utf-16</code>" then set the internal <var>useBOM</var> flag.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If <var>view</var> is not specified, let <var>view</var> be a <code>Uint8Array</code> of length <code>0</code>.
# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var> and set <var>offset</var> to <code>0</code>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set. Otherwise the internal <var>streaming</var> flag is cleared.
# Run or resume the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>.
#** If <var>offset</var> is greater than <code>0</code>, the bytes in <var>byte stream</var> for positions less than <var>offset</var> are provided by buffer(s) passed in to previous calls to <code>decode()</code>.
#** The bytes in <var>byte stream</var> for positions <var>offset</var> through <code><var>offset</var> + <var>view.byteLength</var> - 1</code> are provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>.
#** When accessing the byte in <var>byte stream</var> at position code><var>offset</var> + <var>view.byteLength</var></code>:
#*** If the internal <var>streaming</var> flag is not set, then yield the '''EOF byte'''.
#*** Otherwise, set <var>offset</var> to <code><var>offset</var> + <var>view.byteLength</var></code> and suspend the steps of the decoder algorithm until a subsequent call to <code>decode()</code>
#* If the internal <var>useBOM</var> flag is set and <var>offset</var> is <code>0</code> then:
#** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend the decoding algorithm and return the empty string.
#** If the next two bytes of the stream are '''0xFF 0xFE''' then set <var>offset</var> to <code>2</code>, and clear the <var>utf-16be</var> flag of the decoder state.
#** If the next two bytes of the stream are '''0xFE 0xFF''' then set <var>offset</var> to <code>2</code>, and set the <var>utf-16be</var> flag of the decoder state.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a <var>sequence of emitted code points</var>.
# Return an IDL <code>DOMString</code> value that represents the sequence of code units resulting from encoding the <var>sequence of emitted code points</var> as UTF-16.

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren
* Cameron McCormack

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-09-13T22:58:49Z

Jsbell: Incorporate BOM decoding

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.

* Should the <var>encoding</var> attribute return the <var>Name</var> of the encoding or the name that was passed in?

== Notes to Implementers ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.

<dt>Should legacy encodings be supported?
<dd>Resolution: Consensus on the [http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-August/036825.html WHATWG mailing list] - support legacy encodings for decode, and only 'utf-8', 'utf-16' and 'utf-16be' for encode.

<dt>What do to on encoding errors? If non-UTF encodings are supported then we may want to allow substitution (e.g. ASCII '?') or a script callback (for arbitrary escaping).
<dd>Resolution: not for "v1" (see above)

</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
Uint8Array encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, if the <var>Name</var> of the returned encoding is not one of "<code>utf-8</code>", "<code>utf-16</code>", or "<code>utf-16be</code>" throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

:''NOTE: Because only UTF encodings are supported, and because of the use of the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] and thus Unicode code points, no input can cause the encoding process to emit an encoder error.''

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, a <var>streaming</var> flag which is initially unset, an <var>offset</var> pointer which is initially <code>0</code>, and a <var>useBOM</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# If <var>label</var> is a case-insensitive match for "<code>utf-16</code>" then set the internal <var>useBOM</var> flag.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If <var>view</var> is not specified, let <var>view</var> be a <code>Uint8Array</code> of length <code>0</code>.
# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var> and set <var>offset</var> to <code>0</code>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set. Otherwise the internal <var>streaming</var> flag is cleared.
# Run or resume the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>.
#** If <var>offset</var> is greater than <code>0</code>, the bytes in <var>byte stream</var> for positions less than <var>offset</var> are provided by buffer(s) passed in to previous calls to <code>decode()</code>.
#** The bytes in <var>byte stream</var> for positions <var>offset</var> through <code><var>offset</var> + <var>view.byteLength</var> - 1</code> are provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>.
#** When accessing the byte in <var>byte stream</var> at position code><var>offset</var> + <var>view.byteLength</var></code>:
#*** If the internal <var>streaming</var> flag is not set, then yield the '''EOF byte'''.
#*** Otherwise, set <var>offset</var> to <code><var>offset</var> + <var>view.byteLength</var></code> and suspend the steps of the decoder algorithm until a subsequent call to <code>decode()</code>
#* If the internal <var>useBOM</var> flag is set and <var>offset</var> is <code>0</code> then:
#** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend the decoding algorithm and return the empty string.
#** If the next two bytes of the stream are '''0xFF 0xFE''' then set <var>offset</var> to <code>2</code>, and clear the <var>utf-16be</var> flag of the decoder state.
#** If the next two bytes of the stream are '''0xFE 0xFF''' then set <var>offset</var> to <code>2</code>, and set the <var>utf-16be</var> flag of the decoder state.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a <var>sequence of emitted code points</var>.
# Return an IDL <code>DOMString</code> value that represents the sequence of code units resulting from encoding the <var>sequence of emitted code points</var> as UTF-16.

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren
* Cameron McCormack

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-08-22T00:34:33Z

Jsbell: Decoding WIP

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.

* Should the <var>encoding</var> attribute return the <var>Name</var> of the encoding or the name that was passed in?

== Notes to Implementers ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.

<dt>Should legacy encodings be supported?
<dd>Resolution: Consensus on the [http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-August/036825.html WHATWG mailing list] - support legacy encodings for decode, and only 'utf-8', 'utf-16' and 'utf-16be' for encode.

<dt>What do to on encoding errors? If non-UTF encodings are supported then we may want to allow substitution (e.g. ASCII '?') or a script callback (for arbitrary escaping).
<dd>Resolution: not for "v1" (see above)

</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
Uint8Array encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, if the <var>Name</var> of the returned encoding is not one of "<code>utf-8</code>", "<code>utf-16</code>", or "<code>utf-16be</code>" throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

:''NOTE: Because only UTF encodings are supported, and because of the use of the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] and thus Unicode code points, no input can cause the encoding process to emit an encoder error.''

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, a <var>streaming</var> flag which is initially unset, an <var>offset</var> pointer which is initially <code>0</code>, and a <var>useBOM</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# If <var>label</var> is a case-insensitive match for "<code>utf-16</code>" then set the internal <var>useBOM</var> flag.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If <var>view</var> is not specified, let <var>view</var> be a <code>Uint8Array</code> of length <code>0</code>.
# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var> and set <var>offset</var> to <code>0</code>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set. Otherwise the internal <var>streaming</var> flag is cleared.
# Run or resume the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>.
#** If <var>offset</var> is greater than <code>0</code>, the bytes in <var>byte stream</var> for positions less than <var>offset</var> are provided by buffer(s) passed in to previous calls to <code>decode()</code>.
#** The bytes in <var>byte stream</var> for positions <var>offset</var> through <code><var>offset</var> + <var>view.byteLength</var> - 1</code> are provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>.
#** When accessing the byte in <var>byte stream</var> at position code><var>offset</var> + <var>view.byteLength</var></code>:
#*** If the internal <var>streaming</var> flag is not set, then yield the '''EOF byte'''.
#*** Otherwise, set <var>offset</var> to <code><var>offset</var> + <var>view.byteLength</var></code> and suspend the steps of the decoder algorithm until a subsequent call to <code>decode()</code>

#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a <var>sequence of emitted code points</var>.
# Return an IDL <code>DOMString</code> value that represents the sequence of code units resulting from encoding the <var>sequence of emitted code points</var> as UTF-16.

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren
* Cameron McCormack

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-08-13T16:07:46Z

Jsbell: Clarify informative note.

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.

* Should the <var>encoding</var> attribute return the <var>Name</var> of the encoding or the name that was passed in?

== Notes to Implementers ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.

<dt>Should legacy encodings be supported?
<dd>Resolution: Consensus on the [http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-August/036825.html WHATWG mailing list] - support legacy encodings for decode, and only 'utf-8', 'utf-16' and 'utf-16be' for encode.

<dt>What do to on encoding errors? If non-UTF encodings are supported then we may want to allow substitution (e.g. ASCII '?') or a script callback (for arbitrary escaping).
<dd>Resolution: not for "v1" (see above)

</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
Uint8Array encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, if the <var>Name</var> of the returned encoding is not one of "<code>utf-8</code>", "<code>utf-16</code>", or "<code>utf-16be</code>" throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

:''NOTE: Because only UTF encodings are supported, and because of the use of the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] and thus Unicode code points, no input can cause the encoding process to emit an encoder error.''

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a <var>sequence of emitted code points</var>.
# Return an IDL <code>DOMString</code> value that represents the sequence of code units resulting from encoding the <var>sequence of emitted code points</var> as UTF-16.

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren
* Cameron McCormack

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-08-09T17:33:52Z

Jsbell: Restrict encodings to UTF

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.

* Should the <var>encoding</var> attribute return the <var>Name</var> of the encoding or the name that was passed in?

== Notes to Implementers ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.

<dt>Should legacy encodings be supported?
<dd>Resolution: Consensus on the [http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-August/036825.html WHATWG mailing list] - support legacy encodings for decode, and only 'utf-8', 'utf-16' and 'utf-16be' for encode.

<dt>What do to on encoding errors? If non-UTF encodings are supported then we may want to allow substitution (e.g. ASCII '?') or a script callback (for arbitrary escaping).
<dd>Resolution: not for "v1" (see above)

</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
Uint8Array encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, if the <var>Name</var> of the returned encoding is not one of "<code>utf-8</code>", "<code>utf-16</code>", or "<code>utf-16be</code>" throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

:''NOTE: Because only UTF encodings are supported, and because of the algorithm used to convert a DOMString to a sequence of Unicode characters, no input can cause the encoding process to emit an encoder error.''

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a <var>sequence of emitted code points</var>.
# Return an IDL <code>DOMString</code> value that represents the sequence of code units resulting from encoding the <var>sequence of emitted code points</var> as UTF-16.

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren
* Cameron McCormack

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-08-06T17:49:48Z

Jsbell: Fixed return type of encode (ArrayBufferView -> Uint8Array)

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Right now, the encoding process is defined to throw on encoding errors (i.e. when trying to encode a code point that isn't supported in the specified encoding). There are four reasonable options:
*** Throw (as currently spec'd)
*** Encoding-specific replacement character (e.g. '?')
*** Custom replacement character
*** Custom behavior e.g. a user-defined function to handle the code point, such as: <code>function (cp) { return "&#x" + Number(cp).toString(16) + ";"; }</code>
** Should the spec define an options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.

== Notes to Implementers ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
Uint8Array encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a <var>sequence of emitted code points</var>.
# Return an IDL <code>DOMString</code> value that represents the sequence of code units resulting from encoding the <var>sequence of emitted code points</var> as UTF-16.

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren
* Cameron McCormack

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-08-06T17:49:02Z

Jsbell: Fleshed out description of encoding error options

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Right now, the encoding process is defined to throw on encoding errors (i.e. when trying to encode a code point that isn't supported in the specified encoding). There are four reasonable options:
*** Throw (as currently spec'd)
*** Encoding-specific replacement character (e.g. '?')
*** Custom replacement character
*** Custom behavior e.g. a user-defined function to handle the code point, such as: <code>function (cp) { return "&#x" + Number(cp).toString(16) + ";"; }</code>
** Should the spec define an options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.

== Notes to Implementers ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a <var>sequence of emitted code points</var>.
# Return an IDL <code>DOMString</code> value that represents the sequence of code units resulting from encoding the <var>sequence of emitted code points</var> as UTF-16.

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren
* Cameron McCormack

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-28T15:59:15Z

Jsbell: /* Acknowledgements */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.

== Notes to Implementers ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a <var>sequence of emitted code points</var>.
# Return an IDL <code>DOMString</code> value that represents the sequence of code units resulting from encoding the <var>sequence of emitted code points</var> as UTF-16.

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren
* Cameron McCormack

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-28T15:58:06Z

Jsbell: /* Notes and TODOs */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.

== Notes to Implementers ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a <var>sequence of emitted code points</var>.
# Return an IDL <code>DOMString</code> value that represents the sequence of code units resulting from encoding the <var>sequence of emitted code points</var> as UTF-16.

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-28T15:57:43Z

Jsbell: Based on feedback from Cameron McCormack, don't need detailed steps for Unicode characters -> DOMString conversion

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.

== Notes and TODOs ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a <var>sequence of emitted code points</var>.
# Return an IDL <code>DOMString</code> value that represents the sequence of code units resulting from encoding the <var>sequence of emitted code points</var> as UTF-16.

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T20:49:59Z

Jsbell: Add missing definite article

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.

== Notes and TODOs ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]
* Move the algorithm to convert a sequence of Unicode characters to a DOMString into [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] - [https://www.w3.org/Bugs/Public/show_bug.cgi?id=17620 tracking bug].

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> following the [[#Steps to convert a sequence of Unicode characters to a DOMString|steps to convert a sequence of Unicode characters to a DOMString]].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Algorithms ==

=== Steps to convert a sequence of Unicode characters to a DOMString ===

:TODO: Move the following to [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] - [https://www.w3.org/Bugs/Public/show_bug.cgi?id=17620 tracking bug]

The following algorithm defines a way to convert a sequence of Unicode characters to a DOMString:
# Let <var>U</var>0...<var>n</var>-1 be the sequence of Unicode characters
# Initialize <var>i</var> to 0
# Initialize <var>S</var> to be an empty sequence of [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code units]
# While <var>i</var> < <var>n</var>
## Let <var>c</var> be the code point of the Unicode character in <var>U</var> at index <var>i</var>
## If <var>c</var> ≥ 216, then:
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) / 210 + 0xD800, where "/" represents integer division.
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) % 210 + 0xDC00, where "%" represents the remainder of an integer division.
## Otherwise, append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to <var>c</var>.
## Set <var>i</var> to <var>i</var>+1
# Return the IDL [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString DOMString] value that represents the sequence of code units <var>S</var>.

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T20:33:40Z

Jsbell: /* Notes and TODOs */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.

== Notes and TODOs ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]
* Move the algorithm to convert a sequence of Unicode characters to a DOMString into [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] - [https://www.w3.org/Bugs/Public/show_bug.cgi?id=17620 tracking bug].

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> following the [[#Steps to convert a sequence of Unicode characters to a DOMString|steps to convert a sequence of Unicode characters to a DOMString]].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Algorithms ==

=== Steps to convert a sequence of Unicode characters to a DOMString ===

:TODO: Move the following to [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] - [https://www.w3.org/Bugs/Public/show_bug.cgi?id=17620 tracking bug]

The following algorithm defines a way to convert a sequence of Unicode characters to a DOMString:
# Let <var>U</var>0...<var>n</var>-1 be the sequence of Unicode characters
# Initialize <var>i</var> to 0
# Initialize <var>S</var> to be an empty sequence of [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code units]
# While <var>i</var> < <var>n</var>
## Let <var>c</var> be the code point of the Unicode character in <var>U</var> at index <var>i</var>
## If <var>c</var> ≥ 216, then:
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) / 210 + 0xD800, where "/" represents integer division.
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) % 210 + 0xDC00, where "%" represents the remainder of an integer division.
## Otherwise, append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to <var>c</var>.
## Set <var>i</var> to <var>i</var>+1
# Return the IDL [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString DOMString] value that represents sequence of code units <var>S</var>.

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T20:32:50Z

Jsbell: /* Steps to convert a sequence of Unicode characters to a DOMString */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.

== Notes and TODOs ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]
* Move the algorithm to convert a sequence of Unicode characters to a DOMString into [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> following the [[#Steps to convert a sequence of Unicode characters to a DOMString|steps to convert a sequence of Unicode characters to a DOMString]].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Algorithms ==

=== Steps to convert a sequence of Unicode characters to a DOMString ===

:TODO: Move the following to [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] - [https://www.w3.org/Bugs/Public/show_bug.cgi?id=17620 tracking bug]

The following algorithm defines a way to convert a sequence of Unicode characters to a DOMString:
# Let <var>U</var>0...<var>n</var>-1 be the sequence of Unicode characters
# Initialize <var>i</var> to 0
# Initialize <var>S</var> to be an empty sequence of [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code units]
# While <var>i</var> < <var>n</var>
## Let <var>c</var> be the code point of the Unicode character in <var>U</var> at index <var>i</var>
## If <var>c</var> ≥ 216, then:
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) / 210 + 0xD800, where "/" represents integer division.
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) % 210 + 0xDC00, where "%" represents the remainder of an integer division.
## Otherwise, append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to <var>c</var>.
## Set <var>i</var> to <var>i</var>+1
# Return the IDL [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString DOMString] value that represents sequence of code units <var>S</var>.

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T17:38:31Z

Jsbell: /* Appendix */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.

== Notes and TODOs ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]
* Move the algorithm to convert a sequence of Unicode characters to a DOMString into [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> following the [[#Steps to convert a sequence of Unicode characters to a DOMString|steps to convert a sequence of Unicode characters to a DOMString]].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Algorithms ==

=== Steps to convert a sequence of Unicode characters to a DOMString ===

:TODO: Move the following to [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL]

The following algorithm defines a way to convert a sequence of Unicode characters to a DOMString:
# Let <var>U</var>0...<var>n</var>-1 be the sequence of Unicode characters
# Initialize <var>i</var> to 0
# Initialize <var>S</var> to be an empty sequence of [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code units]
# While <var>i</var> < <var>n</var>
## Let <var>c</var> be the code point of the Unicode character in <var>U</var> at index <var>i</var>
## If <var>c</var> ≥ 216, then:
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) / 210 + 0xD800, where "/" represents integer division.
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) % 210 + 0xDC00, where "%" represents the remainder of an integer division.
## Otherwise, append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to <var>c</var>.
## Set <var>i</var> to <var>i</var>+1
# Return the IDL [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString DOMString] value that represents sequence of code units <var>S</var>.

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in ECMAScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T17:38:16Z

Jsbell: /* Notes and TODOs */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.

== Notes and TODOs ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** This is not yet implemented in the [http://code.google.com/p/stringencoding/ ECMAScript shim]
* Move the algorithm to convert a sequence of Unicode characters to a DOMString into [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> following the [[#Steps to convert a sequence of Unicode characters to a DOMString|steps to convert a sequence of Unicode characters to a DOMString]].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Algorithms ==

=== Steps to convert a sequence of Unicode characters to a DOMString ===

:TODO: Move the following to [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL]

The following algorithm defines a way to convert a sequence of Unicode characters to a DOMString:
# Let <var>U</var>0...<var>n</var>-1 be the sequence of Unicode characters
# Initialize <var>i</var> to 0
# Initialize <var>S</var> to be an empty sequence of [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code units]
# While <var>i</var> < <var>n</var>
## Let <var>c</var> be the code point of the Unicode character in <var>U</var> at index <var>i</var>
## If <var>c</var> ≥ 216, then:
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) / 210 + 0xD800, where "/" represents integer division.
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) % 210 + 0xDC00, where "%" represents the remainder of an integer division.
## Otherwise, append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to <var>c</var>.
## Set <var>i</var> to <var>i</var>+1
# Return the IDL [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString DOMString] value that represents sequence of code units <var>S</var>.

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T17:37:14Z

Jsbell: /* Notes and TODOs */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.

== Notes and TODOs ==

* Streaming decode/encode requires retaining partial buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
* Move the algorithm to convert a sequence of Unicode characters to a DOMString into [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> following the [[#Steps to convert a sequence of Unicode characters to a DOMString|steps to convert a sequence of Unicode characters to a DOMString]].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Algorithms ==

=== Steps to convert a sequence of Unicode characters to a DOMString ===

:TODO: Move the following to [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL]

The following algorithm defines a way to convert a sequence of Unicode characters to a DOMString:
# Let <var>U</var>0...<var>n</var>-1 be the sequence of Unicode characters
# Initialize <var>i</var> to 0
# Initialize <var>S</var> to be an empty sequence of [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code units]
# While <var>i</var> < <var>n</var>
## Let <var>c</var> be the code point of the Unicode character in <var>U</var> at index <var>i</var>
## If <var>c</var> ≥ 216, then:
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) / 210 + 0xD800, where "/" represents integer division.
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) % 210 + 0xDC00, where "%" represents the remainder of an integer division.
## Otherwise, append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to <var>c</var>.
## Set <var>i</var> to <var>i</var>+1
# Return the IDL [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString DOMString] value that represents sequence of code units <var>S</var>.

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T17:36:44Z

Jsbell: /* Open Issues */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.

== Notes and TODOs ==

* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
* Move the algorithm to convert a sequence of Unicode characters to a DOMString into [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> following the [[#Steps to convert a sequence of Unicode characters to a DOMString|steps to convert a sequence of Unicode characters to a DOMString]].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Algorithms ==

=== Steps to convert a sequence of Unicode characters to a DOMString ===

:TODO: Move the following to [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL]

The following algorithm defines a way to convert a sequence of Unicode characters to a DOMString:
# Let <var>U</var>0...<var>n</var>-1 be the sequence of Unicode characters
# Initialize <var>i</var> to 0
# Initialize <var>S</var> to be an empty sequence of [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code units]
# While <var>i</var> < <var>n</var>
## Let <var>c</var> be the code point of the Unicode character in <var>U</var> at index <var>i</var>
## If <var>c</var> ≥ 216, then:
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) / 210 + 0xD800, where "/" represents integer division.
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) % 210 + 0xDC00, where "%" represents the remainder of an integer division.
## Otherwise, append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to <var>c</var>.
## Set <var>i</var> to <var>i</var>+1
# Return the IDL [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString DOMString] value that represents sequence of code units <var>S</var>.

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T17:36:17Z

Jsbell: /* Open Issues */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** ''This is more of a note to implementers than an issue, but it may merit further discussion.''
* Move the algorithm to convert a sequence of Unicode characters to a DOMString into [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> following the [[#Steps to convert a sequence of Unicode characters to a DOMString|steps to convert a sequence of Unicode characters to a DOMString]].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Algorithms ==

=== Steps to convert a sequence of Unicode characters to a DOMString ===

:TODO: Move the following to [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL]

The following algorithm defines a way to convert a sequence of Unicode characters to a DOMString:
# Let <var>U</var>0...<var>n</var>-1 be the sequence of Unicode characters
# Initialize <var>i</var> to 0
# Initialize <var>S</var> to be an empty sequence of [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code units]
# While <var>i</var> < <var>n</var>
## Let <var>c</var> be the code point of the Unicode character in <var>U</var> at index <var>i</var>
## If <var>c</var> ≥ 216, then:
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) / 210 + 0xD800, where "/" represents integer division.
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) % 210 + 0xDC00, where "%" represents the remainder of an integer division.
## Otherwise, append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to <var>c</var>.
## Set <var>i</var> to <var>i</var>+1
# Return the IDL [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString DOMString] value that represents sequence of code units <var>S</var>.

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T17:34:39Z

Jsbell: Correct links to algorithms

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** ''This is more of a note to implementers than an issue, but it may merit further discussion.''
* Move the algorithm to convert a sequence of Unicode characters to a DOMString into [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the [http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode steps to convert a DOMString to a sequence of Unicode characters] in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> following the [[#Steps to convert a sequence of Unicode characters to a DOMString|steps to convert a sequence of Unicode characters to a DOMString]].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Algorithms ==

=== Steps to convert a sequence of Unicode characters to a DOMString ===

:TODO: Move the following to [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL]

The following algorithm defines a way to convert a sequence of Unicode characters to a DOMString:
# Let <var>U</var>0...<var>n</var>-1 be the sequence of Unicode characters
# Initialize <var>i</var> to 0
# Initialize <var>S</var> to be an empty sequence of [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code units]
# While <var>i</var> < <var>n</var>
## Let <var>c</var> be the code point of the Unicode character in <var>U</var> at index <var>i</var>
## If <var>c</var> ≥ 216, then:
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) / 210 + 0xD800, where "/" represents integer division.
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) % 210 + 0xDC00, where "%" represents the remainder of an integer division.
## Otherwise, append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to <var>c</var>.
## Set <var>i</var> to <var>i</var>+1
# Return the IDL [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString DOMString] value that represents sequence of code units <var>S</var>.

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T17:27:44Z

Jsbell: /* TextEncoder */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** ''This is more of a note to implementers than an issue, but it may merit further discussion.''
* Move the algorithm to convert a sequence of Unicode characters to a DOMString into [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the steps to ''[http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode convert a DOMString to a sequence of Unicode characters]'' in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> following the steps to convert a sequence of Unicode characters to a DOMString.

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Algorithms ==

:TODO: Move the following to [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL]

The following algorithm defines a way to convert a sequence of Unicode characters to a DOMString:
# Let <var>U</var>0...<var>n</var>-1 be the sequence of Unicode characters
# Initialize <var>i</var> to 0
# Initialize <var>S</var> to be an empty sequence of [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code units]
# While <var>i</var> < <var>n</var>
## Let <var>c</var> be the code point of the Unicode character in <var>U</var> at index <var>i</var>
## If <var>c</var> ≥ 216, then:
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) / 210 + 0xD800, where "/" represents integer division.
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) % 210 + 0xDC00, where "%" represents the remainder of an integer division.
## Otherwise, append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to <var>c</var>.
## Set <var>i</var> to <var>i</var>+1
# Return the IDL [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString DOMString] value that represents sequence of code units <var>S</var>.

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T17:25:54Z

Jsbell: /* Open Issues */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** ''This is more of a note to implementers than an issue, but it may merit further discussion.''
* Move the algorithm to convert a sequence of Unicode characters to a DOMString into [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the steps to ''[http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode convert a DOMString to a sequence of Unicode characters]'' in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> following the steps to convert a sequence of Unicode characters to a DOMString.

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Algorithms ==

:TODO: Move the following to [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL]

The following algorithm defines a way to convert a sequence of Unicode characters to a DOMString:
# Let <var>U</var>0...<var>n</var>-1 be the sequence of Unicode characters
# Initialize <var>i</var> to 0
# Initialize <var>S</var> to be an empty sequence of [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code units]
# While <var>i</var> < <var>n</var>
## Let <var>c</var> be the code point of the Unicode character in <var>U</var> at index <var>i</var>
## If <var>c</var> ≥ 216, then:
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) / 210 + 0xD800, where "/" represents integer division.
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) % 210 + 0xDC00, where "%" represents the remainder of an integer division.
## Otherwise, append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to <var>c</var>.
## Set <var>i</var> to <var>i</var>+1
# Return the IDL [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString DOMString] value that represents sequence of code units <var>S</var>.

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T17:25:09Z

Jsbell: Define Unicode character sequence -> DOMString

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** ''This is more of a note to implementers than an issue, but it may merit further discussion.''
* Converting a sequence of Unicode characters to a DOMString needs to be defined.
** [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] currently only defines the reverse, convert a DOMString to a sequence of Unicode characters, but that's probably the right place to define it in the platform.

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the steps to ''[http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode convert a DOMString to a sequence of Unicode characters]'' in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> following the steps to convert a sequence of Unicode characters to a DOMString.

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Algorithms ==

:TODO: Move the following to [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL]

The following algorithm defines a way to convert a sequence of Unicode characters to a DOMString:
# Let <var>U</var>0...<var>n</var>-1 be the sequence of Unicode characters
# Initialize <var>i</var> to 0
# Initialize <var>S</var> to be an empty sequence of [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code units]
# While <var>i</var> < <var>n</var>
## Let <var>c</var> be the code point of the Unicode character in <var>U</var> at index <var>i</var>
## If <var>c</var> ≥ 216, then:
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) / 210 + 0xD800, where "/" represents integer division.
### Append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to (<var>c</var> - 216) % 210 + 0xDC00, where "%" represents the remainder of an integer division.
## Otherwise, append to <var>S</var> a [http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit code unit] equal to <var>c</var>.
## Set <var>i</var> to <var>i</var>+1
# Return the IDL [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString DOMString] value that represents sequence of code units <var>S</var>.

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T16:54:39Z

Jsbell: Added Algorithms section

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** ''This is more of a note to implementers than an issue, but it may merit further discussion.''
* Converting a sequence of Unicode characters to a DOMString needs to be defined.
** [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] currently only defines the reverse, convert a DOMString to a sequence of Unicode characters, but that's probably the right place to define it in the platform.

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the steps to ''[http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode convert a DOMString to a sequence of Unicode characters]'' in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> following the steps to convert a sequence of Unicode characters to a DOMString.

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Algorithms ==

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T16:53:47Z

Jsbell: HTML typo

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** ''This is more of a note to implementers than an issue, but it may merit further discussion.''
* Converting a sequence of Unicode characters to a DOMString needs to be defined.
** [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] currently only defines the reverse, convert a DOMString to a sequence of Unicode characters, but that's probably the right place to define it in the platform.

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the steps to ''[http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode convert a DOMString to a sequence of Unicode characters]'' in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> following the steps to convert a sequence of Unicode characters to a DOMString.

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T16:53:04Z

Jsbell: Reference steps for Unicode code points -> DOMString (TBD)

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** ''This is more of a note to implementers than an issue, but it may merit further discussion.''
* Converting a sequence of Unicode characters to a DOMString needs to be defined.
** [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] currently only defines the reverse, convert a DOMString to a sequence of Unicode characters, but that's probably the right place to define it in the platform.

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the steps to ''[http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode convert a DOMString to a sequence of Unicode characters]'' in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> following the steps to convert a sequence of Unicode characters to a DOMString.

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T16:50:25Z

Jsbell: Reference WebIDL for DOMString -> code points

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** ''This is more of a note to implementers than an issue, but it may merit further discussion.''
* Converting a sequence of Unicode characters to a DOMString needs to be defined.
** [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] currently only defines the reverse, convert a DOMString to a sequence of Unicode characters, but that's probably the right place to define it in the platform.

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The stream is composed of the Unicode code points for the Unicode characters produced by following the steps to ''[http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode convert a DOMString to a sequence of Unicode characters]'' in [http://dev.w3.org/2006/webapi/WebIDL/ WebIDL] with <var>string</var> as the input. If <var>string</var> is null, the <var>stream of code points</var> is empty.
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T16:44:53Z

Jsbell: /* Open Issues */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** ''This is more of a note to implementers than an issue, but it may merit further discussion.''
* Converting a sequence of Unicode characters to a DOMString needs to be defined.
** [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] currently only defines the reverse, convert a DOMString to a sequence of Unicode characters, but that's probably the right place to define it in the platform.

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 code units to yield a code point stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-27T16:18:39Z

Jsbell: html typo

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** ''This is more of a note to implementers than an issue, but it may merit further discussion.''
* Interpreting a DOMString as UTF-16 code units to yield a code point stream needs to be defined, including unpaired surrogates.
** [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] currently only defines the reverse, but that's probably the right place to define it in the platform.

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 code units to yield a code point stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-15T21:50:26Z

Jsbell: /* Resolved Issues */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** ''This is more of a note to implementers than an issue, but it may merit further discussion.''
* Interpreting a DOMString as UTF-16 code units to yield a code point stream needs to be defined, including unpaired surrogates.
** [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] currently only defines the reverse, but that's probably the right place to define it in the platform.

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding? ''(a proposed 8-bit-clean encoding for interop with legacy binary data stored in ECMAScript strings)''
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 code units to yield a code point stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream<var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-15T21:48:24Z

Jsbell: /* Open Issues */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** ''This is more of a note to implementers than an issue, but it may merit further discussion.''
* Interpreting a DOMString as UTF-16 code units to yield a code point stream needs to be defined, including unpaired surrogates.
** [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] currently only defines the reverse, but that's probably the right place to define it in the platform.

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding?
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 code units to yield a code point stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream<var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-15T21:47:14Z

Jsbell: /* Open Issues */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** ''This is more of a note to implementers than an issue, but it may merit further discussion.''

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding?
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 code units to yield a code point stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream<var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-15T21:46:57Z

Jsbell:

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write TextDecoder('iso-2022-kr').decode(str) if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** ''This is more of a note to implementers than an issue, but it may merit further discussion.''

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding?
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>

<dt>Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
<dd>There seems to be pretty strong consensus that streaming, stateful coding is high priority and that an object-oriented API is cleanest, and shoe-horning those onto existing objects would be messy.
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 code units to yield a code point stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream<var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-15T21:45:08Z

Jsbell: /* Open Issues */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write TextDecoder('iso-2022-kr').decode(str) if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.
** ''This is more of a note to implementers than an issue, but it may merit further discussion.''

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding?
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 code units to yield a code point stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream<var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-15T21:43:42Z

Jsbell: Removed "binary" encoding

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write TextDecoder('iso-2022-kr').decode(str) if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>

<dt>Remove ''binary'' encoding?
<dd>The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays. Consensus on WHATWG is to add better APIs e.g.
<pre>
partial interface ArrayBufferView {
DOMString toBase64();
};

partial interface ArrayBuffer {
static ArrayBuffer fromBase64(DOMString string);
};
</pre>
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 code units to yield a code point stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream<var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-08T17:01:47Z

Jsbell: /* Open Issues */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write TextDecoder('iso-2022-kr').decode(str) if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Remove ''binary'' encoding? The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays
* Streaming decode/encode requires keeping buffers between calls.
** Some encode/decode algorithms require adjusting the <var>code point pointer</var> or <var>byte pointer</var> by a negative amount. This could occur across "chunk" boundaries. This implies that when the internal <var>streaming</var> flag is set on an encoder/decoder that the last N elements of the stream are saved for the next call and used as a prefix for the stream. N is defined by the specific encoding algorithm.

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 code units to yield a code point stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream<var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

=== Additional Encodings ===

The following additional encodings are defined by this specification. They are specific to the methods defined herein.

<table border=1 cellpadding=5>
<tr><th>Name<th>Labels
<tr><td>binary<td>"binary"
</table>

==== binary ====

The '''binary''' encoding is a single-byte encoding where the input code point and output byte are identical for the range '''U+0000''' to '''U+00ff'''.

:''NOTE: This encoding is intended to allow interoperation with legacy code that encodes binary data in ECMAScript strings, for example the WindowBase64 methods [http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-atob atob()]/[http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-btoa btoa()] methods. It is recommended that new Web applications use Typed Arrays for transmission and storage of binary data.''

If '''binary''' is selected as the encoding, then step 3 of the steps to ''decode a byte stream'' are skipped; no BOM detection is performed.

:''NOTE: If <code>"binary"</code> is specified, byte order marks must be ignored.''

The '''binary decoder''' is:
# Let <var>byte</var> be byte pointer.
# If <byte> is the EOF byte, emit the EOF code point.
# Increase the byte pointer by one.
# Emit a code point whose value is <var>byte</var>

The '''binary encoder''' is:
# Let <var>code point</var> be the code point pointer
# If <var>code point</var> is the EOF code point, emit the EOF byte.
# Increase the code point pointer by one.
# If <var>code point</var> is in the range U+0000 to U+00FF, emit a byte whose value is <var>code point</var>
# Emit an encoder error

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-08T16:58:44Z

Jsbell: /* TextEncoder */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write TextDecoder('iso-2022-kr').decode(str) if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Remove ''binary'' encoding? The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 code units to yield a code point stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.
</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream<var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

=== Additional Encodings ===

The following additional encodings are defined by this specification. They are specific to the methods defined herein.

<table border=1 cellpadding=5>
<tr><th>Name<th>Labels
<tr><td>binary<td>"binary"
</table>

==== binary ====

The '''binary''' encoding is a single-byte encoding where the input code point and output byte are identical for the range '''U+0000''' to '''U+00ff'''.

:''NOTE: This encoding is intended to allow interoperation with legacy code that encodes binary data in ECMAScript strings, for example the WindowBase64 methods [http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-atob atob()]/[http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-btoa btoa()] methods. It is recommended that new Web applications use Typed Arrays for transmission and storage of binary data.''

If '''binary''' is selected as the encoding, then step 3 of the steps to ''decode a byte stream'' are skipped; no BOM detection is performed.

:''NOTE: If <code>"binary"</code> is specified, byte order marks must be ignored.''

The '''binary decoder''' is:
# Let <var>byte</var> be byte pointer.
# If <byte> is the EOF byte, emit the EOF code point.
# Increase the byte pointer by one.
# Emit a code point whose value is <var>byte</var>

The '''binary encoder''' is:
# Let <var>code point</var> be the code point pointer
# If <var>code point</var> is the EOF code point, emit the EOF byte.
# Increase the code point pointer by one.
# If <var>code point</var> is in the range U+0000 to U+00FF, emit a byte whose value is <var>code point</var>
# Emit an encoder error

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-06-08T16:52:15Z

Jsbell: /* TextEncoder */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write TextDecoder('iso-2022-kr').decode(str) if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Remove ''binary'' encoding? The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 code units to yield a code point stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream<var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

=== Additional Encodings ===

The following additional encodings are defined by this specification. They are specific to the methods defined herein.

<table border=1 cellpadding=5>
<tr><th>Name<th>Labels
<tr><td>binary<td>"binary"
</table>

==== binary ====

The '''binary''' encoding is a single-byte encoding where the input code point and output byte are identical for the range '''U+0000''' to '''U+00ff'''.

:''NOTE: This encoding is intended to allow interoperation with legacy code that encodes binary data in ECMAScript strings, for example the WindowBase64 methods [http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-atob atob()]/[http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-btoa btoa()] methods. It is recommended that new Web applications use Typed Arrays for transmission and storage of binary data.''

If '''binary''' is selected as the encoding, then step 3 of the steps to ''decode a byte stream'' are skipped; no BOM detection is performed.

:''NOTE: If <code>"binary"</code> is specified, byte order marks must be ignored.''

The '''binary decoder''' is:
# Let <var>byte</var> be byte pointer.
# If <byte> is the EOF byte, emit the EOF code point.
# Increase the byte pointer by one.
# Emit a code point whose value is <var>byte</var>

The '''binary encoder''' is:
# Let <var>code point</var> be the code point pointer
# If <var>code point</var> is the EOF code point, emit the EOF byte.
# Increase the code point pointer by one.
# If <var>code point</var> is in the range U+0000 to U+00FF, emit a byte whose value is <var>code point</var>
# Emit an encoder error

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-05-24T17:40:58Z

Jsbell: /* Open Issues */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
* Encoding errors
** Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character / replacement callback?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write TextDecoder('iso-2022-kr').decode(str) if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Remove ''binary'' encoding? The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 to yield a code unit stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream<var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

=== Additional Encodings ===

The following additional encodings are defined by this specification. They are specific to the methods defined herein.

<table border=1 cellpadding=5>
<tr><th>Name<th>Labels
<tr><td>binary<td>"binary"
</table>

==== binary ====

The '''binary''' encoding is a single-byte encoding where the input code point and output byte are identical for the range '''U+0000''' to '''U+00ff'''.

:''NOTE: This encoding is intended to allow interoperation with legacy code that encodes binary data in ECMAScript strings, for example the WindowBase64 methods [http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-atob atob()]/[http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-btoa btoa()] methods. It is recommended that new Web applications use Typed Arrays for transmission and storage of binary data.''

If '''binary''' is selected as the encoding, then step 3 of the steps to ''decode a byte stream'' are skipped; no BOM detection is performed.

:''NOTE: If <code>"binary"</code> is specified, byte order marks must be ignored.''

The '''binary decoder''' is:
# Let <var>byte</var> be byte pointer.
# If <byte> is the EOF byte, emit the EOF code point.
# Increase the byte pointer by one.
# Emit a code point whose value is <var>byte</var>

The '''binary encoder''' is:
# Let <var>code point</var> be the code point pointer
# If <var>code point</var> is the EOF code point, emit the EOF byte.
# Increase the code point pointer by one.
# If <var>code point</var> is in the range U+0000 to U+00FF, emit a byte whose value is <var>code point</var>
# Emit an encoder error

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-05-24T17:40:27Z

Jsbell: /* Open Issues */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
* Encoding errors
** TODO: Define options dict for <code>TextEncoder()</code> that allows selecting between throw vs. replacement character?
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write TextDecoder('iso-2022-kr').decode(str) if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Remove ''binary'' encoding? The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 to yield a code unit stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream<var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

=== Additional Encodings ===

The following additional encodings are defined by this specification. They are specific to the methods defined herein.

<table border=1 cellpadding=5>
<tr><th>Name<th>Labels
<tr><td>binary<td>"binary"
</table>

==== binary ====

The '''binary''' encoding is a single-byte encoding where the input code point and output byte are identical for the range '''U+0000''' to '''U+00ff'''.

:''NOTE: This encoding is intended to allow interoperation with legacy code that encodes binary data in ECMAScript strings, for example the WindowBase64 methods [http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-atob atob()]/[http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-btoa btoa()] methods. It is recommended that new Web applications use Typed Arrays for transmission and storage of binary data.''

If '''binary''' is selected as the encoding, then step 3 of the steps to ''decode a byte stream'' are skipped; no BOM detection is performed.

:''NOTE: If <code>"binary"</code> is specified, byte order marks must be ignored.''

The '''binary decoder''' is:
# Let <var>byte</var> be byte pointer.
# If <byte> is the EOF byte, emit the EOF code point.
# Increase the byte pointer by one.
# Emit a code point whose value is <var>byte</var>

The '''binary encoder''' is:
# Let <var>code point</var> be the code point pointer
# If <var>code point</var> is the EOF code point, emit the EOF byte.
# Increase the code point pointer by one.
# If <var>code point</var> is in the range U+0000 to U+00FF, emit a byte whose value is <var>code point</var>
# Emit an encoder error

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-05-24T17:38:29Z

Jsbell: /* Open Issues */

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

* Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
* Encoding errors
* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
** Is that desirable here?
** What does it mean to e.g. write TextDecoder('iso-2022-kr').decode(str) if string turns out to be UTF-8?
** What about streaming and other re-uses of the same decoder object?
* Remove ''binary'' encoding? The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays

== Resolved Issues ==

<dl>
<dt>Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
<dd>Resolution: not for "v1" - can be implemented using this API, and breaking the string down by "character" is unlikely to be as obvious as it sounds - surrogate pairs, combining sequences, etc

<dt>Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
<dd>Resolution: See above, and wait for developer feedback.

<dt>Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
<dd>Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>
</dl>

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 to yield a code unit stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream<var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

=== Additional Encodings ===

The following additional encodings are defined by this specification. They are specific to the methods defined herein.

<table border=1 cellpadding=5>
<tr><th>Name<th>Labels
<tr><td>binary<td>"binary"
</table>

==== binary ====

The '''binary''' encoding is a single-byte encoding where the input code point and output byte are identical for the range '''U+0000''' to '''U+00ff'''.

:''NOTE: This encoding is intended to allow interoperation with legacy code that encodes binary data in ECMAScript strings, for example the WindowBase64 methods [http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-atob atob()]/[http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-btoa btoa()] methods. It is recommended that new Web applications use Typed Arrays for transmission and storage of binary data.''

If '''binary''' is selected as the encoding, then step 3 of the steps to ''decode a byte stream'' are skipped; no BOM detection is performed.

:''NOTE: If <code>"binary"</code> is specified, byte order marks must be ignored.''

The '''binary decoder''' is:
# Let <var>byte</var> be byte pointer.
# If <byte> is the EOF byte, emit the EOF code point.
# Increase the byte pointer by one.
# Emit a code point whose value is <var>byte</var>

The '''binary encoder''' is:
# Let <var>code point</var> be the code point pointer
# If <var>code point</var> is the EOF code point, emit the EOF byte.
# Increase the code point pointer by one.
# If <var>code point</var> is in the range U+0000 to U+00FF, emit a byte whose value is <var>code point</var>
# Emit an encoder error

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-05-24T17:30:48Z

Jsbell: Inline "utf-8" as default encoding

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

General: Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?

=== Scenarios ===

* Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character

=== Desired Features ===

* Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
** ''Tentative Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>''

* Encoding errors

=== API cleanup ===

* Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
** ''Tentative Resolution: Wait for developer feedback.''

=== Spec Issues ===

:''ISSUE: [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] defines the byte order mark as more authoritative than anything else. Is that desirable here?''
* Remove ''binary'' encoding? The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays

== API ==

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 to yield a code unit stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream<var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

=== Additional Encodings ===

The following additional encodings are defined by this specification. They are specific to the methods defined herein.

<table border=1 cellpadding=5>
<tr><th>Name<th>Labels
<tr><td>binary<td>"binary"
</table>

==== binary ====

The '''binary''' encoding is a single-byte encoding where the input code point and output byte are identical for the range '''U+0000''' to '''U+00ff'''.

:''NOTE: This encoding is intended to allow interoperation with legacy code that encodes binary data in ECMAScript strings, for example the WindowBase64 methods [http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-atob atob()]/[http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-btoa btoa()] methods. It is recommended that new Web applications use Typed Arrays for transmission and storage of binary data.''

If '''binary''' is selected as the encoding, then step 3 of the steps to ''decode a byte stream'' are skipped; no BOM detection is performed.

:''NOTE: If <code>"binary"</code> is specified, byte order marks must be ignored.''

The '''binary decoder''' is:
# Let <var>byte</var> be byte pointer.
# If <byte> is the EOF byte, emit the EOF code point.
# Increase the byte pointer by one.
# Emit a code point whose value is <var>byte</var>

The '''binary encoder''' is:
# Let <var>code point</var> be the code point pointer
# If <var>code point</var> is the EOF code point, emit the EOF byte.
# Increase the code point pointer by one.
# If <var>code point</var> is in the range U+0000 to U+00FF, emit a byte whose value is <var>code point</var>
# Emit an encoder error

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-05-24T17:29:49Z

Jsbell: Define TextDecoder constructor algorithmically

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

General: Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?

=== Scenarios ===

* Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character

=== Desired Features ===

* Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
** ''Tentative Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>''

* Encoding errors

=== API cleanup ===

* Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
** ''Tentative Resolution: Wait for developer feedback.''

=== Spec Issues ===

:''ISSUE: [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] defines the byte order mark as more authoritative than anything else. Is that desirable here?''
* Remove ''binary'' encoding? The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays

== API ==

The '''default encoding''' is <code>"utf-8"</code>.

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the '''default encoding'''. Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 to yield a code unit stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor runs the following steps:
# The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.
# If called without an <var>encoding</var> argument, let <var>label</var> be the '''default encoding'''. Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# Run the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
# If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, set the internal <var>fatal</var> flag of the ''decoder object''.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
# Return the ''decoder object''.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream<var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

=== Additional Encodings ===

The following additional encodings are defined by this specification. They are specific to the methods defined herein.

<table border=1 cellpadding=5>
<tr><th>Name<th>Labels
<tr><td>binary<td>"binary"
</table>

==== binary ====

The '''binary''' encoding is a single-byte encoding where the input code point and output byte are identical for the range '''U+0000''' to '''U+00ff'''.

:''NOTE: This encoding is intended to allow interoperation with legacy code that encodes binary data in ECMAScript strings, for example the WindowBase64 methods [http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-atob atob()]/[http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-btoa btoa()] methods. It is recommended that new Web applications use Typed Arrays for transmission and storage of binary data.''

If '''binary''' is selected as the encoding, then step 3 of the steps to ''decode a byte stream'' are skipped; no BOM detection is performed.

:''NOTE: If <code>"binary"</code> is specified, byte order marks must be ignored.''

The '''binary decoder''' is:
# Let <var>byte</var> be byte pointer.
# If <byte> is the EOF byte, emit the EOF code point.
# Increase the byte pointer by one.
# Emit a code point whose value is <var>byte</var>

The '''binary encoder''' is:
# Let <var>code point</var> be the code point pointer
# If <var>code point</var> is the EOF code point, emit the EOF byte.
# Increase the code point pointer by one.
# If <var>code point</var> is in the range U+0000 to U+00FF, emit a byte whose value is <var>code point</var>
# Emit an encoder error

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-05-24T17:25:01Z

Jsbell: Specify TextEncoder constructor algorithmically

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

General: Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?

=== Scenarios ===

* Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character

=== Desired Features ===

* Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
** ''Tentative Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>''

* Encoding errors

=== API cleanup ===

* Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
** ''Tentative Resolution: Wait for developer feedback.''

=== Spec Issues ===

:''ISSUE: [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] defines the byte order mark as more authoritative than anything else. Is that desirable here?''
* Remove ''binary'' encoding? The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays

== API ==

The '''default encoding''' is <code>"utf-8"</code>.

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

The constructor runs the following steps:
# If the constructor is called with no arguments, let <var>label</var> be the '''default encoding'''. Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
# The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>label</var> as <var>label</var>.
#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
#* Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding.
# Initialize the internal <var>streaming</var> flag of the encoder object to false.
# Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 to yield a code unit stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.

If called without an <var>encoding</var> argument, let <var>encoding</var> be the '''default encoding'''.

The constructor follows the steps to get an encoding from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>encoding</var> as <var>label</var>. If the steps result in failure, a <code>DOMException</code> of type <code>EncodingError</code> is thrown. Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.

If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, the internal <var>fatal</var> flag of the ''decoder object'' is set, otherwise the internal <var>fatal</var> flag is cleared.

Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream<var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

=== Additional Encodings ===

The following additional encodings are defined by this specification. They are specific to the methods defined herein.

<table border=1 cellpadding=5>
<tr><th>Name<th>Labels
<tr><td>binary<td>"binary"
</table>

==== binary ====

The '''binary''' encoding is a single-byte encoding where the input code point and output byte are identical for the range '''U+0000''' to '''U+00ff'''.

:''NOTE: This encoding is intended to allow interoperation with legacy code that encodes binary data in ECMAScript strings, for example the WindowBase64 methods [http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-atob atob()]/[http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-btoa btoa()] methods. It is recommended that new Web applications use Typed Arrays for transmission and storage of binary data.''

If '''binary''' is selected as the encoding, then step 3 of the steps to ''decode a byte stream'' are skipped; no BOM detection is performed.

:''NOTE: If <code>"binary"</code> is specified, byte order marks must be ignored.''

The '''binary decoder''' is:
# Let <var>byte</var> be byte pointer.
# If <byte> is the EOF byte, emit the EOF code point.
# Increase the byte pointer by one.
# Emit a code point whose value is <var>byte</var>

The '''binary encoder''' is:
# Let <var>code point</var> be the code point pointer
# If <var>code point</var> is the EOF code point, emit the EOF byte.
# Increase the code point pointer by one.
# If <var>code point</var> is in the range U+0000 to U+00FF, emit a byte whose value is <var>code point</var>
# Emit an encoder error

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-05-24T17:15:17Z

Jsbell: Define encode algorithm inputs/outputs more pedantically

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

General: Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?

=== Scenarios ===

* Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character

=== Desired Features ===

* Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
** ''Tentative Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>''

* Encoding errors

=== API cleanup ===

* Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
** ''Tentative Resolution: Wait for developer feedback.''

=== Spec Issues ===

:''ISSUE: [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] defines the byte order mark as more authoritative than anything else. Is that desirable here?''
* Remove ''binary'' encoding? The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays

== API ==

The '''default encoding''' is <code>"utf-8"</code>.

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

If the constructor is called with no arguments, let <var>encoding</var> be the '''default encoding'''.

The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>encoding</var> as <var>label</var>. If the steps result in failure, an <code>DOMException</code> of type <code>EncodingError</code> is thrown. Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding. Initialize the internal <var>streaming</var> flag of the encoder object to false. Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a <var>stream of code points</var>. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce the <var>stream of code points</var>; if <var>string</var> is null, the <var>stream of code points</var> is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 to yield a code unit stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a <var>sequence of emitted bytes</var>.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the <var>sequence of emitted bytes</var> by encoder algorithm.

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.

If called without an <var>encoding</var> argument, let <var>encoding</var> be the '''default encoding'''.

The constructor follows the steps to get an encoding from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>encoding</var> as <var>label</var>. If the steps result in failure, a <code>DOMException</code> of type <code>EncodingError</code> is thrown. Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.

If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, the internal <var>fatal</var> flag of the ''decoder object'' is set, otherwise the internal <var>fatal</var> flag is cleared.

Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream<var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

=== Additional Encodings ===

The following additional encodings are defined by this specification. They are specific to the methods defined herein.

<table border=1 cellpadding=5>
<tr><th>Name<th>Labels
<tr><td>binary<td>"binary"
</table>

==== binary ====

The '''binary''' encoding is a single-byte encoding where the input code point and output byte are identical for the range '''U+0000''' to '''U+00ff'''.

:''NOTE: This encoding is intended to allow interoperation with legacy code that encodes binary data in ECMAScript strings, for example the WindowBase64 methods [http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-atob atob()]/[http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-btoa btoa()] methods. It is recommended that new Web applications use Typed Arrays for transmission and storage of binary data.''

If '''binary''' is selected as the encoding, then step 3 of the steps to ''decode a byte stream'' are skipped; no BOM detection is performed.

:''NOTE: If <code>"binary"</code> is specified, byte order marks must be ignored.''

The '''binary decoder''' is:
# Let <var>byte</var> be byte pointer.
# If <byte> is the EOF byte, emit the EOF code point.
# Increase the byte pointer by one.
# Emit a code point whose value is <var>byte</var>

The '''binary encoder''' is:
# Let <var>code point</var> be the code point pointer
# If <var>code point</var> is the EOF code point, emit the EOF byte.
# Increase the code point pointer by one.
# If <var>code point</var> is in the range U+0000 to U+00FF, emit a byte whose value is <var>code point</var>
# Emit an encoder error

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]

StringEncoding

2012-05-24T17:13:42Z

Jsbell: Rewrote decode method algorithmically

Proposed Text Encoding Web API for Typed Arrays

== Editors ==

* Joshua Bell (Google, Inc)

== Abstract ==

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Discussion on this topic has so far taken place on the [http://www.khronos.org/webgl/public-mailing-list/ public_webgl@khronos.org] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.

Discussion has since moved to the [http://www.whatwg.org/mailing-list#specs WHATWG spec discussion] mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.

== Open Issues ==

General: Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?

=== Scenarios ===

* Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character

=== Desired Features ===

* Allow arbitrary end byte sequences (e.g. <code>0xFF</code> for UTF-8 strings)
** ''Tentative Resolution: Add <code>indexOf</code> to <code>ArrayBufferView</code>''

* Encoding errors

=== API cleanup ===

* Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
** ''Tentative Resolution: Wait for developer feedback.''

=== Spec Issues ===

:''ISSUE: [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] defines the byte order mark as more authoritative than anything else. Is that desirable here?''
* Remove ''binary'' encoding? The only real use is with <code>atob()</code>/<code>btoa()</code>; a better API would be Base64 directly in/out of Typed Arrays

== API ==

The '''default encoding''' is <code>"utf-8"</code>.

=== TextEncoder ===

'''WebIDL'''
<pre>
dictionary TextEncodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(DOMString encoding)]
interface TextEncoder {
readonly attribute DOMString encoding;
ArrayBufferView encode(DOMString? string, optional TextEncodeOptions options);
};
</pre>

If the constructor is called with no arguments, let <var>encoding</var> be the '''default encoding'''.

The constructor follows the '''steps to get an encoding''' from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>encoding</var> as <var>label</var>. If the steps result in failure, an <code>DOMException</code> of type <code>EncodingError</code> is thrown. Otherwise, set the encoder object's internal <var>encoding</var> property to the returned encoding. Initialize the internal <var>streaming</var> flag of the encoder object to false. Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.

<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
</dl>

<dl>
<dt><code>encode</code>
<dd>

The <code>encode</code> method runs these steps:

# If the internal <var>streaming</var> flag is not set, then reset the <var>encoding algorithm state</var> to the default values for <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>encode</code> on this object.
# If the <var>options</var> parameter is specified and the <code>stream</code> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the steps of the <var>encoding algorithm</var>:
#* The input to the algorithm is a stream of code points. The code units within the DOMString <var>string</var> are interpreted as UTF-16 code units, to produce a stream of code points; if <var>string</var> is null, the stream is empty.
#*:''ISSUE: Interpreting a DOMString as UTF-16 to yield a code unit stream needs to be defined, including unpaired surrogates. [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL] only defines the reverse.''
#* If the <var>options</var> parameter not specified or the <code>stream</code> option is false, then after final code point is yielded by the stream then the '''EOF code point''' is yielded.
#* The output of the the algorithm is a sequence of emitted bytes.
# Returns a <code>Unit8Array</code> object wrapping an <code>ArrayBuffer</code> containing the sequence of bytes emitted by encoder algorithm.

</dl>

=== TextDecoder ===

'''WebIDL'''
<pre>
dictionary TextDecoderOptions {
boolean fatal = false;
};

dictionary TextDecodeOptions {
boolean stream = false;
};

[Constructor,
Constructor(optional DOMString encoding, optional TextDecoderOptions options)]
interface TextDecoder {
readonly attribute DOMString encoding;
DOMString decode(optional ArrayBufferView view, optional TextDecodeOptions options);
};
</pre>

The constructor creates a ''decoder object''. It has the internal properties <var>encoding</var>, a <var>fatal</var> flag which is initially unset, and a <var>streaming</var> flag which is initially unset.

If called without an <var>encoding</var> argument, let <var>encoding</var> be the '''default encoding'''.

The constructor follows the steps to get an encoding from [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], with <var>encoding</var> as <var>label</var>. If the steps result in failure, a <code>DOMException</code> of type <code>EncodingError</code> is thrown. Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.

If the constructor is called with an <var>options</var> argument, and the <var>fatal</var> property of the dictionary is set, the internal <var>fatal</var> flag of the ''decoder object'' is set, otherwise the internal <var>fatal</var> flag is cleared.

Initialize the internal <var>encoding algorithm state</var> to the default values for the encoding <var>encoding</var>.
<dl>
<dt><code>encoding</code> of type DOMString, readonly
<dd>
Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].

:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.

<dt><code>decode</code>
<dd>

The <code>decode</code> method runs the following steps:

# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
# If the <var>options</var> parameter is specified and the <var>stream<var> option is '''true''', then the internal <var>streaming</var> flag is set; otherwise the internal <var>streaming</var> flag is cleared.
# Run the decoder algorithm of the ''decoder object's'' encoder
#* The input to the algorithm is a <var>byte stream</var>. The <var>byte stream</var> is provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. If <var>view</var> is not specified, the stream is empty.
#* If the <var>options</var> parameter not specified or the '''stream''' option is '''false''', then after <var>view.byteLength</var> bytes are yielded by the stream the '''EOF byte''' is yielded.
#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
#* The output of the algorithm is a sequence of <var>emitted code points</var>.
# Return a <code>DOMString</code> by encoding the <var>sequence of emitted code points</var> as UTF-16 as per [http://dev.w3.org/2006/webapi/WebIDL/#idl-DOMString WebIDL].

:''ISSUE: Need to handle BOMs. [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding] specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.''

</dl>

== Examples ==

=== Example #1 - encoding strings ===

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

<pre>
function encodeArrayOfStrings(strings, encoding) {
var encoder, encoded, len, i, bytes, view, offset;

encoder = TextEncoder(encoding);
encoded = [];

len = Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < strings.length; i += 1) {
len += Uint32Array.BYTES_PER_ELEMENT;
encoded[i] = TextEncoder(encoding).encode(strings[i]);
len += encoded[i].byteLength;
}

bytes = new Uint8Array(len);
view = new DataView(bytes.buffer);
offset = 0;

view.setUint32(offset, strings.length);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < encoded.length; i += 1) {
len = encoded[i].byteLength;
view.setUint32(offset, len);
offset += Uint32Array.BYTES_PER_ELEMENT;
bytes.set(encoded[i], offset);
offset += len;
}
return bytes.buffer;
}
</pre>

=== Example #2 - decoding strings ===

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

<pre>
function decodeArrayOfStrings(buffer, encoding) {
var decoder, view, offset, num_strings, strings, i, len;

decoder = TextDecoder(encoding);
view = new DataView(buffer);
offset = 0;
strings = [];

num_strings = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
for (i = 0; i < num_strings; i += 1) {
len = view.getUint32(offset);
offset += Uint32Array.BYTES_PER_ELEMENT;
strings[i] = decoder.decode(
new DataView(view.buffer, offset, len));
offset += len;
}
return strings;
}
</pre>

== Encodings ==

Encodings are defined and implemented per [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding]. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.

User agents MUST NOT support any other encodings or labels than those defined in [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

:''NOTE: In [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding].''

=== Additional Encodings ===

The following additional encodings are defined by this specification. They are specific to the methods defined herein.

<table border=1 cellpadding=5>
<tr><th>Name<th>Labels
<tr><td>binary<td>"binary"
</table>

==== binary ====

The '''binary''' encoding is a single-byte encoding where the input code point and output byte are identical for the range '''U+0000''' to '''U+00ff'''.

:''NOTE: This encoding is intended to allow interoperation with legacy code that encodes binary data in ECMAScript strings, for example the WindowBase64 methods [http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-atob atob()]/[http://dev.w3.org/html5/spec-author-view/webappapis.html#dom-windowbase64-btoa btoa()] methods. It is recommended that new Web applications use Typed Arrays for transmission and storage of binary data.''

If '''binary''' is selected as the encoding, then step 3 of the steps to ''decode a byte stream'' are skipped; no BOM detection is performed.

:''NOTE: If <code>"binary"</code> is specified, byte order marks must be ignored.''

The '''binary decoder''' is:
# Let <var>byte</var> be byte pointer.
# If <byte> is the EOF byte, emit the EOF code point.
# Increase the byte pointer by one.
# Emit a code point whose value is <var>byte</var>

The '''binary encoder''' is:
# Let <var>code point</var> be the code point pointer
# If <var>code point</var> is the EOF code point, emit the EOF byte.
# Increase the code point pointer by one.
# If <var>code point</var> is in the range U+0000 to U+00FF, emit a byte whose value is <var>code point</var>
# Emit an encoder error

== References ==

* WebIDL http://dev.w3.org/2006/webapi/WebIDL
* Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html

== Acknowledgements ==

* Alan Chaney
* Ben Noordhuis
* Glenn Maynard
* John Tamplin
* Kenneth Russell (Google, Inc)
* Robert Mustacchi
* Ryan Dahl
* Anne van Kesteren

== Appendix ==

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

:http://code.google.com/p/stringencoding/

[[Category:Proposals]]