A user account is required in order to edit this wiki, but we've had to disable public user registrations due to spam.
To request an account, ask an autoconfirmed user on Chat (such as one of these permanent autoconfirmed members).
StringEncoding: Difference between revisions
(Removed nullTerminator (hasta la vista, baby)) |
|||
Line 112: | Line 112: | ||
<dt><code>decode</code> | <dt><code>decode</code> | ||
<dd> | <dd> | ||
:''ISSUE: Make the first argument optional, so that terminating a stream doesn't require a dummy array?'' | |||
This method runs the decoder algorithm of the object's encoder over the byte stream from <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. | This method runs the decoder algorithm of the object's encoder over the byte stream from <var>view.buffer</var> starting at offset <var>view.byteOffset</var>. A maximum of <var>view.byteLength</var> bytes are yielded by the stream from <var>view.buffer</var>. | ||
Revision as of 05:03, 27 March 2012
Proposed String Encoding API for Typed Arrays
Editors
- Joshua Bell (Google, Inc)
Abstract
This specification defines an API for encoding strings to binary data, and decoding strings from binary data.
NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.
Discussion on this topic has so far taken place on the [email protected] mailing list. See http://www.khronos.org/webgl/public-mailing-list/archives/1111/msg00017.html for the initial discussion thread.
Discussion has since moved to the WHATWG spec discussion mailing list. See http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2012-March/035038.html for the latest discussion thread.
Open Issues
General: Should this be a standalone API (as written), or live on e.g. DataView, or on e.g. String?
Scenarios
- Encode as many characters as possible into a fixed-size buffer for transmission, and repeat starting with next unencoded character
- Parse/emit legacy data formats that do not use UTF-8
Desired Features
- Allow arbitrary end byte sequences (e.g.
0xFF
for UTF-8 strings) - Conversion errors should either produce replacement characters (U+FFFD, etc) or API should allow selection between replacement vs. throwing, or some other means of reporting errors (e.g. a count). i.e. do we set the "fatal flag" for
API cleanup
- Support two versions of encode; one which takes target buffer, and one which creates/returns a right-sized buffer
Spec Issues
- Resolve behavior when writing to a buffer that's too small - partial fill or don't change?
- ISSUE: Encoding defines the byte order mark as more authoritative than anything else. Is that desirable here?
API
The default encoding is "utf-8"
.
TextEncoder
WebIDL
dictionary TextEncodeOptions { boolean stream = false; }; [Constructor, Constructor(DOMString encoding)] interface TextEncoder { readonly attribute DOMString encoding; ArrayBufferView encode(DOMString string, optional TextEncodeOptions options); };
If the constructor is called with no arguments, let encoding be the default encoding.
The constructor follows the steps to get an encoding from Encoding, with encoding as label. If the steps result in failure, an DOMException
of type EncodingError
is thrown. Otherwise, set the encoder object's internal encoding property to the returned encoding. Initialize the internal streaming flag of the encoder object to false.
encoding
of type DOMString, readonly- Returns the Name of the encoder object's encoding, per Encoding. Note that this may differ from the name of the encoding specified during the call to the constructor.
encode
-
The code units within DOMString string are interpreted as UTF-16 code units, to produce a stream of code points stream.
- ISSUE: Interpreting a DOMString as UTF-16 to yield a code unit stream needs to be defined, including unpaired surrogates. WebIDL only defines the reverse.
encode
on this object. After the above inspection of the internal streaming flag, if the options parameter is specified and the stream option is true, then the internal streaming flag is set; otherwise the internal streaming flag is cleared. If the options parameter not specified or the stream option is false, then after final code point is yielded by the stream then the EOF code point is yielded. The method returns aUnit8Array
object wrapping anArrayBuffer
containing the bytes emitted by encoder algorithm.
TextDecoder
WebIDL
dictionary TextDecodeOptions { boolean fatal = false; boolean stream = false; }; [Constructor, Constructor(DOMString encoding)] interface TextDecoder { readonly attribute DOMString encoding; DOMString decode(ArrayBufferView view, optional TextDecodeOptions options); };
If the constructor is called with no arguments, let encoding be the default encoding.
The constructor follows the steps to get an encoding from Encoding, with encoding as label. If the steps result in failure, a DOMException
of type EncodingError
is thrown. Otherwise, set the decoder object's internal encoding property to the returned encoding. Initialize the internal streaming flag of the decoder object to false.
encoding
of type DOMString, readonly- Returns the Name of the decoder object's encoding, per Encoding. Note that this may differ from the name of the encoding specified during the call to the constructor.
decode
-
- ISSUE: Make the first argument optional, so that terminating a stream doesn't require a dummy array?
- ISSUE: Need to handle BOMs. Encoding specifies this by using the caller-specified encoding as a suggestion, and consuming the BOM as a part of selecting the real encoder where BOM takes precedence. At the very least a matching BOM should be ignored. A mismatching BOM could throw, be a decoding error, or could actually switch the decoder for this stream (call or sequence of calls), possibly if-and-only-if the constructor was called without an encoding.
decode
on this object. After the above inspection of the internal streaming flag, if the options parameter is specified and the stream option is true, then the internal streaming flag is set; otherwise the internal streaming flag is cleared. If the options parameter not specified or the stream option is false, then after view.byteLength bytes are yielded by the stream the EOF byte is yielded. If the options parameter is specified and the fatal option is true, then a decoder error causes anDOMException
of typeEncodingError
to be thrown rather than emitting a code point. Once the algorithm has no more bytes to process, the method returns aDOMString
by encoding the stream of code points emitted by the steps as UTF-16 as per WebIDL.
Examples
Example #1 - encoding strings
The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.
function encodeArrayOfStrings(strings, encoding) { var encoder, encoded, len, i, bytes, view, offset; encoder = TextEncoder(encoding); encoded = []; len = Uint32Array.BYTES_PER_ELEMENT; for (i = 0; i < strings.length; i += 1) { len += Uint32Array.BYTES_PER_ELEMENT; encoded[i] = TextEncoder(encoding).encode(strings[i]); len += encoded[i].byteLength; } bytes = new Uint8Array(len); view = new DataView(bytes.buffer); offset = 0; view.setUint32(offset, strings.length); offset += Uint32Array.BYTES_PER_ELEMENT; for (i = 0; i < encoded.length; i += 1) { len = encoded[i].byteLength; view.setUint32(offset, len); offset += Uint32Array.BYTES_PER_ELEMENT; bytes.set(encoded[i], offset); offset += len; } return bytes.buffer; }
Example #2 - decoding strings
The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.
function decodeArrayOfStrings(buffer, encoding) { var decoder, view, offset, num_strings, strings, i, len; decoder = TextDecoder(encoding); view = new DataView(buffer); offset = 0; strings = []; num_strings = view.getUint32(offset); offset += Uint32Array.BYTES_PER_ELEMENT; for (i = 0; i < num_strings; i += 1) { len = view.getUint32(offset); offset += Uint32Array.BYTES_PER_ELEMENT; strings[i] = decoder.decode( new DataView(view.buffer, offset, len)); offset += len; } return strings; }
Encodings
Encodings are defined and implemented per Encoding. This implicitly includes the steps to get an encoding from a string, and the logic for label matching and case-insensitivity.
User agents MUST NOT support any other encodings or labels than those defined in Encoding, with the additions defined below.
- ISSUE: Is it a MUST to support all encodings from Encoding spec, or list a subset here?
- ISSUE: Should anything be said about Unicode normalization forms?
- ISSUE: In Encoding, "ascii" is a label for windows-1252; there is no 7-bit-or-raise-exception encoding.
- ISSUE: In Encoding, "iso-8859-1" is a label for windows-1252 for compatibility with existing web content. Is there any reason a new API like this might distinguish them?
- NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by Encoding.
Additional Encodings
The following additional encodings are defined by this specification. They are specific to the methods defined herein.
Name | Labels |
---|---|
binary | "binary" |
binary
The binary encoding is a single-byte encoding where the input code point and output byte are identical for the range U+0000 to U+00ff.
- NOTE: This encoding is intended to allow interoperation with legacy code that encodes binary data in ECMAScript strings, for example the WindowBase64 methods atob()/btoa() methods. It is recommended that new Web applications use Typed Arrays for transmission and storage of binary data.
If binary is selected as the encoding, then step 3 of the steps to decode a byte stream are skipped.
- NOTE: If
"binary"
is specified, byte order marks must be ignored.
The binary decoder is:
- Let byte be byte pointer.
- If <byte> is the EOF byte, emit the EOF code point.
- Increase the byte pointer by one.
- If byte is in the range 0x00 to 0xFF, emit a code point whose value is byte
- Return failure
The binary encoder is:
- Let code point be the code point pointer
- If code point is the EOF code point, emit the EOF byte.
- Increase the code point pointer by one.
- If code point is in the range U+0000 to U+00FF, emit a byte whose value is code point
- Return failure
References
- WebIDL http://dev.w3.org/2006/webapi/WebIDL
- Encoding http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html
Acknowledgements
- Alan Chaney
- Ben Noordhuis
- Glenn Maynard
- John Tamplin
- Kenneth Russell (Google, Inc)
- Robert Mustacchi
- Ryan Dahl
- Anne van Kesteren
Appendix
A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at: