StringEncoding

Proposed String Encoding API for Typed Arrays

Editors

• Joshua Bell (Google, Inc)

Abstract

This specification defines an API for encoding strings to binary data, and decoding strings from binary data.

NOTE: This specification intentionally does not address the opposite scenario of encoding binary data as strings and decoding binary data from strings, for example using Base64 encoding.

Issues

• Align with Blob API http://www.w3.org/TR/FileAPI/#readAsDataText - e.g. handling of invalid code points
• Resolve behavior when writing to a buffer that's too small - partial fill or don't change?

API

Scripts in pages access the API through the top-level window.stringEncoding object which holds methods for encoding/decoding strings. Worker scripts can similarly use the self.stringEncoding object. (Since window and self are the page and worker global object, respectively, scripts can simply refer to stringEncoding without a prefix.)

WebIDL

partial interface Window {
   readonly attribute StringEncoding stringEncoding;
};

partial interface WorkerUtils {
   readonly attribute StringEncoding stringEncoding;
};

The stringEncoding object exposes static methods for encoding and decoding strings from objects containing binary data as specified in the Typed Array specification.

WebIDL

interface [
 OmitConstructor
] StringEncoding {

 DOMString decode(any array,
                  optional unsigned long byteOffset,
                  optional unsigned long byteLength,
                  optional DOMString encoding)
                        raises(DOMException);

 DOMString stringLength(any array,
                        optional unsigned long byteOffset,
                        optional long byteLength,
                        optional DOMString encoding)
                                   raises(DOMException);

 unsigned long encode(DOMString value, 
                      any array,
                      optional unsigned long byteOffset,
                      optional DOMString encoding)
                          raises(DOMException);

 unsigned long encodedLength(DOMString value,
                             optional DOMString encoding)
                                 raises(DOMException);
}

decode

This method decodes a string at the given byteOffset and byteLength, using the specified encoding. The array parameter must be an ArrayBuffer or DataView, otherwise a TypeError exception is raised. If array is a DataView, the byteOffset is in addition to any offset between the DataView and the underlying ArrayBuffer. If the byteOffset parameter is omitted it defaults to 0. If the byteLength parameter is omitted it defaults to the length of the buffer (or view on the buffer), minus the byteOffset. If the encoding parameter is omitted it defaults to UTF-8.

Data is decoded starting at byteOffset. Decoding continues until byteLength bytes have been processed.

NOTE: U+0000 characters have no special meaning and are returned as part of the string.

NOTE: If the encoded string includes a BOM that is considered part of the length. For example, to decode the UTF-16BE sequence 0xFE 0xFF 0x00 0x41 0x00 0x42 0x00 0x43 as the string ABC a byteLength of 8 must be specified.

An exception (TBD) is raised if the method would read past the end of the underlying buffer. If the binary data or data range is not valid according to the specified encoding an exception (TBD) is raised. If the specified encoding is not known, an exception (TBD) is raised. If byteLength is negative and the specified encoding does not support encoding U+0000 an exception (TBD) is raised.

ISSUE: If a BOM is present, is it returned as part of the string? FileReader.readAsText() strips the BOM.

ISSUE: Behavior if byteLength stops inside a multi-byte sequence.

stringLength

This method determines the length of a "null-terminated" string at the given byteOffset up to byteLength bytes, using the specified encoding. The array parameter must be an ArrayBuffer or DataView, otherwise a TypeError exception is raised. If array is a DataView, the byteOffset is in addition to any offset between the DataView and the underlying ArrayBuffer. If the byteOffset parameter is omitted it defaults to 0. If the byteLength parameter is omitted it defaults to the length of the buffer (or view on the buffer), minus the byteOffset. If the encoding parameter is omitted it defaults to UTF-8.

Data is decoded starting at byteOffset. Decoding continues until a U+0000 character is decoded or byteLength bytes have been processed. If a U+0000 character is decoded, the number of bytes processed, not including those representing the U+0000 character, are returned. If byteLength bytes are processed without a U+0000 character decoded, -1 is returned.

NOTE: The byte sequence representing terminator is encoding specific. For example, in UTF-16 encodings it would be the even-aligned two-octet sequence 0x00 0x00

NOTE: If the encoded string includes a BOM that is considered part of the length. For example, stringLength would return a length of 8 for the UTF-16BE sequence 0xFE 0xFF 0x00 0x41 0x00 0x42 0x00 0x43 0x00 0x00.

An exception (TBD) is raised if the method would read past the end of the underlying buffer. If the binary data or data range is not valid according to the specified encoding an exception (TBD) is raised. If the specified encoding is not known, an exception (TBD) is raised. If the specified encoding does not support encoding U+0000 an exception (TBD) is raised.

ISSUE: Add an optional unsigned short terminator member, defaults to 0?

ISSUE: To allow terminators which aren't code points (e.g. 0xFF in UTF-8), make the optional terminator either a code point (default 0) or an Array of octets (e.g. [ 0xFF, 0XFF ] ?

encode

Encodes the string value into the specified array at the given byteOffset, using the specified encoding. The array parameter must be an ArrayBuffer or DataView, otherwise a TypeError exception is raised. If the byteOffset parameter is omitted it defaults to 0. If the encoding parameter is omitted it defaults to UTF-8. If array is a DataView, the byteOffset is in addition to any offset between the DataView and its underlying ArrayBuffer. The return value is the length of the encoded string, in bytes. No "null terminator" is added to the string, although a trailing \x00 character in the string will be encoded if present and if it can be expressed in the specified encoding. An exception (TBD) is raised if the method would write past the end of the underlying buffer. If value cannot be encoded with the specified encoding an exception (TBD) is raised. If the specified encoding is not known, an exception (TBD) is raised.

If an exception is thrown by this method, the target buffer MUST NOT be changed.

ISSUE: Alternately, we could allow "partial fill" and return an object with bytesWritten and charactersWritten properties.

encodedLength

Computes and returns the length, in bytes, of the string value if it were to be encoded using the specified encoding. If the encoding parameter is omitted it defaults to UTF-8. If value cannot be encoded with the specified encoding an exception (TBD) is raised. If the specified encoding is not known, an exception (TBD) is raised.

NOTE: If the encoding includes a BOM, the length of the BOM is included. For example, the string ABC may be encoded in UTF-16 as the octets 0xFE 0xFF 0x00 0x65 0x00 0x66 0x00 0x67 and have a length of 8.

Examples

Example #1 - encoding strings

The following example uses the API to encode an array of strings into a ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32), followed by the length of the first string (as a Uint32), the UTF-8 encoded string data, the length of the second string (as a Uint32), the string data, and so on.

function encodeArrayOfStrings(strings) {
  var len, i, bytes, view, offset;

  len = Uint32Array.BYTES_PER_ELEMENT;
  for (i = 0; i < strings.length; i += 1) {
    len += Uint32Array.BYTES_PER_ELEMENT;
    len += stringEncoding.encodedLength(strings[i], "UTF-8");
  }

  bytes = new Uint8Array(len);
  view = new DataView(bytes.buffer);
  offset = 0;

  view.setUint32(offset, strings.length);
  offset += Uint32Array.BYTES_PER_ELEMENT;
  for (i = 0; i < strings.length; i += 1) {
    len = stringEncoding.encode(strings[i], view,
                                offset + Uint32Array.BYTES_PER_ELEMENT,
                                "UTF-8");
    view.setUint32(offset, len);
    offset += Uint32Array.BYTES_PER_ELEMENT + len;
  }
  return bytes.buffer;
}

Example #2 - decoding strings

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

function decodeArrayOfStrings(buffer) {
  var view, offset, num_strings, strings, i, len;

  view = new DataView(buffer);
  offset = 0;
  strings = [];

  num_strings = view.getUint32(offset);
  offset += Uint32Array.BYTES_PER_ELEMENT;
  for (i = 0; i < num_strings; i += 1) {
    len = view.getUint32(offset);
    offset += Uint32Array.BYTES_PER_ELEMENT;
    strings[i] = stringEncoding.decode(view, offset, 
                                       len, "UTF-8");
    offset += len;
  }
  return strings;
}

Encodings

Encoding names are case-insensitive.

Standard Encodings

Implementations MUST support all of the following encodings:

ISSUE: Should anything be said about canonical forms in the Unicode encodings?

ASCII

• decode: exception thrown if any octet in array is greater than 0x7F
• encode: exception thrown if value string contains a character beyond U+007F

ISO-8859-1

• decode: No encoding-specific exceptions thrown
• encode: exception thrown if value string contains a character beyond U+00FF

BINARY

• decode: No encoding-specific exceptions are thrown
• encode: exception thrown if value string contains a character beyond U+00FF

NOTE: ISO-8859-1 and BINARY are functionally identical in this specification. Both are included so that callers can be more explicit about the type of data being handled. Storing binary data in ECMAScript strings, one byte per character, was a common approach before Typed Array support was available. This "BINARY" encoding allows for easy interoperation with this legacy style of binary storage.

UTF-8

• decode: BOM accepted (0xEF 0xBB 0xBF), exception thrown on invalid/truncated UTF-8 sequence; non-BMP characters in the UTF-8 encoded string yield UTF-16 surrogate pairs in the DOMString. BOM is not returned as part of the DOMString.
• encode: BOM is not written. Exception (TBD) thrown when there is no valid UTF-8 encoding of the string (e.g. "abc\uD800def" which contains a UTF-16 "surrogate half")

ISSUE: Are over-long UTF-8 sequences allowed?

UTF-16

• decode: An exception thrown if BOM not present. BOM is not returned as part of the DOMString.
• encode: outputs a BOM prefix; can be either LE or BE. Implementations may choose to always use the same endianness, or may match the machine architecture for better performance. Callers should not make assumptions about the endianness, and should use the UTF-16BE or UTF-16LE encodings if a specific endianness is desired.

UTF-16LE

• decode: BOM not required, but accepted (0xFF 0xFE); throws if incorrect BOM found or overall length is odd number of bytes. BOM is not returned as part of the DOMString.
• encode: does not write a BOM

ISSUE: throw if invalid surrogate pair encountered?

UTF-16BE

• decode: BOM not required, but accepted (0xFE 0xFF); throws if incorrect BOM found or overall length is odd number of bytes. BOM is not returned as part of the DOMString.
• encode: does not write a BOM

ISSUE: throw if invalid surrogate pair encountered?

Other Encodings

Browsers MAY support additional encodings.

TODO: Suggest other encodings - the suggested default encodings table from http://www.whatwg.org/specs/web-apps/current-work/multipage/parsing.html#encoding-sniffing-algorithm may be prove handy.

Acknowledgements

• Alan Chaney
• Ben Noordhuis
• Glenn Maynard
• John Tamplin
• Kenneth Russell (Google, Inc)
• Robert Mustacchi
• Ryan Dahl

Appendix

A "shim" implementation in JavaScript (that may not fully match the current version of the spec) plus some initial unit tests can be found at:

http://code.google.com/p/stringencoding/