StringEncoding - Revision history

Annevk at 15:16, 27 September 2014

2014-09-27T15:16:35Z

← Older revision		Revision as of 15:16, 27 September 2014
Line 1:		Line 1:
	{{obsolete\|spec=[~~http~~://encoding.spec.whatwg.org/#api Encoding Standard: API]}}		{{obsolete\|spec=[https://encoding.spec.whatwg.org/#api Encoding Standard: API]}}

	Proposed Text Encoding Web API for Typed Arrays		Proposed Text Encoding Web API for Typed Arrays

Annevk: removed Category:Proposals using HotCat

2014-09-27T15:13:11Z

removed Category:Proposals using HotCat

← Older revision		Revision as of 15:13, 27 September 2014
Line 286:		Line 286:

	:http://code.google.com/p/stringencoding/		:http://code.google.com/p/stringencoding/

	~~[[Category:Proposals]]~~

GPHemsley: Point to current spec.

2012-10-23T19:35:01Z

Point to current spec.

← Older revision		Revision as of 19:35, 23 October 2012
Line 1:		Line 1:
	{{obsolete\|~~see~~=[http://encoding.spec.whatwg.org/#api Encoding Standard: API]}}		{{obsolete\|spec=[http://encoding.spec.whatwg.org/#api Encoding Standard: API]}}

	Proposed Text Encoding Web API for Typed Arrays		Proposed Text Encoding Web API for Typed Arrays

GPHemsley: Use new {{obsolete}} template.

2012-10-20T14:32:04Z

Use new {{obsolete}} template.

← Older revision		Revision as of 14:32, 20 October 2012
Line 1:		Line 1:
			{{obsolete\|see=[http://encoding.spec.whatwg.org/#api Encoding Standard: API]}}

	Proposed Text Encoding Web API for Typed Arrays		Proposed Text Encoding Web API for Typed Arrays

	~~= This Document Is Obsolete =~~

	~~<font size="6">See [http://encoding.spec.whatwg.org/#api Encoding Standard - API] instead.</font>~~

	== Editors ==		== Editors ==

Jsbell: Obsoleted

2012-10-19T16:21:04Z

Obsoleted

← Older revision		Revision as of 16:21, 19 October 2012
Line 1:		Line 1:
	Proposed Text Encoding Web API for Typed Arrays		Proposed Text Encoding Web API for Typed Arrays

			= This Document Is Obsolete =

			<font size="6">See [http://encoding.spec.whatwg.org/#api Encoding Standard - API] instead.</font>

	== Editors ==		== Editors ==

Jsbell: Encoding standard URL update

2012-09-21T20:27:32Z

Encoding standard URL update

Jsbell: HTML typo

2012-09-17T21:06:11Z

HTML typo

← Older revision		Revision as of 21:06, 17 September 2012
Line 163:		Line 163:
	#** If <var>offset</var> is greater than <code>0</code>, the bytes in <var>byte stream</var> for positions less than <var>offset</var> are provided by buffer(s) passed in to previous calls to <code>decode()</code>.		#** If <var>offset</var> is greater than <code>0</code>, the bytes in <var>byte stream</var> for positions less than <var>offset</var> are provided by buffer(s) passed in to previous calls to <code>decode()</code>.
	#** The bytes in <var>byte stream</var> for positions <var>offset</var> through <code><var>offset</var> + <var>view.byteLength</var> - 1</code> are provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>.		#** The bytes in <var>byte stream</var> for positions <var>offset</var> through <code><var>offset</var> + <var>view.byteLength</var> - 1</code> are provided by the bytes in <var>view.buffer</var> starting at offset <var>view.byteOffset</var>.
	#** When accessing the byte in <var>byte stream</var> at position code><var>offset</var> + <var>view.byteLength</var></code>:		#** When accessing the byte in <var>byte stream</var> at position <code><var>offset</var> + <var>view.byteLength</var></code>:
	#*** If the internal <var>streaming</var> flag is not set, then yield the '''EOF byte'''.		#*** If the internal <var>streaming</var> flag is not set, then yield the '''EOF byte'''.
	#*** Otherwise, set <var>offset</var> to <code><var>offset</var> + <var>view.byteLength</var></code> and suspend the steps of the decoder algorithm until a subsequent call to <code>decode()</code>		#*** Otherwise, set <var>offset</var> to <code><var>offset</var> + <var>view.byteLength</var></code> and suspend the steps of the decoder algorithm until a subsequent call to <code>decode()</code>

Jsbell: Make BOM handling algorithmic

2012-09-17T21:05:24Z

Make BOM handling algorithmic

← Older revision		Revision as of 21:05, 17 September 2012
Line 159:		Line 159:
	# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var> and set <var>offset</var> to <code>0</code>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.		# If the internal <var>streaming</var> flag of the ''decoder object'' is not set, then reset the <var>encoding algorithm state</var> to the default values for encoding <var>encoding</var> and set <var>offset</var> to <code>0</code>. Otherwise, the <var>encoding algorithm state</var> is re-used from the previous call to <code>decode</code> on this object.
	# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set. Otherwise the internal <var>streaming</var> flag is cleared.		# If the <var>options</var> parameter is specified and the <var>stream</var> option is '''true''', then the internal <var>streaming</var> flag is set. Otherwise the internal <var>streaming</var> flag is cleared.
	# Run or resume the decoder algorithm of the ''decoder object's'' encoder		# Run or resume the decoder algorithm of the ''decoder object's'' encoder, with the following additions:
	#* The input to the algorithm is a <var>byte stream</var>.		#* The input to the algorithm is a <var>byte stream</var>.
	#** If <var>offset</var> is greater than <code>0</code>, the bytes in <var>byte stream</var> for positions less than <var>offset</var> are provided by buffer(s) passed in to previous calls to <code>decode()</code>.		#** If <var>offset</var> is greater than <code>0</code>, the bytes in <var>byte stream</var> for positions less than <var>offset</var> are provided by buffer(s) passed in to previous calls to <code>decode()</code>.
Line 166:		Line 166:
	#*** If the internal <var>streaming</var> flag is not set, then yield the '''EOF byte'''.		#*** If the internal <var>streaming</var> flag is not set, then yield the '''EOF byte'''.
	#*** Otherwise, set <var>offset</var> to <code><var>offset</var> + <var>view.byteLength</var></code> and suspend the steps of the decoder algorithm until a subsequent call to <code>decode()</code>		#*** Otherwise, set <var>offset</var> to <code><var>offset</var> + <var>view.byteLength</var></code> and suspend the steps of the decoder algorithm until a subsequent call to <code>decode()</code>
	#* If ~~the internal~~ <~~var~~>~~useBOM~~</~~var~~> ~~flag is set~~ and <var>offset</var> is <code>0</code> then:		#* If encoding is one of "<code>utf-8</code>", "<code>utf-16</code>" or "<code>utf-16be</code>" and <var>offset</var> is <code>0</code>, then prior to running the steps of the decoder algorithm:
	#** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.		#** If the internal <var>useBOM</var> flag is set, then:
	#** If the next two bytes of the stream are '''0xFF 0xFE''' then set <var>offset</var> to <code>2</code>, and clear the <var>utf-16be</var> flag of the decoder state.		#*** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
	#** If the next two bytes of the stream are '''0xFE 0xFF''' then set <var>offset</var> to <code>2</code>, and set the <var>utf-16be</var> flag of the decoder state.		#*** If the next two bytes of the stream are '''0xFF 0xFE''' then set <var>offset</var> to <code>2</code>, and clear the <var>utf-16be</var> flag of the decoder state.
	#* ~~Subsequent to~~ the ~~above~~ <var>~~useBOM~~</var> ~~test~~ and ~~prior to executing~~ the ~~decoder algorithm, if encoding is one~~ of "<code>~~utf-8~~</code>", "<code>utf-16</code>" ~~or "~~<~~code~~>~~utf-16be~~</~~code~~>" and <var>offset</var> is <code>0</code>~~, then if a matching BOM~~ is ~~present then~~ <~~var~~>~~offset~~</~~var~~> ~~should be advanced past it.~~ If ~~not enough~~ bytes are present in the stream ~~for a match~~ and the internal <var>streaming</var> flag is ~~set~~ suspend this algorithm and return the empty string.		#*** If the next two bytes of the stream are '''0xFE 0xFF''' then set <var>offset</var> to <code>2</code>, and set the <var>utf-16be</var> flag of the decoder state.
			#** If encoding is "<code>utf-8</code>", then:
			#*** If less than three bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
			#*** If the next three bytes of the stream are '''0xEF 0xBB 0xBF''' then set <var>offset</var> to <code>3</code>.
			#** If encoding is "<code>utf-16</code>", then:
			#*** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
			#*** If the next two bytes of the stream are '''0xFF 0xFE''' then set <var>offset</var> to <code>2</code>.
			#** If encoding is "<code>utf-16be</code>", then:
			#*** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
			#*** If the next two bytes of the stream are '''0xFE 0xFF''' then set <var>offset</var> to <code>2</code>.
	#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.		#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
	#* The output of the algorithm is a <var>sequence of emitted code points</var>.		#* The output of the algorithm is a <var>sequence of emitted code points</var>.

Jsbell: /* TextDecoder */

2012-09-13T23:09:13Z

TextDecoder

← Older revision		Revision as of 23:09, 13 September 2012
Line 167:		Line 167:
	#*** Otherwise, set <var>offset</var> to <code><var>offset</var> + <var>view.byteLength</var></code> and suspend the steps of the decoder algorithm until a subsequent call to <code>decode()</code>		#*** Otherwise, set <var>offset</var> to <code><var>offset</var> + <var>view.byteLength</var></code> and suspend the steps of the decoder algorithm until a subsequent call to <code>decode()</code>
	#* If the internal <var>useBOM</var> flag is set and <var>offset</var> is <code>0</code> then:		#* If the internal <var>useBOM</var> flag is set and <var>offset</var> is <code>0</code> then:
	#** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend ~~the decoding~~ algorithm and return the empty string.		#** If less than two bytes are present in the stream and the internal <var>streaming</var> flag is true, suspend this algorithm and return the empty string.
	#** If the next two bytes of the stream are '''0xFF 0xFE''' then set <var>offset</var> to <code>2</code>, and clear the <var>utf-16be</var> flag of the decoder state.		#** If the next two bytes of the stream are '''0xFF 0xFE''' then set <var>offset</var> to <code>2</code>, and clear the <var>utf-16be</var> flag of the decoder state.
	#** If the next two bytes of the stream are '''0xFE 0xFF''' then set <var>offset</var> to <code>2</code>, and set the <var>utf-16be</var> flag of the decoder state.		#** If the next two bytes of the stream are '''0xFE 0xFF''' then set <var>offset</var> to <code>2</code>, and set the <var>utf-16be</var> flag of the decoder state.
			#* Subsequent to the above <var>useBOM</var> test and prior to executing the decoder algorithm, if encoding is one of "<code>utf-8</code>", "<code>utf-16</code>" or "<code>utf-16be</code>" and <var>offset</var> is <code>0</code>, then if a matching BOM is present then <var>offset</var> should be advanced past it. If not enough bytes are present in the stream for a match and the internal <var>streaming</var> flag is set suspend this algorithm and return the empty string.
	#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.		#* If the internal <var>fatal</var> flag of the ''decoder object'' is set, then a '''decoder error''' causes an <code>DOMException</code> of type <code>EncodingError</code> to be thrown rather than emitting a fallback code point.
	#* The output of the algorithm is a <var>sequence of emitted code points</var>.		#* The output of the algorithm is a <var>sequence of emitted code points</var>.

Jsbell at 23:01, 13 September 2012

2012-09-13T23:01:22Z

← Older revision		Revision as of 23:01, 13 September 2012
Line 16:		Line 16:

	== Open Issues ==		== Open Issues ==

	* The [http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html Encoding specification] defines the byte order mark as more authoritative than an explicit encoding label when decoding content.
	** Is that desirable here?
	** What does it mean to e.g. write <code>TextDecoder('iso-2022-kr').decode(str)</code> if string turns out to be UTF-8?
	** What about streaming and other re-uses of the same decoder object?
	** At the very least, a matching BOM should be ignored (i.e. not returned as part of the output) and a mismatching BOM should signal an error.

	* Should the <var>encoding</var> attribute return the <var>Name</var> of the encoding or the name that was passed in?		* Should the <var>encoding</var> attribute return the <var>Name</var> of the encoding or the name that was passed in?
Line 63:		Line 57:
	<dt>What do to on encoding errors? If non-UTF encodings are supported then we may want to allow substitution (e.g. ASCII '?') or a script callback (for arbitrary escaping).		<dt>What do to on encoding errors? If non-UTF encodings are supported then we may want to allow substitution (e.g. ASCII '?') or a script callback (for arbitrary escaping).
	<dd>Resolution: not for "v1" (see above)		<dd>Resolution: not for "v1" (see above)

			<dt>How are byte order marks handled?
			<dd>BOM is respected if and only if the requested coding is a case-insensitive match for "<code>utf-16</code>"

	</dl>		</dl>

← Older revision		Revision as of 20:27, 21 September 2012
Line 82:		Line 82:
	The constructor runs the following steps:		The constructor runs the following steps:
	# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.		# If the constructor is called with no arguments, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
	# Run the '''steps to get an encoding''' from [http://~~dvcs~~.w3.org/~~hg/encoding/raw-file/tip/Overview.html~~ Encoding], with <var>label</var> as <var>label</var>.		# Run the '''steps to get an encoding''' from [http://encoding.spec.whatwg.org/ Encoding], with <var>label</var> as <var>label</var>.
	#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.		#* If the steps result in failure, throw an "<code>EncodingError</code>" exception and terminate these steps.
	#* Otherwise, if the <var>Name</var> of the returned encoding is not one of "<code>utf-8</code>", "<code>utf-16</code>", or "<code>utf-16be</code>" throw an "<code>EncodingError</code>" exception and terminate these steps.		#* Otherwise, if the <var>Name</var> of the returned encoding is not one of "<code>utf-8</code>", "<code>utf-16</code>", or "<code>utf-16be</code>" throw an "<code>EncodingError</code>" exception and terminate these steps.
Line 91:		Line 91:
	<dl>		<dl>
	<dt><code>encoding</code> of type DOMString, readonly		<dt><code>encoding</code> of type DOMString, readonly
	<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://~~dvcs~~.w3.org/~~hg/encoding/raw-file/tip/Overview.html~~ Encoding].		<dd>Returns the <var>Name</var> of the encoder object's <var>encoding</var>, per [http://encoding.spec.whatwg.org/ Encoding].

	:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.		:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''encoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
Line 137:		Line 137:
	# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.		# If called without an <var>encoding</var> argument, let <var>label</var> be the "<code>utf-8</code>". Otherwise, let <var>label</var> be the value of the <var>encoding</var> argument.
	# If <var>label</var> is a case-insensitive match for "<code>utf-16</code>" then set the internal <var>useBOM</var> flag.		# If <var>label</var> is a case-insensitive match for "<code>utf-16</code>" then set the internal <var>useBOM</var> flag.
	# Run the '''steps to get an encoding''' from [http://~~dvcs~~.w3.org/~~hg/encoding/raw-file/tip/Overview.html~~ Encoding], with <var>label</var> as <var>label</var>.		# Run the '''steps to get an encoding''' from [http://encoding.spec.whatwg.org/ Encoding], with <var>label</var> as <var>label</var>.
	## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.		## If the steps result in failure, throw a "<code>EncodingError</code>" exception and terminate these steps.
	## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.		## Otherwise, set the ''decoder object's'' internal <var>encoding</var> property to the returned encoding.
Line 147:		Line 147:
	<dt><code>encoding</code> of type DOMString, readonly		<dt><code>encoding</code> of type DOMString, readonly
	<dd>		<dd>
	Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://~~dvcs~~.w3.org/~~hg/encoding/raw-file/tip/Overview.html~~ Encoding].		Returns the <var>Name</var> of the decoder object's <var>encoding</var>, per [http://encoding.spec.whatwg.org/ Encoding].

	:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.		:Note that this may differ from the name of the encoding specified during the call to the constructor. For example, if the constructor is called with <var>encoding</var> of <code>"ascii"</code> the <var>encoding</var> attribute of the ''decoder object'' would have the value <code>"windows-1252"</code> as <code>"ascii"</code> is a label for that encoding.
Line 251:		Line 251:
	== Encodings ==		== Encodings ==

	Encodings are defined and implemented per [http://~~dvcs~~.w3.org/~~hg/encoding/raw-file/tip/Overview.html~~ Encoding]. This implicitly includes the steps to <em>get an encoding</em> from a string, and the logic for label matching and case-insensitivity.		Encodings are defined and implemented per [http://encoding.spec.whatwg.org/ Encoding]. This implicitly includes the steps to <em>get an encoding</em> from a string, and the logic for label matching and case-insensitivity.

	User agents MUST NOT support any other encodings or labels than those defined in [http://~~dvcs~~.w3.org/~~hg/encoding/raw-file/tip/Overview.html~~ Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.		User agents MUST NOT support any other encodings or labels than those defined in [http://encoding.spec.whatwg.org/ Encoding], and MUST support all encodings and labels defined in that specification, with the additions defined below.

	:''NOTE: In [http://~~dvcs~~.w3.org/~~hg/encoding/raw-file/tip/Overview.html~~ Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''		:''NOTE: In [http://encoding.spec.whatwg.org/ Encoding], "ascii" is a label for '''windows-1252'''; there is no 7-bit-or-raise-exception encoding. Applications that are required to restrict the content of decoded strings should implement validation after decoding.''

	:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''		:''NOTE: Unicode normalization forms are outside the scope of this specification. No normalization is done prior to encoding or after decoding.''

	:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://~~dvcs~~.w3.org/~~hg/encoding/raw-file/tip/Overview.html~~ Encoding].''		:''NOTE: Handling of encoding-specific issues, e.g. over-long UTF-8 encodings, byte order marks, unmatched surrogate pairs, and so on is defined by [http://encoding.spec.whatwg.org/ Encoding].''

	== References ==		== References ==

	* WebIDL http://dev.w3.org/2006/webapi/WebIDL		* WebIDL http://dev.w3.org/2006/webapi/WebIDL
	* Encoding http://~~dvcs~~.w3.org/~~hg/encoding/raw-file/tip/Overview.html~~		* Encoding http://encoding.spec.whatwg.org/