A user account is required in order to edit this wiki, but we've had to disable public user registrations due to spam.
To request an account, ask an autoconfirmed user on Chat (such as one of these permanent autoconfirmed members).
Web Encodings: Difference between revisions
(→Firefox: Updated to Firefox 3.5.2 source; removed aliases that appear in the properties file but are not implemented) |
|||
| Line 320: | Line 320: | ||
=== Firefox === | === Firefox === | ||
==== Matching ==== | |||
ASCII lowercasing. | |||
==== Encodings ==== | |||
{|border=1 cellpadding=4 cellspacing=0 | {|border=1 cellpadding=4 cellspacing=0 | ||
| Line 350: | Line 356: | ||
| EUC-KR | | EUC-KR | ||
| 5601, csksc56011987, csueckr, euc-kr, iso-ir-149, korean, ks_c_5601-1989, ksc5601, ksc_5601 | | 5601, csksc56011987, csueckr, euc-kr, iso-ir-149, korean, ks_c_5601-1989, ksc5601, ksc_5601 | ||
| | | x-windows-949 | ||
| | | | ||
|- | |- | ||
| Line 360: | Line 366: | ||
| GB2312 | | GB2312 | ||
| chinese, csgb2312, csiso58gb231280, gb2312, gb_2312, gb_2312-80, iso-ir-58, zh_cn.euc | | chinese, csgb2312, csiso58gb231280, gb2312, gb_2312, gb_2312-80, iso-ir-58, zh_cn.euc | ||
| | | x-gbk | ||
| | | | ||
|- | |- | ||
| Line 442: | Line 448: | ||
| windows-874 | | windows-874 | ||
| | | | ||
|- | |- | ||
| ISO-8859-13 | | ISO-8859-13 | ||
| Line 566: | Line 567: | ||
| utf-16 | | utf-16 | ||
| | | | ||
| | | Recognized as BE or LE by BOM or byte sniffing | ||
|- | |- | ||
| UTF-16BE | | UTF-16BE | ||
| Line 577: | Line 578: | ||
| | | | ||
| | | | ||
|- | |||
| UTF-32 | |||
| utf-32 | |||
| | |||
| Recognized as BE or LE by BOM or byte sniffing | |||
|- | |- | ||
| UTF-32BE | | UTF-32BE | ||
| Line 597: | Line 603: | ||
| | | | ||
| | | | ||
|- | |- | ||
| VISCII | | VISCII | ||
| Line 755: | Line 756: | ||
| x-mac-ukrainian | | x-mac-ukrainian | ||
| x-mac-ukrainian | | x-mac-ukrainian | ||
| | | | ||
| | | | ||
| Line 782: | Line 768: | ||
| | | | ||
| | | | ||
|- | |- | ||
| x-viet-vps | | x-viet-vps | ||
| Line 799: | Line 780: | ||
|} | |} | ||
Table generated from <http://mxr.mozilla.org/ | Table generated from <http://mxr.mozilla.org/mozilla1.9.1/source/intl/uconv/src/charsetalias.properties> (corresponds to Firefox 3.5.2). | ||
Aliases (used for parsing, apparently not for serialisation) are scattered around in a large number of files; cf. <http://mxr.mozilla.org/firefox/source/intl/uconv/ucvlatin/nsISO885911ToUnicode.cpp> for the mapping from ISO-8859-11 to windows-874. | Aliases (used for parsing, apparently not for serialisation) are scattered around in a large number of files; cf. <http://mxr.mozilla.org/firefox/source/intl/uconv/ucvlatin/nsISO885911ToUnicode.cpp> for the mapping from ISO-8859-11 to windows-874. | ||
Revision as of 15:40, 23 August 2009
My scratchpad for encoding related notes.
Goals
- Document existing practices for
- Supported encodings
- Supported aliases
- Supported matching algorithm
- Converge the various used algorithms
- Get the new rules implemented
Current Implementations
Does this differ per platform? Opera might differ a bit on Mac.
Data
Integrate this awesome data somehow:
- http://coq.no/character-tables/mime/en
- http://coq.no/character-tables/mime/iso-2022/en
- http://coq.no/character-tables/mime/euc/en
- http://coq.no/character-tables/mime/locale-specific/en
Opera
Matching
UTS22 and ASCII lowercasing.
Encodings
| Encoding | Aliases | Decoded As | Notes |
|---|---|---|---|
| big5 | big5, cnbig5, csbig5 | ||
| big5-hkscs | big5hkscs | ||
| euc-jp | cseucpkdfmtjapanese, eucjp, extendedunixcodepackedformatforjapanese | ||
| euc-kr | cseuckr, csksc56011987, euckr, isoir149, korean, ksc5601, ksc56011987, ksc56011989, windows949 | ||
| euc-tw | euctw | ||
| gb18030 | gb18030 | ||
| gbk | chinese, cngb, cp936, csgb2312, csiso58gb231280, euccn, gb2312, gb231280, gbk, isoir58, ms936, windows936 | ||
| hz-gb-2312 | hzgb2312 | ||
| ibm866 | 866, cp866, csibm866, ibm866 | ||
| iso-2022-cn | iso2022cn | ||
| iso-2022-jp | csiso2022jp, iso2022jp | ||
| iso-2022-jp-1 | iso2022jp1 | ||
| iso-2022-kr | csiso2022kr, iso2022kr | ||
| iso-8859-1 | cp819, csisolatin1, ibm819, iso88591, iso885911987, isoir100, l1, latin1 | windows-1252 | |
| iso-8859-2 | csisolatin2, iso88592, iso885921987, isoir101, l2, latin2 | ||
| iso-8859-3 | csisolatin3, iso88593, iso885931988, isoir109, l3, latin3 | ||
| iso-8859-4 | csisolatin4, iso88594, iso885941988, isoir110, l4, latin4 | ||
| iso-8859-5 | csisolatincyrillic, cyrillic, iso88595, iso885951988, isoir144 | ||
| iso-8859-6 | arabic, asmo708, csiso88596e, csisolatinarabic, ecma114, iso88596, iso885961987, iso88596e, isoir127 | ||
| iso-8859-6-i | csiso88596i, iso88596i | ||
| iso-8859-7 | csisolatingreek, ecma118, elot928, greek, greek8, iso88597, iso885971987, isoir126 | ||
| iso-8859-8 | csiso88598e, csisolatinhebrew, hebrew, iso88598, iso885981988, iso88598e, isoir138, visual | ||
| iso-8859-8-i | csiso88598i, iso88598i | ||
| iso-8859-9 | csisolatin5, iso88599, iso885991989, isoir148, l5, latin5 | ||
| iso-8859-10 | csisolatin6, iso885910, iso8859101992, isoir157, l6, latin6 | ||
| iso-8859-11 | iso885911, tis620, tis6202533, windows874 | ||
| iso-8859-13 | iso885913 | ||
| iso-8859-14 | iso885914, iso8859141998, isoceltic, isoir199, l8, latin8 | ||
| iso-8859-15 | iso885915, latin9 | ||
| iso-8859-16 | iso885916, iso8859162001, isoir226, l10, latin10 | ||
| koi8-r | cskoi8r, koi8r | ||
| koi8-u | koi8u | ||
| macintosh | csmacintosh, mac, macintosh, macroman | Likely disabled. | |
| shift_jis | cp932, csshiftjis, cswindows31j, ms932, mskanji, shiftjis, sjis, windows31j | ||
| tcvn | tcvn, viettcvn | ||
| us-ascii | ansix341968, ansix341986, ascii, cp367, csascii, csinvariant, csiso646basic1983, ibm367, invariant, iso646basic1983, iso646irv1991, iso646us, isoir6, ref, us, usascii | windows-1252 | |
| utf-16 | csunicode, csunicode11, csunicodeascii, iso10646j1, iso10646ucs2, iso10646ucsbasic, utf16 | ||
| utf-16be | utf16be | ||
| utf-16le | utf16le | ||
| utf-8 | utf8 | ||
| viscii | csviscii, viscii | ||
| windows-1250 | cp1250, microsoftcp1250, windows1250 | ||
| windows-1251 | cp1251, microsoftcp1251, windows1251 | ||
| windows-1252 | cp1252, microsoftcp1252, windows1252 | ||
| windows-1253 | cp1253, microsoftcp1253, windows1253 | ||
| windows-1254 | cp1254, microsoftcp1254, windows1254 | ||
| windows-1255 | cp1255, microsoftcp1255, windows1255 | ||
| windows-1256 | cp1256, microsoftcp1256, windows1256 | ||
| windows-1257 | cp1257, microsoftcp1257, windows1257 | ||
| windows-1258 | cp1258, microsoftcp1258, windows1258 | ||
| windows-sami-2 | samiws2, windowssami2, ws2 | ||
| x-mac-ce | macce | Likely disabled. | |
| x-mac-cyrillic | maccyrillic | Likely disabled. | |
| x-mac-greek | macgreek | Likely disabled. | |
| x-mac-turkish | macturkish | Likely disabled. | |
| x-vps | vps |
Firefox
Matching
ASCII lowercasing.
Encodings
| Encoding | Aliases | Decoded As | Notes |
|---|---|---|---|
| armscii-8 | armscii-8 | ||
| Big5 | big5, csbig5, x-x-big5, zh_tw-big5 | ||
| Big5-HKSCS | big5-hkscs | ||
| EUC-JP | cseucjpkdfmtjapanese, euc-jp, x-euc-jp | ||
| EUC-KR | 5601, csksc56011987, csueckr, euc-kr, iso-ir-149, korean, ks_c_5601-1989, ksc5601, ksc_5601 | x-windows-949 | |
| gb18030 | gb18030 | ||
| GB2312 | chinese, csgb2312, csiso58gb231280, gb2312, gb_2312, gb_2312-80, iso-ir-58, zh_cn.euc | x-gbk | |
| GEOSTD8 | geostd8 | Does not seem to work. | |
| HZ-GB-2312 | hz-gb-2312 | ||
| IBM850 | 850, cp850, csIBM850, ibm850 | csIBM850 not recognised. | |
| IBM852 | 852, cp852, csIBM852, ibm852 | csIBM852 not recognised. | |
| IBM855 | 855, cp855, csIBM855, ibm855 | csIBM855 not recognised. | |
| IBM857 | 857, cp857, csIBM857, ibm857 | csIBM857 not recognised. | |
| IBM862 | 862, cp862, csIBM862, ibm862 | csIBM862 not recognised. | |
| IBM864 | 864, cp864, csIBM864, ibm-864, ibm864 | csIBM864 not recognised. | |
| IBM864i | 864i, cp864i, csibm864i, ibm-864i, ibm864i | ||
| IBM866 | 866, cp-866, cp866, csIBM866, ibm866 | csIBM866 not recognised. | |
| ISO-2022-CN | iso-2022-cn, iso-2022-cn-ext | ||
| ISO-2022-JP | csiso2022jp, csiso2022jp2, iso-2022-jp, iso-2022-jp-2 | ||
| ISO-2022-KR | csiso2022kr, iso-2022-kr | ||
| ISO-8859-1 | cp819, csisolatin1, ibm819, iso-8859-1, iso-ir-100, iso8859-1, iso88591, iso_8859-1, l1, latin1 | windows-1252 | |
| ISO-8859-10 | csisolatin6, iso-8859-10, iso-ir-157, iso8859-10, iso885910, l6, latin6 | ||
| ISO-8859-11 | iso-8859-11, iso8859-11, iso885911 | windows-874 | |
| ISO-8859-13 | iso-8859-13, iso8859-13, iso885913 | ||
| ISO-8859-14 | iso-8859-14, iso8859-14, iso885914 | ||
| ISO-8859-15 | iso-8859-15, iso8859-15, iso885915, iso_8859-15 | ||
| ISO-8859-16 | iso-8859-16 | ||
| ISO-8859-2 | csisolatin2, iso-8859-2, iso-ir-101, iso8859-2, iso88592, iso_8859-2, l2, latin2 | ||
| ISO-8859-3 | csisolatin3, iso-8859-3, iso-ir-109, iso8859-3, iso88593, iso_8859-3, l3, latin3 | ||
| ISO-8859-4 | csisolatin4, iso-8859-4, iso-ir-110, iso8859-4, iso88594, iso_8859-4, l4, latin4 | ||
| ISO-8859-5 | csisolatincyrillic, cyrillic, iso-8859-5, iso-ir-144, iso8859-5, iso88595, iso_8859-5 | ||
| ISO-8859-6 | arabic, asmo-708, csisolatinarabic, ecma-114, iso-8859-6, iso-ir-127, iso8859-6, iso88596, iso_8859-6 | ||
| ISO-8859-6-E | csiso88596e, iso-8859-6-e | ||
| ISO-8859-6-I | csiso88596i, iso-8859-6-i | ||
| ISO-8859-7 | csisolatingreek, ecma-118, elot_928, greek, greek8, iso-8859-7, iso-ir-126, iso8859-7, iso88597, iso_8859-7, sun_eu_greek | ||
| ISO-8859-8 | csisolatinhebrew, hebrew, iso-8859-8, iso-ir-138, iso8859-8, iso88598, iso_8859-8, visual | ||
| ISO-8859-8-E | csiso88598e, iso-8859-8-e | ||
| ISO-8859-8-I | csiso88598i, iso-8859-8-i, iso-8859-8i | ||
| ISO-8859-9 | csisolatin5, iso-8859-9, iso-ir-148, iso8859-9, iso88599, iso_8859-9, l5, latin5 | ||
| ISO-IR-111 | csiso111ecmacyrillic, ecma-cyrillic, iso-ir-111 | ||
| KOI8-R | koi8-r | ||
| KOI8-U | koi8-u | ||
| Shift_JIS | csshiftjis, ms_kanji, shift-jis, shift_jis, windows-31j, x-sjis | ||
| T.61-8bit | csiso103t618bit, iso-ir-103, t.61, t.61-8bit | ||
| TIS-620 | tis-620, tis620 | windows-874 | |
| us-ascii | 646, ansi_x3.4-1968, ascii, us-ascii | windows-1252 | |
| UTF-16 | utf-16 | Recognized as BE or LE by BOM or byte sniffing | |
| UTF-16BE | csunicode, csunicode11, csunicodeascii, csunicodelatin1, iso-10646, iso-10646-j-1, iso-10646-ucs-2, iso-10646-ucs-basic, iso-10646-unicode-latin1, utf-16be, x-iso-10646-ucs-2-be | ||
| UTF-16LE | utf-16le, x-iso-10646-ucs-2-le | ||
| UTF-32 | utf-32 | Recognized as BE or LE by BOM or byte sniffing | |
| UTF-32BE | iso-10646-ucs-4, utf-32be, x-iso-10646-ucs-4-be | ||
| UTF-32LE | utf-32le, x-iso-10646-ucs-4-le | ||
| UTF-7 | csunicode11utf7, unicode-1-1-utf-7, unicode-2-0-utf-7, utf-7, x-unicode-2-0-utf-7 | ||
| UTF-8 | unicode-1-1-utf-8, utf-8, utf8 | ||
| VISCII | csviscii, viscii | ||
| windows-1250 | cp1250, windows-1250, x-cp1250 | ||
| windows-1251 | ansi-1251, cp1251, windows-1251, x-cp1251 | ||
| windows-1252 | cp1252, windows-1252, x-cp1252 | ||
| windows-1253 | cp1253, windows-1253, x-cp1253 | ||
| windows-1254 | cp1254, windows-1254, x-cp1254 | ||
| windows-1255 | cp1255, windows-1255, x-cp1255 | ||
| windows-1256 | cp1256, windows-1256, x-cp1256 | ||
| windows-1257 | cp1257, windows-1257, x-cp1257 | ||
| windows-1258 | cp1258, windows-1258, x-cp1258 | ||
| windows-874 | ibm874, windows-874 | ||
| windows-936 | windows-936 | ||
| x-euc-tw | cns11643, x-euc-tw, zh_tw-euc | ||
| x-gbk | gbk, x-gbk | ||
| x-imap4-modified-utf7 | x-imap4-modified-utf7 | ||
| x-johab | x-johab | ||
| x-mac-arabic | x-mac-arabic | ||
| x-mac-ce | x-mac-ce | ||
| x-mac-croatian | x-mac-croatian | ||
| x-mac-cyrillic | x-mac-cyrillic | ||
| x-mac-devanagari | x-mac-devanagari | ||
| x-mac-farsi | x-mac-farsi | ||
| x-mac-greek | x-mac-greek | ||
| x-mac-gujarati | x-mac-gujarati | ||
| x-mac-gurmukhi | x-mac-gurmukhi | ||
| x-mac-hebrew | x-mac-hebrew | ||
| x-mac-icelandic | x-mac-icelandic | ||
| x-mac-roman | csMacintosh, mac, macintosh, x-mac-roman | csMacintosh not recognised. | |
| x-mac-romanian | x-mac-romanian | ||
| x-mac-turkish | x-mac-turkish | ||
| x-mac-ukrainian | x-mac-ukrainian | ||
| x-user-defined | x-user-defined | ||
| x-viet-tcvn5712 | x-viet-tcvn5712 | ||
| x-viet-vps | x-viet-vps | ||
| x-windows-949 | ks_c_5601-1987, x-windows-949 |
Table generated from <http://mxr.mozilla.org/mozilla1.9.1/source/intl/uconv/src/charsetalias.properties> (corresponds to Firefox 3.5.2).
Aliases (used for parsing, apparently not for serialisation) are scattered around in a large number of files; cf. <http://mxr.mozilla.org/firefox/source/intl/uconv/ucvlatin/nsISO885911ToUnicode.cpp> for the mapping from ISO-8859-11 to windows-874.
8-bit encodings (excluding UTFs, CJK encodings and T.61) tested using <http://coq.no/X/charset5/tests8bit.html> (fail/pass should not be taken too seriously yet, especially not for more obscure encodings), Firefox version 3.5.1, OS X.
Missing: Mappings for CJK encodings. Information about the encodings found in charsetalias.properties, but apparently not working as expected.
Bugs: Filed <https://bugzilla.mozilla.org/show_bug.cgi?id=512060> for the labels marked 'not recognised' in the table above since the lack of support for these is clearly accidental rather than deliberate (though it seems to suggest that these particular labels are not particularly widely used). In most other cases, research and deliberation will be needed to distinguish between bugs and features.
Chrome
FIXME
Internet Explorer
Needs sorting out:
- http://blogs.msdn.com/shawnste/archive/2009/08/18/alternate-encoding-names-recognized-by-net-ie.aspx
- http://msdn.microsoft.com/en-us/library/system.text.encoding.getencodings.aspx
- http://web.archive.org/web/20080204211015/http://www.hitachi-to.co.jp/prod/prod_2/inter/emk/help/TextEncoder/CodePage.htm
Safari
Apple’s reference <http://developer.apple.com/documentation/Carbon/Conceptual/ProgWithTECM/tecmgr_encnames/tecmgr_encnames.html#//apple_ref/doc/uid/TP40000932-CH204-TPXREF103> is incomplete or at least severely out of date even as a description of the Text Encoding Conversion Manager (TECM).
Querying TECM programmatically gives labels for more encodings, but the set of encodings does not match what Safari currently supports.
According to webkit/WebCore/platform/text/TextCodecICU.cpp, WebKit now uses ICU <http://site.icu-project.org/> with additional aliases (webkit/WebCore/platform/text/TextCodecICU.cpp), additional encodings (webkit/WebCore/platform/text/mac/mac-encodings.txt) possibly implemented using TECM at least on the Mac, a list of official IANA labels (webkit/WebCore/platform/text/mac/character-sets.txt) and probably a few more which I have not noticed.
ICU 4.2’s icu/source/data/mappings/convrtrs.txt or <http://demo.icu-project.org/icu-bin/convexp> lists encodings and labels not supported in Safari 4.0 on Leopard, and webkit/WebCore/platform/text/TextCodecICU.cpp mentions that Tiger included ICU 3.2.
Does encoding support in Safari depend on OS and OS version, and will Safari ultimately support everything in ICU 4.2?