URL: Difference between revisions

Latest revision as of 15:31, 20 June 2015

The contents of this page, URL, and all edits made to this page in its history, are hereby released under the CC0 Public Domain Dedication, as described in WHATWG Wiki:Copyrights.

This documents research and notes around URLs for the URL standard.

Implementations

Tests

https://github.com/cweb/iri-tests

Variants of the following code (runs in Live DOM Viewer) are useful to test which code points are URL escaped in browsers:

<!DOCTYPE html>
<script>
var a = document.createElement("a")

i = 0
cp = 0x100

while ( i < cp ) {
  a.href = "http://x" + String.fromCharCode(i) + "@x/"
  if(a.href.length != "http://x)@x/".length) {
    w(a.href)
  }
  i++
}
</script>

Parsing

https://github.com/annevk/url/blob/master/url.js
http://lists.w3.org/Archives/Public/public-whatwg-archive/2012Sep/0305.html has notes on file URLs in Gecko.

JavaScript libraries

For improving the API we might want to take inspiration from:

Schemes

Apart from the scheme-types listed below, the URL Standard identifies "relative schemes", used for parsing a URL into a parsed URL.

Purpose-specific schemes

URL schemes are purpose-specific schemes if they only work in one context. These only work for WebSocket:

ws
wss

Fetch schemes

URL schemes are resource schemes if fetching the URL results in either a network error or a resource with associated MIME type (potentially sniffed).

ftp
http
https: These all can be used by the corresponding protocol directly.
file: Needs platform-specific interpretation and mapping to a resource on a the local file system.
data: Needs its resource and MIME type information retrieved from its scheme data/query.
blob
about: The resource is effectively the result of passing scheme data to a hash table (not sure if case-sensitive or not; definitely no percent decoding). Query and fragment can be used by the resource.

(The same-origin definition should maybe account for about/blob/data.)

Navigate schemes

The "fetch schemes" -> use "fetch"
javascript
Not the "purpose-specific schemes" -> error
All other schemes (including "external schemes")

External schemes

Depending on the context, schemes not listed above will either launch an external application or result in a network error. Examples:

mailto
skype

IDNA

Definitions

IDNA2003+: IDNA2003 with Unicode updated to the latest version. (So not NFKC from Unicode 3.2., although Python might do that... ) Restrictions on display might be in place.
IDNA2008+: IDNA2008 with RFC 5895 section 2 mapping and IDNA2003 domain label separators. Display is restricted to IDNA2008, lookup is unrestricted (everything gets Punycoded).

Implementations

IDNA2003+: Safari, Chrome, Firefox
- Changing, see e.g. https://codereview.chromium.org/23642003#msg4
IDNA2008+: Internet Explorer?

Tests

http://mathias.html5.org/tests/url/idna2003-separators/ IDNA2003 domain label separators are supported everywhere

Algorithms

ToLabels(domain string) -> ASCII-label list (empty label at the end signifies trailing dot) or failure.
ToASCII(Unicode-label) -> ASCII-label.
ToUnicode(ASCII-label) -> Unicode label.

(For convenience maybe ToASCII and ToUnicode should accept lists too.)

UI

Note that this has potential security implications too, but does not matter for interoperability.

http://www.chromium.org/developers/design-documents/idn-in-google-chrome (also includes summary for other browsers)
https://wiki.mozilla.org/IDN_Display_Algorithm
http://www.alvestrand.no/pipermail/idna-update/2011-December/date.html (has lots of background discussion)
https://bugs.webkit.org/show_bug.cgi?id=126627

Notes

Input to DNS is a byte array. (This means that "_" and byte 0x03 can be valid input. Not sure whether "." works within a label. Higher than 0x7F cannot happen if IDNA is used.)
DNS is of course not the only system in place, but browsers do not seem to care as far as mapping is concerned.
http://www.unicode.org/mail-arch/unicode-ml/y2011-m07/0036.html http://www.unicode.org/mail-arch/unicode-ml/y2011-m07/0057.html
http://tools.ietf.org/html/rfc6055 has historical deliberations

Requests and Issues

How to compare URLs

There is at least one specification (being implemented) that needs a reference for how to compare URLs for equivalency.

http://microformats.org/wiki/representative-hcard-parsing

Hence, request:

New section/feature in the URL spec:

"How to compare URLs"

with something like: "parse them first and then compare the serialization"

See: http://krijnhoetmer.nl/irc-logs/whatwg/20141006#l-843 for background / discussion.

@@ Line 1: / Line 1: @@
-This documents research and notes around the [http://dvcs.w3.org/hg/url/raw-file/tip/Overview.html URL specification].
+{{CC0 page}}
+This documents research and notes around URLs for the [http://url.spec.whatwg.org/ URL standard].
 ==Implementations==
@@ Line 7: / Line 9: @@
 * http://trac.webkit.org/browser/trunk/Source/WebCore/platform/KURLGoogle.cpp
 * http://trac.webkit.org/browser/trunk/Source/WebCore/platform/network/DataURL.cpp (data URLs)
+* http://mxr.mozilla.org/mozilla-central/source/netwerk/base/src/nsStandardURL.cpp
+* http://mxr.mozilla.org/mozilla-central/source/dom/src/jsurl/nsJSProtocolHandler.cpp (javascript URLs)
+* http://mxr.mozilla.org/mozilla-central/source/nsprpub/pr/src/misc/prnetdb.c#1544 (IPv6)
+* https://code.google.com/p/chromium/codesearch
+==Tests==
+* https://github.com/cweb/iri-tests
+Variants of the following code (runs in Live DOM Viewer) are useful to test which code points are URL escaped in browsers:
+<pre><!DOCTYPE html>
+<script>
+var a = document.createElement("a")
-==Model==
+i = 0
+cp = 0x100
- URL (.href)
+while ( i < cp ) {
- - invalid?
+  a.href = "http://x" + String.fromCharCode(i) + "@x/"
- - scheme (.protocol)
+  if(a.href.length != "http://x)@x/".length) {
- - authority
+    w(a.href)
-   - username (proposed .username)
+  }
-   - password (proposed .password)
+  i++
-   - ip/host (.hostname)
+}
-   - port (.port)
+</script></pre>
- - path (.pathname)
- - query (.search)
- - fragment (.hash)
 ==Parsing==
- parse (urlstr, optional baseURL)
+* https://github.com/annevk/url/blob/master/url.js
-  url = new URL
+* http://lists.w3.org/Archives/Public/public-whatwg-archive/2012Sep/0305.html has notes on file URLs in Gecko.
-  tokenize(urlstr)
+==JavaScript libraries==
-  SCHEME CHECK START
-    if char is in ALPHA
+For improving the API we might want to take inspiration from:
-      buffer += char
-      -> SCHEME CHECK NEXT
+* http://medialize.github.com/URI.js/
-    else
+* https://github.com/joyent/node/blob/master/doc/api/url.markdown
-      unconsume char
+* https://github.com/bestiejs/punycode.js (just Punycode)
-      -> NO SCHEME
+==Schemes==
-  SCHEME CHECK NEXT
-    if char is in ALPHA / DIGIT / "+" / "-" / "."
+Apart from the scheme-types listed below, the URL Standard identifies "relative schemes", used for parsing a URL into a parsed URL.
-      buffer += char
-      -> continue
+=== Purpose-specific schemes ===
-    elif char is ":"
-      url.scheme = buffer.toASCIILowercase()
+URL schemes are purpose-specific schemes if they only work in one context. These only work for WebSocket:
-      buffer = ""
-      -> SCHEME
+* ws
-    else:
+* wss
-      input.reset()
-      -> NO SCHEME
+=== Fetch schemes ===
-  SCHEME
+URL schemes are resource schemes if fetching the URL results in either a network error or a resource with associated MIME type (potentially sniffed).
-    if url.scheme is not hierarchical (data:)
-      -> NON-HIERARCHICAL
+; ftp
-    elif baseURL and url.scheme is baseURL.scheme (http:?test)
+; http
-      -> RELATIVE
+; https : These all can be used by the corresponding protocol directly.
-    else  (https://test.com/)
+; file : Needs platform-specific interpretation and mapping to a resource on a the local file system.
-      -> AUTHORITY START
+; data : Needs its resource and MIME type information retrieved from its scheme data/query.
+; blob
-  NO SCHEME
+; about : The resource is effectively the result of passing scheme data to a hash table (not sure if case-sensitive or not; definitely no percent decoding). Query and fragment can be used by the resource.
-    if not baseURL or baseURL.scheme is not hierarchical
-      url.invalid = true
+(The same-origin definition should maybe account for about/blob/data.)
-      return url
-    else
+=== Navigate schemes ===
-      -> RELATIVE
+* The "fetch schemes" -> use "fetch"
-  NON-HIERARCHICAL (could merge with PATH)
+* javascript
-    if curChar is "#"
+* Not the "purpose-specific schemes" -> error
-      FRAGMENT
+* All other schemes (including "external schemes")
-    else
-      ...
+=== External schemes ===
-  RELATIVE
+Depending on the context, schemes not listed above will either launch an external application or result in a network error. Examples:
-    if char is EOI (end-of-input)
-      url = baseURL
+* mailto
-      url.fragment = null
+* skype
-      exit
+==IDNA==
-    elif char is "/" or char is "\"
-      if next char "/" or next char is "\"
+=== Definitions ===
-        url.scheme = baseURL.scheme
-        -> AUTHORITY START
+* IDNA2003+: IDNA2003 with Unicode updated to the latest version. (So not NFKC from Unicode 3.2., although [http://docs.python.org/2/library/unicodedata.html#unicodedata.ucd_3_2_0 Python might do that]... ) Restrictions on display might be in place.
-      else
+* IDNA2008+: IDNA2008 with [http://tools.ietf.org/html/rfc5895#section-2 RFC 5895 section 2] mapping and IDNA2003 domain label separators. Display is restricted to IDNA2008, lookup is unrestricted (everything gets Punycoded).
-        url.scheme = baseURL.scheme
-        url.authority = baseURL.authority
+=== Implementations ===
-        -> PATH
+* IDNA2003+: Safari, Chrome, Firefox
-    elif char is "?"
+** Changing, see e.g. https://codereview.chromium.org/23642003#msg4
-        url.scheme = baseURL.scheme
+* IDNA2008+: Internet Explorer?
-        url.authority = baseURL.authority
-        url.path = baseURL.path
+=== Tests ===
-        -> QUERY
+* http://mathias.html5.org/tests/url/idna2003-separators/ IDNA2003 domain label separators are supported everywhere
-    elif char is "#"
-        url.scheme = baseURL.scheme
+=== Algorithms ===
-        url.authority = baseURL.authority
-        url.path = baseURL.path
+* ToLabels(domain string) -> ASCII-label list (empty label at the end signifies trailing dot) or failure.
-        url.query = baseURL.query
+* ToASCII(Unicode-label) -> ASCII-label.
-        -> FRAGMENT
+* ToUnicode(ASCII-label) -> Unicode label.
-    else
+(For convenience maybe ToASCII and ToUnicode should accept lists too.)
-      url.scheme = baseURL.scheme
-      url.authority = baseURL.authority
+=== UI ===
-      prepend input by baseURL.path up to the last /
-      -> PATH
+Note that this has potential security implications too, but does not matter for interoperability.
-  AUTHORITY START
+* http://www.chromium.org/developers/design-documents/idn-in-google-chrome (also includes summary for other browsers)
-    if char is "/" or char is "\"
+* https://wiki.mozilla.org/IDN_Display_Algorithm
-      -> continue
+* http://www.alvestrand.no/pipermail/idna-update/2011-December/date.html (has lots of background discussion)
-    else
+* https://bugs.webkit.org/show_bug.cgi?id=126627
-      -> AUTHORITY
+=== Notes ===
-  AUTHORITY
-    ...
+* Input to DNS is a byte array. (This means that "_" and byte 0x03 can be valid input. Not sure whether "." works within a label. Higher than 0x7F cannot happen if IDNA is used.)
+* DNS is of course not the only system in place, but browsers do not seem to care as far as mapping is concerned.
-  PATH
+* http://www.unicode.org/mail-arch/unicode-ml/y2011-m07/0036.html http://www.unicode.org/mail-arch/unicode-ml/y2011-m07/0057.html
-    if char is "?"
+* http://tools.ietf.org/html/rfc6055 has historical deliberations
-      -> QUERY
-    if char is "#"
-      -> FRAGMENT
-    else
-      buffer += char
-  QUERY
-    if char is "#"
-      -> FRAGMENT
-  FRAGMENT
-    ...
 [[Category:Spec coordination]]
+== Requests and Issues ==
+=== How to compare URLs ===
+There is at least one specification (being implemented) that needs a reference for how to compare URLs for equivalency.
+* http://microformats.org/wiki/representative-hcard-parsing
+Hence, request:
+New section/feature in the URL spec:
+"How to compare URLs"
+with something like: "parse them first and then compare the serialization"
+See: http://krijnhoetmer.nl/irc-logs/whatwg/20141006#l-843
+for background / discussion.

URL: Difference between revisions

Latest revision as of 15:31, 20 June 2015

Contents

Implementations

Tests

Parsing

JavaScript libraries

Schemes

Purpose-specific schemes

Fetch schemes

Navigate schemes

External schemes

IDNA

Definitions

Implementations

Tests

Algorithms

UI

Notes

Requests and Issues

How to compare URLs

Navigation menu

URL: Difference between revisions

Latest revision as of 15:31, 20 June 2015

Implementations

Tests

Parsing

JavaScript libraries

Schemes

Purpose-specific schemes

Fetch schemes

Navigate schemes

External schemes

IDNA

Definitions

Implementations

Tests

Algorithms

UI

Notes

Requests and Issues

How to compare URLs

Navigation menu

Search