A user account is required in order to edit this wiki, but we've had to disable public user registrations due to spam.

To request an account, ask an autoconfirmed user on IRC (such as one of these permanent autoconfirmed members).

Difference between revisions of "Fetch"

From WHATWG Wiki
Jump to: navigation, search
(Response)
(Pseudo-code: this is no longer accurate, but lets keep it in sync for now until the HTML is up)
 
(15 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
{{CC0 page}}
 
== Plan ==
 
== Plan ==
  
Fetch is [http://fetch.spec.whatwg.org/ fetch.spec.whatwg.org] and will define [http://www.whatwg.org/specs/web-apps/current-work/multipage/fetching-resources.html#fetch HTML fetch] and CORS as a set of coherent algorithms rather than the intertwined mess we have now. It will also deal with the following:
+
Fetch is [http://fetch.spec.whatwg.org/ fetch.spec.whatwg.org] and will define [http://www.whatwg.org/specs/web-apps/current-work/multipage/fetching-resources.html#fetch HTML fetch] and CORS as a set of coherent algorithms rather than the intertwined mess we have now.
  
* Deal with authentication (URLs containing username/password, servers responding with 401)
+
== Goals ==
* Deal with URL processing
+
 
* Define HTTP context for data: URLs, about:blank, and file: URLs.
+
* Centralize redirect handling
* Progress Events
+
* Centralize CORS
 +
* Centralize fetching and thereby consistify handling of e.g. data URLs across APIs
 +
* Allow resources to opt into CORS without the API asking for it (stop requiring a crossorigin attribute, remove "No CORS" mode)
  
 
== Model ==
 
== Model ==
Line 34: Line 37:
 
=== Basic Fetch ===
 
=== Basic Fetch ===
  
See also [[URL]]. Fetch depends on the URL scheme. Unless it is a scheme listed below the response is a network error.
+
See also [[URL]]. Fetch specifics depends on the URL scheme. Where possible fetching happens incrementally. If it completely fails you get a network error. Otherwise a response. A response might be exposed from the moment status/headers are available and the entity body is still loading.
  
 
==== about ====
 
==== about ====
Line 46: Line 49:
 
==== data ====
 
==== data ====
  
Response object as per http://xhr.spec.whatwg.org/#data:-urls-and-http except that failing to parse needs to be added.
+
Response object as per http://xhr.spec.whatwg.org/#data:-urls-and-http
 +
 
 +
If the data URL fails to parse however return a network error.
  
 
==== file ====
 
==== file ====
Line 58: Line 63:
 
==== http and https ====
 
==== http and https ====
  
Construct the request per the HTTP specification, do the request, and expose the response.
+
* Construct the request per the HTTP specification
 +
* Do the request
 +
* Expose the response.
 +
 
 +
==== any other scheme ====
 +
 
 +
Network error.
  
 
=== Fetch ===
 
=== Fetch ===
  
Fetch defers to Basic Fetch except for http and https. It has the ability to handle redirects automatically (and somehow hide this redirecting happening from the caller). And deals with more complicated preflight requests if CORS is opted into and if CORS is not opted into it still examines the response to see if it can be marked CORS same-origin after all.
+
Basic Fetch deals with a single request/response whereas Fetch tackles the more complicated setups required by http/https:
  
So basically Basic Fetch deals with a single request/response whereas Fetch tackles the more complicated setups required by most platform APIs.
+
* Redirects
 +
** We want to follow them if the status code is 301, 302, 303, 307, 308 and there's a Location header (and if there's more than one Location header we need to pick probably), unless a flag is set to not follow redirects).
 +
** We need to carefully consider what happens if the new Location is not same-origin and especially what if it's not http or https. That might depend on the API.
 +
* Authentication (challenge request, prompt the user for certain legacy APIs)
 +
* CORS (preflight requests, marking the response as same-origin or not)
 +
* Cookies
 +
** Include them in the request or not?
 +
** Deal with cookies in the response (before handling redirects and such) and the storage mutex.
 +
* 304 needs to be turned into a 200 for XMLHttpRequest unless explicitly requested otherwise. We should make it clearer how that works.
 +
 
 +
For everything else Fetch will simply defer to Basic Fetch, but Fetch will be the algorithm used by the whole platform.
  
 
=== Response ===
 
=== Response ===
Line 77: Line 98:
 
* Entity body
 
* Entity body
  
== Notes ==
+
== User agent advice ==
 +
 
 +
For http/https, maybe give advice on proxies, e.g.: "If the user agent allows the end user to configure a proxy it should modify the request appropriately; i.e., connect to the proxy host instead of the origin server, modify the Request-Line and send Proxy-Authorization headers as specified." Though ideally HTTP does this.
 +
 
 +
For http/https, advice to not include to much random headers, even though it'll happen to some extent anyway.
 +
 
 +
== Pseudo-code ==
 +
 
 +
  // "No CORS" means no Origin header, no CORS checking, mark as "cross-origin", unless
 +
  // no_tainting is defined, in which case NetworkError()
 +
  // "Anonymous" means omit_credentials, normal CORS stuff; should probably mean no
 +
  // Referer and globally unique identifier for Origin
 +
  // "Use Credentials" means normal CORS stuff
 +
 
 +
  class Request:
 +
    url
 +
    origin              // currently "source origin" in some specs
 +
    referrer_source
 +
    method              // e.g. GET
 +
    author_headers = []
 +
    ua_headers = []
 +
    headers = []        // unholy union of author_headers and ua_headers
 +
    entity_body
 +
    force_preflight    // CORS prefligth required
 +
    force_same_origin  // HTML uses this for XMLDocument.load() and Workers
 +
    cors                // True / False for now
 +
    omit_credentials    // set to True to omit cookies for cross-origin requests (HTTP authentication?)
 +
    tainted
 +
    no_tainting        // set to True to terminate in a "network_error" when tainted is set
 +
                        // XMLHttpRequest wants this, <track> too apparently
 +
    redirect_count      // to prevent loops just terminate when this is 10 or so
 +
 
 +
  class Response:
 +
    type                // "", "redirect", "network_error", "abort_error", "response"
 +
    location            // if type == "redirect", a URL, None otherwise
 +
    status_code
 +
    status_text
 +
    headers
 +
    headers_exposed    // not all headers can be exposed to web-facing APIs
 +
    entity_body
 +
    cors                // "same-origin" or "cross-origin"
 +
 
 +
  class NetworkError(Response):
 +
    ... state is "done", type is "network_error"
 +
 
 +
  class Notifier:
 +
    taint = False
 +
    response = Response | None
 +
    def handle_response(response):
 +
      if taint:
 +
        response.cors = "cross-origin"
 +
    def handle_data(data):
 +
      response.feed(data)
 +
    def notifier_wait_for_response:
 +
      // magic dust
 +
      return response
 +
 
 +
  class Fetch:
 +
    request = Request
 +
    notifier = Notifier
 +
 
 +
    def fetch():
 +
      url = request.url
 +
      origin = request.origin
 +
      if url.origin == origin or url.scheme in ("about", "blob", "data"):
 +
        redirect_fetch()
 +
      elif request.force_same_origin or (not request.cors and request.no_tainting):
 +
        error()
 +
      elif not request.cors:
 +
        request.tainted = True
 +
        redirect_fetch() // url.scheme == "file" ends up here
 +
      elif url.scheme not in ("http", "https"):
 +
        error()
 +
      elif not requires_cors_with_preflight():
 +
        cors_fetch()
 +
      else:
 +
        cors_fetch_with_preflight()
 +
      if request.synchronous:
 +
        return notifier.wait_for_response()
 +
 
 +
    def basic_fetch():
 +
      url = request.url
 +
      if url.scheme == "about":
 +
        if url.scheme_data == "blank":
 +
          return Response({type:"text/html", encoding:"utf-8", body:""})
 +
        else:
 +
          return NetworkError()
 +
      elif url.scheme == "blob":
 +
        ...
 +
      elif url.scheme == "data":
 +
        ...
 +
      elif url.scheme == "file":
 +
        ...
 +
      elif url.scheme == "ftp":
 +
        ...
 +
      elif url.scheme in ("http", "https"):
 +
        ... count redirects
 +
      else:
 +
        return NetworkError()
 +
 
 +
    def redirect_fetch(request):
 +
      response = basic_fetch(request)
 +
      response.wait_for_headers() // design something better
 +
      if response.type == "redirect":
 +
        request.url = response.location
 +
        return fetch(request)
 +
      else:
 +
        return response
 +
 
 +
    def requires_cors_with_preflight(request):
 +
      if request.force_preflight:
 +
        return True
 +
      if request.method not in cors_simple_methods:
 +
        return True
 +
      for header in request.author_headers:
 +
        if not is_cors_simple_header(header):
 +
          return True
 +
 
 +
    def cors_simple_header(header):
 +
      ...
 +
 
 +
    def cors_fetch(request):
 +
      response = basic_fetch(request)
 +
      response.wait_for_headers()
 +
      if response.type == "redirect":
 +
        if request.url.origin != response.location.origin:
 +
          request.origin = uuid()
 +
        request.url = response.location
 +
        if request.url.username or request.url.password or not cors_check(response):
 +
          response.type = "network_error"
 +
          return response
 +
        return cors_fetch(request)
 +
      // XXX should this succeed for about: / blob: data: (think redirects)
 +
      elif response.type == "response" and not cors_check(response):
 +
        response.type = "network_error
 +
      return response
 +
 
 +
    def cors_fetch_with_preflight(request):
 +
      if not in_cors_preflight_cache(request):
 +
        result = cors_preflight_fetch(request)
 +
        if result.type in ("abort_error", "network_error"):
 +
          return result
 +
      response = basic_fetch(request)
 +
      response.wait_for_headers()
 +
      if response.type in ("redirect", "network_error") or (response.type == "response" and not cors_check(response)):
 +
        clear_cors_preflight_cache(request)
 +
        response.type = "network_error"
 +
      return response
  
Algorithm: "fetch"
+
    def cors_preflight_fetch(request):
 +
      preflight_request = Request()
 +
      preflight_request.url = request.url
 +
      preflight_request.origin = request.origin
 +
      preflight_request.referrer_source = request.referrer_source
 +
      preflight_request.method = OPTIONS
 +
      // block cookies?
 +
      preflight_request.ua_headers.append(Access-Control-Request-Method, request.method)
 +
      preflight_request.ua_headers.append(Access-Control-Request-Headers, request.author_headers
 +
     
 +
      response = basic_fetch(preflight_request)
 +
      if response.status_code >= 200 and response.status_code < 300:
 +
        more checks
 +
      elif response.type != "abort_error":
 +
        response.type = "network_error"
 +
      return response
  
* Sets up the request. Methods, headers, etc.
+
    def in_cors_preflight_cache(request):
* Then branches based on specifics.
+
      ...
  
Actually fetching: "do the fetch"
+
    def clear_cors_preflight_cache(request):
 +
      ...
  
* Should probably branch on URL scheme first.
+
    def cors_check(response):
* Gets the data.
+
      ...
* Needs to queue tasks unless suppressed (sync / CORS prefetch)
 
* Needs to be able to automatically follow redirects (http/https only).
 
  
 
[[Category:Spec coordination]]
 
[[Category:Spec coordination]]

Latest revision as of 12:31, 1 March 2013

The contents of this page, Fetch, and all edits made to this page in its history, are hereby released under the CC0 Public Domain Dedication, as described in WHATWG Wiki:Copyrights.

Plan

Fetch is fetch.spec.whatwg.org and will define HTML fetch and CORS as a set of coherent algorithms rather than the intertwined mess we have now.

Goals

  • Centralize redirect handling
  • Centralize CORS
  • Centralize fetching and thereby consistify handling of e.g. data URLs across APIs
  • Allow resources to opt into CORS without the API asking for it (stop requiring a crossorigin attribute, remove "No CORS" mode)

Model

The basic model is Request -> Fetch -> Response.

Request

  • Parsed URL (object)
  • method (probably with restrictions as seen in XHR)
  • UA headers
  • author headers (maybe rename because people get upset with "author", with implicit restrictions as seen in XHR / CORS)
  • entity body
  • origin (object)
  • referrer source (Document / URL)
  • manual redirect flag
  • omit credentials flag (will replace HTML fetch block cookies flag but also has other features)
  • force preflight flag (set for upload progress notifications (to not reveal existence of server in case of POST I suppose; see bug 20322))
  • synchronous flag
  • force same-origin flag (looks identical to No CORS, fail, filed bug 20951)
  • CORS mode
    • No CORS, taint (<link>, <script>, ...); still need to allow the server to opt in to CORS anyway to effectively make the resource CORS same-origin even if not requested as such (HTML does not have this feature
    • No CORS, fail (<track>)
    • Anonymous
    • Credentialed

Basic Fetch

See also URL. Fetch specifics depends on the URL scheme. Where possible fetching happens incrementally. If it completely fails you get a network error. Otherwise a response. A response might be exposed from the moment status/headers are available and the entity body is still loading.

about

See URL schemes.

blob

http://dev.w3.org/2006/webapi/FileAPI/#processingModel

data

Response object as per http://xhr.spec.whatwg.org/#data:-urls-and-http

If the data URL fails to parse however return a network error.

file

This is by and large platform-specific.

ftp

Construct the request per the FTP specification, do the request, and expose the response.

http and https

  • Construct the request per the HTTP specification
  • Do the request
  • Expose the response.

any other scheme

Network error.

Fetch

Basic Fetch deals with a single request/response whereas Fetch tackles the more complicated setups required by http/https:

  • Redirects
    • We want to follow them if the status code is 301, 302, 303, 307, 308 and there's a Location header (and if there's more than one Location header we need to pick probably), unless a flag is set to not follow redirects).
    • We need to carefully consider what happens if the new Location is not same-origin and especially what if it's not http or https. That might depend on the API.
  • Authentication (challenge request, prompt the user for certain legacy APIs)
  • CORS (preflight requests, marking the response as same-origin or not)
  • Cookies
    • Include them in the request or not?
    • Deal with cookies in the response (before handling redirects and such) and the storage mutex.
  • 304 needs to be turned into a 200 for XMLHttpRequest unless explicitly requested otherwise. We should make it clearer how that works.

For everything else Fetch will simply defer to Basic Fetch, but Fetch will be the algorithm used by the whole platform.

Response

Both intermediate updates (progress, headers received, ...) and final. Also indicates network error / CORS error (exposed as network error), ...

More importantly, it expresses everything in terms of HTTP, regardless of whether the request was for file/blob/data/etc. For this we need to expose:

  • Status code
  • Status text
  • Headers
  • Entity body

User agent advice

For http/https, maybe give advice on proxies, e.g.: "If the user agent allows the end user to configure a proxy it should modify the request appropriately; i.e., connect to the proxy host instead of the origin server, modify the Request-Line and send Proxy-Authorization headers as specified." Though ideally HTTP does this.

For http/https, advice to not include to much random headers, even though it'll happen to some extent anyway.

Pseudo-code

 // "No CORS" means no Origin header, no CORS checking, mark as "cross-origin", unless
 // no_tainting is defined, in which case NetworkError()
 // "Anonymous" means omit_credentials, normal CORS stuff; should probably mean no
 // Referer and globally unique identifier for Origin
 // "Use Credentials" means normal CORS stuff
 class Request:
   url
   origin              // currently "source origin" in some specs
   referrer_source
   method              // e.g. GET
   author_headers = []
   ua_headers = []
   headers = []        // unholy union of author_headers and ua_headers
   entity_body
   force_preflight     // CORS prefligth required
   force_same_origin   // HTML uses this for XMLDocument.load() and Workers
   cors                // True / False for now
   omit_credentials    // set to True to omit cookies for cross-origin requests (HTTP authentication?)
   tainted
   no_tainting         // set to True to terminate in a "network_error" when tainted is set
                       // XMLHttpRequest wants this, <track> too apparently
   redirect_count      // to prevent loops just terminate when this is 10 or so
 class Response:
   type                // "", "redirect", "network_error", "abort_error", "response"
   location            // if type == "redirect", a URL, None otherwise
   status_code
   status_text
   headers
   headers_exposed     // not all headers can be exposed to web-facing APIs
   entity_body
   cors                // "same-origin" or "cross-origin"
 class NetworkError(Response):
   ... state is "done", type is "network_error"
 class Notifier:
   taint = False
   response = Response | None
   def handle_response(response):
     if taint:
       response.cors = "cross-origin"
   def handle_data(data):
     response.feed(data)
   def notifier_wait_for_response:
     // magic dust
     return response
 class Fetch:
   request = Request
   notifier = Notifier
   def fetch():
     url = request.url
     origin = request.origin
     if url.origin == origin or url.scheme in ("about", "blob", "data"):
       redirect_fetch()
     elif request.force_same_origin or (not request.cors and request.no_tainting):
       error()
     elif not request.cors:
       request.tainted = True
       redirect_fetch() // url.scheme == "file" ends up here
     elif url.scheme not in ("http", "https"):
       error()
     elif not requires_cors_with_preflight():
       cors_fetch()
     else:
       cors_fetch_with_preflight()
     if request.synchronous:
       return notifier.wait_for_response()
   def basic_fetch():
     url = request.url
     if url.scheme == "about":
       if url.scheme_data == "blank":
         return Response({type:"text/html", encoding:"utf-8", body:""})
       else:
         return NetworkError()
     elif url.scheme == "blob":
       ...
     elif url.scheme == "data":
       ...
     elif url.scheme == "file":
       ...
     elif url.scheme == "ftp":
       ...
     elif url.scheme in ("http", "https"):
       ... count redirects
     else:
       return NetworkError()
   def redirect_fetch(request):
     response = basic_fetch(request)
     response.wait_for_headers() // design something better
     if response.type == "redirect":
       request.url = response.location
       return fetch(request)
     else:
       return response
   def requires_cors_with_preflight(request):
     if request.force_preflight:
       return True
     if request.method not in cors_simple_methods:
       return True
     for header in request.author_headers:
       if not is_cors_simple_header(header):
         return True
   def cors_simple_header(header):
     ...
   def cors_fetch(request):
     response = basic_fetch(request)
     response.wait_for_headers()
     if response.type == "redirect":
       if request.url.origin != response.location.origin:
         request.origin = uuid()
       request.url = response.location
       if request.url.username or request.url.password or not cors_check(response):
         response.type = "network_error"
         return response
       return cors_fetch(request)
     // XXX should this succeed for about: / blob: data: (think redirects)
     elif response.type == "response" and not cors_check(response):
       response.type = "network_error
     return response
   def cors_fetch_with_preflight(request):
     if not in_cors_preflight_cache(request):
       result = cors_preflight_fetch(request)
       if result.type in ("abort_error", "network_error"):
         return result
     response = basic_fetch(request)
     response.wait_for_headers()
     if response.type in ("redirect", "network_error") or (response.type == "response" and not cors_check(response)):
       clear_cors_preflight_cache(request)
       response.type = "network_error"
     return response
   def cors_preflight_fetch(request):
     preflight_request = Request()
     preflight_request.url = request.url
     preflight_request.origin = request.origin
     preflight_request.referrer_source = request.referrer_source
     preflight_request.method = OPTIONS
     // block cookies?
     preflight_request.ua_headers.append(Access-Control-Request-Method, request.method)
     preflight_request.ua_headers.append(Access-Control-Request-Headers, request.author_headers
     
     response = basic_fetch(preflight_request)
     if response.status_code >= 200 and response.status_code < 300:
       more checks
     elif response.type != "abort_error":
       response.type = "network_error"
     return response
   def in_cors_preflight_cache(request):
     ...
   def clear_cors_preflight_cache(request):
     ...
   def cors_check(response):
     ...