A user account is required in order to edit this wiki, but we've had to disable public user registrations due to spam.

To request an account, ask an autoconfirmed user on IRC (such as one of these permanent autoconfirmed members).

Difference between revisions of "Testsuite"

From WHATWG Wiki
Jump to: navigation, search
(add easy reviewing to list of requirements)
(Test metadata and automation)
Line 9: Line 9:
 
* If test can be done using JavaScript preferably require it to be in JavaScript so engines can be more efficiently tested (i.e. automated).
 
* If test can be done using JavaScript preferably require it to be in JavaScript so engines can be more efficiently tested (i.e. automated).
 
* It must be easy to review tests
 
* It must be easy to review tests
 +
* Standardize a test format?
  
 
== Non-requirements ==
 
== Non-requirements ==
  
 
* There does not need to be a single consistent test harness for the whole of HTML5. (When sections can be tested in isolation, each section should use a test harness that is suited to that section's testing requirements. E.g. there is little value in fitting canvas tests and parser tests into the same framework, and it may add a lot of complexity.)
 
* There does not need to be a single consistent test harness for the whole of HTML5. (When sections can be tested in isolation, each section should use a test harness that is suited to that section's testing requirements. E.g. there is little value in fitting canvas tests and parser tests into the same framework, and it may add a lot of complexity.)
 +
 +
== Tests metadata ==
 +
 +
Having a standardized way of specifying tests metadata could be helpful. In particular, they should contain information about:
 +
* What part of the specification the test is related to.
 +
* Requirements for running the tests, that could be expressed as flags.
 +
 +
The CSS Test suite defines a format for specifying test metadata that could be used: http://wiki.csswg.org/test/css2.1/format.
 +
 +
Testcase metadata sample (from http://samples.msdn.microsoft.com/IETestCenter/HTML5/DOMStorage/localStorage_clear.htm):
 +
    <head>
 +
        <title>HTML 5 Test Suite: localStorage API - clear()</title>
 +
        <link rel="author" title="Microsoft" href="http://www.microsoft.com/" />
 +
        <link rel="help" href="http://www.w3.org/html/wg/html5/#structured-client-side-storage" />
 +
        <meta name="assert" content="5.10.1.2 - The clear() method must atomically cause the list associated with the object to be emptied of all key/value pairs, if there are any." />
 +
    </head>
 +
 +
== Automation ==
 +
 +
Test cases should be able to run automatically whenever possible for several reasons:
 +
* No need to run the test manually, which can saves time and produce more reliable results.
 +
* Browser vendors can integrate the test in their automated regression harnesses.
 +
 +
The [http://www.browsertests.org Browser Tests project] uses 3 formats that could be used for automated tests. They are based on automated tests from WebKit and Mozilla and are documented here: http://code.google.com/p/browsertests/wiki/StartPage#Tests_Format
  
 
== Existing tests ==
 
== Existing tests ==
Line 18: Line 43:
 
* [http://tc.labs.opera.com/html/ Opera's tests]
 
* [http://tc.labs.opera.com/html/ Opera's tests]
 
* [http://trac.webkit.org/browser/trunk/LayoutTests (Some of?) WebKit's tests]
 
* [http://trac.webkit.org/browser/trunk/LayoutTests (Some of?) WebKit's tests]
 +
* [http://mxr.mozilla.org/mozilla-central/find?string=test_.*%28html|svg%29%24&tree=mozilla-central&hint= Some of Mozilla's mochitests]
 +
* [http://mxr.mozilla.org/mozilla-central/find?string=reftest.list Some of Mozilla's reftests]
 
* [http://philip.html5.org/tests/canvas/suite/tests/ Philip's canvas tests]
 
* [http://philip.html5.org/tests/canvas/suite/tests/ Philip's canvas tests]
 
* [http://hixie.ch/tests/adhoc/html/ Hixie's ad-hoc tests]
 
* [http://hixie.ch/tests/adhoc/html/ Hixie's ad-hoc tests]

Revision as of 12:38, 3 March 2009

Existing tests URI: http://dev.w3.org/html5/tests/

Requirements

  • Each test needs a "reviewed" marker of some sort
  • It must be easy to find tests where the spec has changed under them
  • The barrier to contribution must be as low as possible
  • Testcases should have somewhat stable URIs
  • If test can be done using JavaScript preferably require it to be in JavaScript so engines can be more efficiently tested (i.e. automated).
  • It must be easy to review tests
  • Standardize a test format?

Non-requirements

  • There does not need to be a single consistent test harness for the whole of HTML5. (When sections can be tested in isolation, each section should use a test harness that is suited to that section's testing requirements. E.g. there is little value in fitting canvas tests and parser tests into the same framework, and it may add a lot of complexity.)

Tests metadata

Having a standardized way of specifying tests metadata could be helpful. In particular, they should contain information about:

  • What part of the specification the test is related to.
  • Requirements for running the tests, that could be expressed as flags.

The CSS Test suite defines a format for specifying test metadata that could be used: http://wiki.csswg.org/test/css2.1/format.

Testcase metadata sample (from http://samples.msdn.microsoft.com/IETestCenter/HTML5/DOMStorage/localStorage_clear.htm):

   <head>
       <title>HTML 5 Test Suite: localStorage API - clear()</title>
       <link rel="author" title="Microsoft" href="http://www.microsoft.com/" />
       <link rel="help" href="http://www.w3.org/html/wg/html5/#structured-client-side-storage" />
       <meta name="assert" content="5.10.1.2 - The clear() method must atomically cause the list associated with the object to be emptied of all key/value pairs, if there are any." />
   </head>

Automation

Test cases should be able to run automatically whenever possible for several reasons:

  • No need to run the test manually, which can saves time and produce more reliable results.
  • Browser vendors can integrate the test in their automated regression harnesses.

The Browser Tests project uses 3 formats that could be used for automated tests. They are based on automated tests from WebKit and Mozilla and are documented here: http://code.google.com/p/browsertests/wiki/StartPage#Tests_Format

Existing tests