Existing tests URI: http://dev.w3.org/html5/tests/
- Each test needs a "reviewed" marker of some sort
- It must be easy to find tests where the spec has changed under them
- The barrier to contribution must be as low as possible
- Testcases should have somewhat stable URIs
- It must be easy to review tests
- Standardize a test format?
- There does not need to be a single consistent test harness for the whole of HTML5. (When sections can be tested in isolation, each section should use a test harness that is suited to that section's testing requirements. E.g. there is little value in fitting canvas tests and parser tests into the same framework, and it may add a lot of complexity.)
Having a standardized way of specifying tests metadata could be helpful. In particular, they should contain information about:
- What part of the specification the test is related to.
- Requirements for running the tests, that could be expressed as flags.
The CSS Test suite defines a format for specifying test metadata that could be used: http://wiki.csswg.org/test/css2.1/format.
Testcase metadata sample (from http://samples.msdn.microsoft.com/IETestCenter/HTML5/DOMStorage/localStorage_clear.htm):
<head> <title>HTML 5 Test Suite: localStorage API - clear()</title> <link rel="author" title="Microsoft" href="http://www.microsoft.com/" /> <link rel="help" href="http://www.w3.org/html/wg/html5/#structured-client-side-storage" /> <meta name="assert" content="126.96.36.199 - The clear() method must atomically cause the list associated with the object to be emptied of all key/value pairs, if there are any." /> </head>
Test cases should be able to run automatically whenever possible for several reasons:
- No need to run the test manually, which can saves time and produce more reliable results.
- Browser vendors can integrate the test in their automated regression harnesses.
The Browser Tests project uses 3 formats that could be used for automated tests. They are based on automated tests from WebKit and Mozilla and are documented here: http://code.google.com/p/browsertests/wiki/StartPage#Tests_Format