HTML Testsuite Requirements
Improved Test Runner
- Produces a machine-readable report in some format (could be current XML or some other possibly non-XML format).
- Loads tests automatically based on test manifest files containing metadata about test uri, type, etc.
- Allows all or a subset of tests to be run
- Allows reftests to be run by humans e.g. by allowing user to flick between test view and ref view to spot differences (automatic running of reftests will require browser-specific code and is explicitly out of scope)
- Allows manual tests to be run by humans
- Tests that require a top level browsing context
Annotation of Spec
In order to know which areas of the spec are well tested and hence have a sense for (an upper bound on) the completeness of the testsuite as well as the areas where it would be most profitable to direct new testing effort, it would be beneficial to produce an annotated version of the spec documenting tests that cover given requirements.
- Map each test onto a piece of spec
- Fine grained definition of "piece"; some sections are long and contain many normative requirements so paragraph-level is probably the minimum useful level
- Good behaviour in the face of spec modifications, deletions, insertions and rearragements.