Testsuite/Requirements

HTML Testsuite Requirements

Improved Test Runner

The current test runner is optimised mainly for manual tests. An in-browser test runner that deals automatically with the results of javascript tests is essential. It should also deal with self-describing reftests and manual tests in a sane way.

Requirements

• Produces a machine-readable report in some format (could be current XML or some other possibly non-XML format).

• Loads tests automatically based on test manifest files containing metadata about test uri, type, etc.

• Allows all or a subset of tests to be run

• Hooks into testharness to extract the results of the javascript tests without human intervention

• Allows reftests to be run by humans e.g. by allowing user to flick between test view and ref view to spot differences (automatic running of reftests will require browser-specific code and is explicitly out of scope)

• Allows manual tests to be run by humans

Possible considerations

• Tests that require a top level browsing context

Annotation of Spec

In order to know which areas of the spec are well tested and hence have a sense for (an upper bound on) the completeness of the testsuite as well as the areas where it would be most profitable to direct new testing effort, it would be beneficial to produce an annotated version of the spec documenting tests that cover given requirements.

Requirements

• Map each test onto a piece of spec

• Fine grained definition of "piece"; some sections are long and contain many normative requirements so paragraph-level is probably the minimum useful level

• Good behaviour in the face of spec modifications, deletions, insertions and rearragements.