HTML5Lib

HTML5Lib

HTML5Lib is a project to create a Python-based implementation of various parts of the WHATWG spec, in particular, a tokenizer and parser. It is not an offical WHATWG project, however we plan to use this wiki to document and discuss the library design. The code is avaliable under an open-source MIT license.

SVN

Please commit often with sort of detailed descriptions of what you did. If you want to make sure you're not going to redo work talk to people on #whatwg.

General

In comments "XXX" indicates something that has yet to be done. Something might be wrong, has not yet been written and other things in that general direction.

In comments "AT" indicates that the comment documents an alternate implementation technique or strategy.

HTMLTokenizer

The tokenizer is controlled by a single HTMLTokenizer class stored in tokenizer.py at the moment. You initialize the HTMLTokenizer with a stream argument that holds an HTMLInputStream. You can iterate over the object created to get tokens back.

Currently tokens are objects, they will become dicts.

Interface

The parser needs to change the self.contentModelFlag attribute which affects how certain states are handled.

Issues

• Use of if statements in the states may be suboptimal (but we should time this)

HTMLParser

Profiling on web-apps.htm

I did some profiling on web-apps.htm which is a rather large document. Based on that I already changed a number of things which speed us up a bit. Below are some things to consider for future revisions:

• utils.MethodDispatcher is invoked way too often. By pre declaring some of it in InBody I managed to decrease the amount of invocatoins by over 24.000, but InBody.__init__ is invoked about 7000 times for web-apps.htm so that amount could be higher. Not sure how to put them somewhere else though. First thing I tried was HTMLParser but references get all messed up then...

• 713194 calls to __contains__ in sets.py makes us slow. Takes about 1.0x CPU seconds.

• 440382 calls to char in tokenizer.py is the runner up with 0.8x CPU seconds.

• dataState in tokenizer.py with 0.7 CPU seconds is next.

• __iter_ in tokenizer.py with 0.59x CPU seconds...

• Creation of all node objects in web-apps takes .57x CPU seconds.

• etc.

Testcases

Testcases are under the /tests directory. They require simplejson. New code should not be checked in if it regresses previously functional unit tests. Ideally new features should be accompanied by new unit tests for those features. Documentation of the test format is available at Parser_tests.