A user account is required in order to edit this wiki, but we've had to disable public user registrations due to spam.

To request an account, ask an autoconfirmed user on IRC (such as one of these permanent autoconfirmed members).

Difference between revisions of "HTML5Lib"

From WHATWG Wiki
Jump to: navigation, search
m
Line 1: Line 1:
[http://wainfan.t35.com/view/new/il-moto.htm il moto cros] [http://grahrens.125mb.com/view/new/pac-eco/ pac eco 70] [http://oedi.freehostia.com/text/directory/brachi/ brachi-] [http://coarmago.t35.com/content/view/lg-tv/ lg tv rz-20lz50] [http://grahrens.125mb.com/view/new/juliana-lopes/ juliana lopes bbb4] [http://boreldar.t35.com/text/directory/panasonic-viera.htm panasonic viera 42pa50e] [http://gawelzba.t35.com/library/html/maria-paola.htm maria paola] [http://greensticky.freehostia.com/library/html/jvc-mg/ jvc mg30] [http://lorcaxon.homeblock.com/data/gino-vannelli/ gino vannelli] [http://oedi.freehostia.com/text/directory/il-canale/ il canale degli angeli] [http://www.freewebtown.com/prncessprncipe/library/html/roper-pcmcia.htm roper pcmcia] [http://boreldar.t35.com/text/directory/rubeola.htm rubeola] [http://beshennayabelka.freehostia.com/content/view/agente-cody.htm agente cody banks] [http://www.freewebtown.com/boogiejames/topic/bd-spencer.htm bd spencer] [http://failtoexpress.freehostia.com/html/configurazione-vi.htm configurazione v3i] [http://flheddin.125mb.com/content/view/veterinari-ravenna.htm veterinari ravenna] [http://mauballe.125mb.com/web/dany-brillant/ dany brillant] [http://flheddin.125mb.com/content/view/tel-one.htm tel one] [http://benevolently.freehostia.com/content/view/di/ di 614] [http://benevolently.freehostia.com/content/view/volevo-i/ volevo i pantaloni] [http://scaleslea.freewebpage.org/text/directory/y-tu/ y tu mama tambien - anche tua madre] [http://mcwardlo.t35.com/library/html/copy/ copy] [http://coarmago.t35.com/content/view/body-performance/ body performance] [http://www.freewebtown.com/prncessprncipe/library/html/www-cis.htm www cis it] [http://licohwar.homeblock.com/library/html/nokia-/ nokia 7610 imp] [http://coarmago.t35.com/content/view/collare-cuoio/ collare cuoio] [http://lorcaxon.homeblock.com/data/lexmark-inchiostro/ lexmark inchiostro] [http://coarmago.t35.com/content/view/roksan/ roksan] [http://flheddin.125mb.com/content/view/giochi-per.htm giochi per motorola c550] [http://huruple.125mb.com/text/directory/sissoko-let/ sissoko let the music take control] [http://wainfan.t35.com/view/new/asus-mypal.htm asus mypal a730] [http://greensticky.freehostia.com/library/html/echinopsis/ echinopsis] [http://flheddin.125mb.com/content/view/ricetta-riso.htm ricetta riso] [http://kopapguw.homeblock.com/topic/regioni-d/ regioni d italia] [http://getraylo.freeweb7.com/img/styles/batteria-acer.htm batteria acer aspire 1680] [http://www.freewebtown.com/boogiejames/topic/kasaioccidental.htm kasai-occidental] [http://beshennayabelka.freehostia.com/content/view/il-sindacalista.htm il sindacalista] [http://licohwar.homeblock.com/library/html/rossana-hentai/ rossana hentai] [http://benevolently.freehostia.com/content/view/decatlon-articolo/ decatlon articolo sportivi] [http://zoelverd.freeweb7.com/description/lib/attrice-italiana/ attrice italiana g] [http://coarmago.t35.com/content/view/mystery-island/ mystery island] [http://licohwar.homeblock.com/library/html/la-bella/ la bella otero] [http://flheddin.125mb.com/content/view/super-merk.htm super merk2] [http://getm0ney.freewebpage.org/lib/nuova-toyota/ nuova toyota corolla benzina auto nuove] [http://beshennayabelka.freehostia.com/content/view/multilevel-governance.htm multi-level governance] [http://mauballe.125mb.com/web/usb-adapter/ usb adapter ipaq] [http://oedi.freehostia.com/text/directory/gioiellerie-morellato/ gioiellerie morellato a salerno] [http://boreldar.t35.com/text/directory/hotel-castello.htm hotel castello santonio] [http://zoelverd.freeweb7.com/description/lib/staffa-a/ staffa a muro] [http://huruple.125mb.com/text/directory/annuncio-cani/ annuncio cani] [http://wainfan.t35.com/view/new/galerias-de.htm galerias de mujeres] [http://lorcaxon.homeblock.com/data/ogni-volta/ ogni volta che vedo il mare] [http://getraylo.freeweb7.com/img/styles/unoaerre-italia.htm unoaerre italia spa] [http://lorcaxon.homeblock.com/data/nokia-/ nokia 7710 grey pearl] [http://gawelzba.t35.com/library/html/socket-.htm socket 478 i] [http://oedi.freehostia.com/text/directory/enrico-vincenzi/ enrico vincenzi] [http://coarmago.t35.com/content/view/lexar-compact/ lexar compact flash 80x] [http://coarmago.t35.com/content/view/le-sorprese/ le sorprese dellamore] [http://getraylo.freeweb7.com/img/styles/ganggbang.htm ganggbang] [http://grahrens.125mb.com/view/new/only-big/ only big real great beautiful natural ti] [http://coarmago.t35.com/content/view/arcuri-panorama/ arcuri panorama] [http://www.freewebtown.com/starbriteyellow/resources/articles/cartoonnetvork.htm cartoonnetvork] [http://www.freewebtown.com/starbriteyellow/resources/articles/sarcinella.htm sarcinella] [http://lorcaxon.homeblock.com/data/sevillana/ sevillana] [http://beshennayabelka.freehostia.com/content/view/telegiornali-sexi.htm telegiornali sexi] [http://scaleslea.freewebpage.org/text/directory/donne-mature/ donne mature 50 60anni] [http://zoelverd.freeweb7.com/description/lib/super-furry/ super furry animals. phantom power] [http://failtoexpress.freehostia.com/html/ippolito-nino.htm ippolito nino] [http://www.freewebtown.com/prncessprncipe/library/html/www-liberoit.htm www liberoit] [http://grahrens.125mb.com/view/new/lavatrice-gettone/ lavatrice gettone] [http://www.freewebtown.com/starbriteyellow/resources/articles/p.htm p4 3 0] [http://oedi.freehostia.com/text/directory/hp/ hp 710] [http://zoelverd.freeweb7.com/description/lib/shema-israel/ shema israel] [http://failtoexpress.freehostia.com/html/artrosi-cervicale.htm artrosi cervicale] [http://greensticky.freehostia.com/library/html/junior-jack/ junior jack robert smith] [http://wainfan.t35.com/view/new/kardia-mou.htm kardia mou min anisihis] [http://www.freewebtown.com/boogiejames/topic/losredondos.htm losredondos] [http://huruple.125mb.com/text/directory/psycho-sistem/ psycho sistem of a down] [http://lorcaxon.homeblock.com/data/weston-edward/ weston, edward (ingegnere)] [http://huruple.125mb.com/text/directory/fifa-com/ fifa99 com] [http://mauballe.125mb.com/web/ixus/ ixus 5] [http://boreldar.t35.com/text/directory/delta-force.htm delta force2] [http://beshennayabelka.freehostia.com/content/view/mambomania.htm mambomania] [http://wainfan.t35.com/view/new/thi.htm thi] [http://www.freewebtown.com/boogiejames/topic/anna-e.htm anna e il re] [http://flheddin.125mb.com/content/view/elisha-cuthbert.htm elisha cuthbert] [http://gawelzba.t35.com/library/html/lettore-karaoke.htm lettore karaoke divx] [http://getm0ney.freewebpage.org/lib/sessuologo-messina/ sessuologo messina] [http://scaleslea.freewebpage.org/text/directory/versilia-albergo/ versilia albergo] [http://huruple.125mb.com/text/directory/no-tiene/ no tiene dinero] [http://zoelverd.freeweb7.com/description/lib/sony-radio/ sony radio portatile] [http://scaleslea.freewebpage.org/text/directory/mi-chiamo/ mi chiamo mimi] [http://greensticky.freehostia.com/library/html/dvd-samsung/ dvd samsung vhs] [http://failtoexpress.freehostia.com/html/atisapphire-radeon.htm ati-sapphire radeon 9600 se] [http://www.freewebtown.com/starbriteyellow/resources/articles/canon-.htm canon - obiettivo zoom ef 28-135mm is] [http://mcwardlo.t35.com/library/html/www-adecco/ www adecco it] [http://failtoexpress.freehostia.com/html/soundblaster-live.htm soundblaster live 5 1] [http://failtoexpress.freehostia.com/html/dvd-srink.htm dvd srink download] [http://gawelzba.t35.com/library/html/sennheiser.htm sennheiser 30] [http://kopapguw.homeblock.com/topic/film-di/ film di dragonball z] [http://wainfan.t35.com/view/new/giovanna.htm giovanna 41] [http://flheddin.125mb.com/content/view/sexy-sexy.htm sexy sexy photo] [http://www.freewebtown.com/prncessprncipe/library/html/www-go.htm www go party it] [http://grahrens.125mb.com/view/new/magnum-pi/ magnum p.i. stagione 2] [http://wainfan.t35.com/view/new/sarah-kane.htm sarah kane libri] [http://scaleslea.freewebpage.org/text/directory/majhon/ majhon] [http://zoelverd.freeweb7.com/description/lib/porno/ porno] [http://beshennayabelka.freehostia.com/content/view/tine.htm tine] [http://zoelverd.freeweb7.com/description/lib/ci-/ ci 1300 fb] [http://oedi.freehostia.com/text/directory/masterizzatore-dvd/ masterizzatore dvd per mac] [http://huruple.125mb.com/text/directory/www-kladi/ www kladi it] [http://getm0ney.freewebpage.org/lib/supporto-tv/ supporto tv plasma parete] [http://getraylo.freeweb7.com/img/styles/maria-cristina.htm maria cristina blu] [http://oedi.freehostia.com/text/directory/singoli/ singoli] [http://mauballe.125mb.com/web/usb-motorola/ usb motorola pc 220] [http://oedi.freehostia.com/text/directory/brigata-cani/ brigata cani da birra] [http://selemku.t35.com/text/directory/link-siti.htm link siti porrno] [http://getm0ney.freewebpage.org/lib/banca-mutuo/ banca mutuo casa] [http://www.freewebtown.com/starbriteyellow/resources/articles/shox-.htm shox 2 45 nike calzature] [http://selemku.t35.com/text/directory/uomo-giacca.htm uomo giacca] [http://www.freewebtown.com/prncessprncipe/library/html/lee-ritenour.htm lee ritenour. live in montreal] [http://selemku.t35.com/text/directory/ericsson-z.htm ericsson z1010] [http://www.freewebtown.com/starbriteyellow/resources/articles/montevarchi.htm montevarchi] [http://zoelverd.freeweb7.com/description/lib/midi-no/ midi no me quiero enamorar] [http://www.freewebtown.com/starbriteyellow/resources/articles/bellisimi.htm bellisimi] [http://mcwardlo.t35.com/library/html/not-that/ not that kind anastacia] [http://saline-joy.freehostia.com/web/air-silver.htm air silver] [http://flheddin.125mb.com/content/view/hoobastanks.htm hoobastanks] [http://getraylo.freeweb7.com/img/styles/rolf.htm rolf] [http://www.freewebtown.com/boogiejames/topic/scarica-copertina.htm scarica copertina cd] [http://zoelverd.freeweb7.com/description/lib/ggggggggggggggggggggggggggggggggg/ ggggggggggggggggggggggggggggggggg] [http://www.freewebtown.com/prncessprncipe/library/html/la-donna.htm la donna dell amico mio testo carlos] [http://mcwardlo.t35.com/library/html/guardie-giurate/ guardie giurate] [http://greensticky.freehostia.com/library/html/arius/ arius] [http://www.freewebtown.com/prncessprncipe/library/html/lcd-televisori.htm lcd televisori samsung] [http://selemku.t35.com/text/directory/cagliari-immagini.htm cagliari immagini] [http://mauballe.125mb.com/web/de-ce/ de ce ma minti remix] [http://selemku.t35.com/text/directory/studi-consulenti.htm studi consulenti fiscali trapani] [http://selemku.t35.com/text/directory/vendita-fiori.htm vendita fiori on line] [http://scaleslea.freewebpage.org/text/directory/preamplificatori-audio/ preamplificatori audio] [http://www.freewebtown.com/starbriteyellow/resources/articles/siglia-di.htm siglia di buffy] [http://gawelzba.t35.com/library/html/annuncio-privati.htm annuncio privati incontri] [http://boreldar.t35.com/text/directory/ufficio-campobasso.htm ufficio campobasso] [http://benevolently.freehostia.com/content/view/wd-oregon/ wd338 oregon scientific cordless] [http://getraylo.freeweb7.com/img/styles/jolie-nuda.htm jolie nuda] [http://benevolently.freehostia.com/content/view/giulia-michelini/ giulia michelini] [http://www.freewebtown.com/prncessprncipe/library/html/trentino-donne.htm trentino donne] [http://getm0ney.freewebpage.org/lib/gente-motore/ gente motore] [http://mauballe.125mb.com/web/fbi-operazione/ f.b.i. operazione pakistan] [http://mauballe.125mb.com/web/buonanotte-fiorellino/ buonanotte fiorellino de gregori] [http://beshennayabelka.freehostia.com/content/view/tavoletta-grafica.htm tavoletta grafica wacom a6] [http://www.freewebtown.com/boogiejames/topic/muvo-black.htm muvo black] [http://flheddin.125mb.com/content/view/voli-israele.htm voli israele last minute] [http://failtoexpress.freehostia.com/html/hd-autoalimentato.htm hd autoalimentato] [http://selemku.t35.com/text/directory/jenny-garth.htm jenny garth] [http://www.freewebtown.com/boogiejames/topic/capitan-uncino.htm capitan uncino bennato] [http://kopapguw.homeblock.com/topic/volo-cesenatico/ volo cesenatico] [http://mauballe.125mb.com/web/hotel-luino/ hotel luino] [http://beshennayabelka.freehostia.com/content/view/multifunzione-laser.htm multifunzione laser color epson] [http://gawelzba.t35.com/library/html/colori-che.htm colori che esplodono] [http://flheddin.125mb.com/content/view/master-of.htm master of orion 2] [http://saline-joy.freehostia.com/web/dancan.htm dancan] [http://kopapguw.homeblock.com/topic/impanatura/ impanatura] [http://kopapguw.homeblock.com/topic/lacie-safe/ lacie safe mobile hard drive 80gb] [http://lorcaxon.homeblock.com/data/di-/ di  natale] [http://benevolently.freehostia.com/content/view/download-gratis/ download gratis driver stampanti] [http://gawelzba.t35.com/library/html/star-wars.htm star wars la vendetta dei sith] [http://zoelverd.freeweb7.com/description/lib/hp/ hp 715] [http://saline-joy.freehostia.com/web/plasma-.htm plasma 42 pollici a parete tv televisori] [http://zoelverd.freeweb7.com/description/lib/dsc-h/ dsc h1 sony] [http://oedi.freehostia.com/text/directory/punto-/ punto 2005 mi] [http://www.freewebtown.com/boogiejames/topic/bouble-bubbles.htm bouble bubbles] [http://boreldar.t35.com/text/directory/alloggi-venezia.htm alloggi venezia] [http://boreldar.t35.com/text/directory/spille-di.htm spille di sicurezza] [http://mauballe.125mb.com/web/radioactiva/ radioactiva] [http://lorcaxon.homeblock.com/data/maserati-coupe/ maserati coupe cambiocorsa] [http://benevolently.freehostia.com/content/view/real-doll/ real doll com] [http://selemku.t35.com/text/directory/il-porto.htm il porto del vizio] [http://lorcaxon.homeblock.com/data/tonino-carotone/ tonino carotone] [http://mauballe.125mb.com/web/festival-crociera/ festival crociera] [http://www.freewebtown.com/starbriteyellow/resources/articles/la-serenissima.htm la serenissima 2004] [http://saline-joy.freehostia.com/web/cxmi-acer.htm c311xmi acer] [http://www.freewebtown.com/starbriteyellow/resources/articles/www-bambola.htm www bambola ramona] [http://grahrens.125mb.com/view/new/bando-forze/ bando forze armate] [http://www.freewebtown.com/starbriteyellow/resources/articles/camcorder.htm camcorder] [http://saline-joy.freehostia.com/web/congelatore-verticale.htm congelatore verticale aeg] [http://www.freewebtown.com/boogiejames/topic/epson-rx.htm epson rx600] [http://failtoexpress.freehostia.com/html/dissipatori-raffreddamento.htm dissipatori raffreddamento a liquido] [http://getraylo.freeweb7.com/img/styles/desenzano-del.htm desenzano del garda art discoteca] [http://www.freewebtown.com/prncessprncipe/library/html/tastiere-computer.htm tastiere computer] [http://mcwardlo.t35.com/library/html/mp-di/ mp3 di slide along side] [http://grahrens.125mb.com/view/new/escort-palermo/ escort palermo] [http://benevolently.freehostia.com/content/view/iracheno-decapitato/ iracheno decapitato] [http://grahrens.125mb.com/view/new/cerco-donna/ cerco donna a rimini] [http://grahrens.125mb.com/view/new/jimmy-eat/ jimmy eat world. believe in what you want] [http://getraylo.freeweb7.com/img/styles/draghixa.htm draghixa] [http://www.freewebtown.com/starbriteyellow/resources/articles/dragonjar.htm dragonjar] [http://www.freewebtown.com/prncessprncipe/library/html/minipc.htm minipc] [http://lorcaxon.homeblock.com/data/auto-noleggio/ auto noleggio chiavari] [http://scaleslea.freewebpage.org/text/directory/frigorifero-indesit/ frigorifero indesit] = HTML5Lib =
+
= HTML5Lib =
  
 
[http://code.google.com/p/html5lib/ HTML5Lib] is a project to create both a Python-based and Ruby-based implementations of various parts of the WHATWG spec, in particular, a tokenizer, a parser, and a serializer. It is '''not''' an offical WHATWG project, however we plan to use this wiki to document and discuss the library design. The code is avaliable under an open-source MIT license.
 
[http://code.google.com/p/html5lib/ HTML5Lib] is a project to create both a Python-based and Ruby-based implementations of various parts of the WHATWG spec, in particular, a tokenizer, a parser, and a serializer. It is '''not''' an offical WHATWG project, however we plan to use this wiki to document and discuss the library design. The code is avaliable under an open-source MIT license.

Revision as of 16:12, 15 August 2007

HTML5Lib

HTML5Lib is a project to create both a Python-based and Ruby-based implementations of various parts of the WHATWG spec, in particular, a tokenizer, a parser, and a serializer. It is not an offical WHATWG project, however we plan to use this wiki to document and discuss the library design. The code is avaliable under an open-source MIT license.

SVN

Please commit often with sort of detailed descriptions of what you did. If you want to make sure you're not going to redo ask on the mailing list. For questions that could benefit from quick turnaround, talk to people on #whatwg.

General

In comments "XXX" indicates something that has yet to be done. Something might be wrong, has not yet been written and other things in that general direction.

In comments "AT" indicates that the comment documents an alternate implementation technique or strategy.

HTMLTokenizer

The tokenizer is controlled by a single HTMLTokenizer class stored in tokenizer.py at the moment. You initialize the HTMLTokenizer with a stream argument that holds an HTMLInputStream. You can iterate over the object created to get tokens back.

Currently tokens are objects, they will become dicts.

Interface

The parser needs to change the self.contentModelFlag attribute which affects how certain states are handled.

Issues

  • Use of if statements in the states may be suboptimal (but we should time this)

HTMLParser

Profiling on web-apps.htm

I did some profiling on web-apps.htm which is a rather large document. Based on that I already changed a number of things which speed us up a bit. Below are some things to consider for future revisions:

  • utils.MethodDispatcher is invoked way too often. By pre declaring some of it in InBody I managed to decrease the amount of invocatoins by over 24.000, but InBody.__init__ is invoked about 7000 times for web-apps.htm so that amount could be higher. Not sure how to put them somewhere else though. First thing I tried was HTMLParser but references get all messed up then...
We should be able to store a single instance of each InsertionMode rather than creating a new one every time the mode switches. Hopefully we have been disiplined enough not to keep any state in those classes so the change should be painless.
That's an interesting idea. How would that work? Annevk 12:49, 25 December 2006 (UTC)
I got an idea on how it might work and it worked! Still about 3863 invocations to utils.MethodDispatcher but it takes 0.000 CPU seconds. I suppose we can decrease that amount even more, but I wonder if it's worth it. Annevk 11:37, 26 December 2006 (UTC)
  • 713194 calls to __contains__ in sets.py makes us slow. Takes about 1.0x CPU seconds.
I've just switched to the built-in sets type. hopefully this will help a bit Jgraham 00:30, 25 December 2006 (UTC)
It did. (Not surprisingly when 700.000 method calls are gone...) Annevk 12:49, 25 December 2006 (UTC)
  • 440382 calls to char in tokenizer.py is the runner up with 0.8x CPU seconds.
This is now the largest time consumer. Annevk 12:49, 25 December 2006 (UTC)
  • dataState in tokenizer.py with 0.7 CPU seconds is next.
This is now at 0.429 CPU seconds. Probably becase the tokenizer switched to dicts instead of custom Token objects. Annevk
  • __iter__ in tokenizer.py with 0.59x CPU seconds...
  • Creation of all node objects in web-apps takes .57x CPU seconds.
  • etc.

Testcases

Testcases are under the /tests directory. They require simplejson. New code should not be checked in if it regresses previously functional unit tests. Similarly, new tests that don't pass should not be checked in without both informing others on the mailing list and a concrete plan. Ideally new features should be accompanied by new unit tests for those features. Documentation of the test format is available at Parser_tests.