4 ms·

To be clear, this only implements the parser section of the specification; that is the stuff at [1]. It is pretty useful if you want a reliable-with-real-world-

To be clear, this only implements the parser section of the specification; that is the stuff at [1]. It is pretty useful if you want a reliable-with-real-world-content HTML parser that can be used with many python tree libraries that only ship XML parsers or ship unreliable HTML parsers (e.g. ElementTree, lxml, minidom). It is probably a bad idea to use it if performance is a big concern. (disclaimer: I am one of the maintainers of this library) [1] http://www.whatwg.org/specs/web-apps/current-work/multipage/parsing.html#parsing http://www.whatwg.org/specs/web-apps/current-work/multipage/...

coderdude 16y ago
I've relied heavily on lxml in the past to parse HTML and I've found it to be quite reliable against tag soup. Do you have any specific gripes or sources I could reference? I'm interested in finding out where it is deficient.
- hoppipolla 16y ago
  Hmm, I only have vague recollections about people reporting problems with lxml and I can't trivially find any supporting evidence with Google so it is possible that it is good enough in almost all cases (since afaik it doesn't actually implement the HTML5 algorithm yet it presumably doesn't work when you need browser-grade compatibility, but such extreme requirements are rare). In any case lxml is a superb library which I would recommend to anyone; personally I often use html5lib with the lxml treebuilder to take advantage of all the nice features in the lxml tree. It is also the best choice I am aware of for fast parsing of possibly broken HTML in python.
- ianb 16y ago
  libxml2's parser (which lxml uses) was written before HTML 5. It generally works quite well, but I've found a bug or two in it, and it doesn't get updated much. For instance, weird things can happen with tags after the </body> tag (I think maybe you can get a second body or head element). lxml can't parse Google's homepage as another example, but it's an extreme example.

coderdude 16y ago

I've relied heavily on lxml in the past to parse HTML and I've found it to be quite reliable against tag soup. Do you have any specific gripes or sources I could reference? I'm interested in finding out where it is deficient.

hoppipolla 16y ago
Hmm, I only have vague recollections about people reporting problems with lxml and I can't trivially find any supporting evidence with Google so it is possible that it is good enough in almost all cases (since afaik it doesn't actually implement the HTML5 algorithm yet it presumably doesn't work when you need browser-grade compatibility, but such extreme requirements are rare). In any case lxml is a superb library which I would recommend to anyone; personally I often use html5lib with the lxml treebuilder to take advantage of all the nice features in the lxml tree. It is also the best choice I am aware of for fast parsing of possibly broken HTML in python.
ianb 16y ago
libxml2's parser (which lxml uses) was written before HTML 5. It generally works quite well, but I've found a bug or two in it, and it doesn't get updated much. For instance, weird things can happen with tags after the </body> tag (I think maybe you can get a second body or head element). lxml can't parse Google's homepage as another example, but it's an extreme example.