3 ms·

On the off chance you were not aware, bs4 also supports[0] getting parse events from html5lib[1] which (as its name implies) is far more likely to parse the tex

On the off chance you were not aware, bs4 also supports[0] getting parse events from html5lib[1] which (as its name implies) is far more likely to parse the text the same way the browser would 0: https://www.crummy.com/software/BeautifulSoup/bs4/doc/index.html#installing-a-parser https://www.crummy.com/software/BeautifulSoup/bs4/doc/index.... 1: https://pypi.org/project/html5lib/ https://pypi.org/project/html5lib/

westurner 4y ago
BeautifulSoup is an API for multiple parsers https://beautiful-soup-4.readthedocs.io/en/latest/#installing-a-parser https://beautiful-soup-4.readthedocs.io/en/latest/#installin... : BeautifulSoup(markup, "html.parser") BeautifulSoup(markup, "lxml") BeautifulSoup(markup, "lxml-xml") BeautifulSoup(markup, "xml") BeautifulSoup(markup, "html5lib") Looks like lxml w/ xpath is still the fastest with Python 3.10.4 from "Pyquery, lxml, BeautifulSoup comparison" https://gist.github.com/MercuryRising/4061368 https://gist.github.com/MercuryRising/4061368 ; which is fine for parsing (X)HTML(5) that validates< (EDIT: Is xml/html5 a good format for data serialization? defusedxml ... Simdjson, Apache arrow.js)
- aumerle 4y ago
  You want a proper html 5 parser that can handle non valid documents. And the fastest one is https://github.com/kovidgoyal/html5-parser https://github.com/kovidgoyal/html5-parser over 30x faster than html5lib
- driscoll42 4y ago
  I was curious, so I tried that performance test you linked to on my machine with the various parsers: ==== Total trials: 100000 ===== bs4 lxml total time: 110.9 bs4 html.parser total time: 87.6 bs4 lxml-xml total time: 0.5 bs4 xml total time: 0.5 bs4 html5lib total time: 103.6 pq total time: 8.7 lxml (cssselect) total time: 8.8 lxml (xpath) total time: 5.6 regex total time: 13.8 (doesn't find all p) bs4 is damn fast with the lxml-xml or xml parsers

westurner 4y ago

BeautifulSoup is an API for multiple parsers https://beautiful-soup-4.readthedocs.io/en/latest/#installing-a-parser https://beautiful-soup-4.readthedocs.io/en/latest/#installin... : BeautifulSoup(markup, "html.parser") BeautifulSoup(markup, "lxml") BeautifulSoup(markup, "lxml-xml") BeautifulSoup(markup, "xml") BeautifulSoup(markup, "html5lib") Looks like lxml w/ xpath is still the fastest with Python 3.10.4 from "Pyquery, lxml, BeautifulSoup comparison" https://gist.github.com/MercuryRising/4061368 https://gist.github.com/MercuryRising/4061368 ; which is fine for parsing (X)HTML(5) that validates< (EDIT: Is xml/html5 a good format for data serialization? defusedxml ... Simdjson, Apache arrow.js)

aumerle 4y ago
You want a proper html 5 parser that can handle non valid documents. And the fastest one is https://github.com/kovidgoyal/html5-parser https://github.com/kovidgoyal/html5-parser over 30x faster than html5lib
driscoll42 4y ago
I was curious, so I tried that performance test you linked to on my machine with the various parsers: ==== Total trials: 100000 ===== bs4 lxml total time: 110.9 bs4 html.parser total time: 87.6 bs4 lxml-xml total time: 0.5 bs4 xml total time: 0.5 bs4 html5lib total time: 103.6 pq total time: 8.7 lxml (cssselect) total time: 8.8 lxml (xpath) total time: 5.6 regex total time: 13.8 (doesn't find all p) bs4 is damn fast with the lxml-xml or xml parsers