22 ms·

Since everyone is asking (and no one is doing), I put together a simple benchmark for pyquery, bs4, and lxml (cssselect/xpath). https://gist.github.com/4061368

Since everyone is asking (and no one is doing), I put together a simple benchmark for pyquery, bs4, and lxml (cssselect/xpath). https://gist.github.com/4061368 https://gist.github.com/4061368 All it does is grab paragraphs from python.org's html a couple thousand times. ==== Total trials: 100000 ===== bs4 total time: 31.6 pq total time: 9.3 lxml (cssselect) total time: 5.4 lxml (xpath) total time: 4.3 regex total time: 8.9 (doesn't find all p) What does it mean? Unless you're running thousands of queries for parsing, it doesn't matter which library you choose. My computer old and slow. Pick which one is the easiest, that you'll fight the least with. Don't put energy into unnecessary optimization. Using a good library is like choking someone, they'll fight for a little while until they pass out. (you'll remember this analogy next time you want to switch libraries, do you really need to choke someone to get your job done?) After they pass out, it's smooth sailing and you don't have to worry. Don't rock the boat unless you have to.

kami8845 14y ago
Haha, thank you mercuryrising. That's quite the interesting comparison. A few thoughts: - Python.org is quite a simple web app, it would be interesting to run this against something a little more complex like a long wikipedia article or http://www.nytimes.com/ http://www.nytimes.com/ or even the Alexa Top 500 - It would also be interesting to split the times between parsing and selecting as I feel that's where the difference between pq and lxml comes in. All in all it seems BS4 is quite a bit faster than I gave it credit for (especially factoring in parsing+selecting instead of just the former)
- mercuryrising 14y ago
  Python.org was pretty simple. bs4 really hits the brakes on the NYT page. ==== Total trials: 100000 ===== bs4 parsing time: 0.6 bs4 selecting time: 146.3 pq parsing time: 0.0 pq selecting: 15.7 lxml parsing time: 0.0 lxml (cssselect) selecting time: 12.4 lxml parsing time: 0.0 lxml (xpath) selecting: 11.5
  - kami8845 14y ago
    That seems more in line with my experience :) This is the stuff lxml shines at. Thank you for testing this out.
robocaptain 14y ago
I'm only upvoting for the choking analogy. That one is going into my permanent memory file.

kami8845 14y ago

Haha, thank you mercuryrising. That's quite the interesting comparison. A few thoughts: - Python.org is quite a simple web app, it would be interesting to run this against something a little more complex like a long wikipedia article or http://www.nytimes.com/ http://www.nytimes.com/ or even the Alexa Top 500 - It would also be interesting to split the times between parsing and selecting as I feel that's where the difference between pq and lxml comes in. All in all it seems BS4 is quite a bit faster than I gave it credit for (especially factoring in parsing+selecting instead of just the former)

mercuryrising 14y ago
Python.org was pretty simple. bs4 really hits the brakes on the NYT page. ==== Total trials: 100000 ===== bs4 parsing time: 0.6 bs4 selecting time: 146.3 pq parsing time: 0.0 pq selecting: 15.7 lxml parsing time: 0.0 lxml (cssselect) selecting time: 12.4 lxml parsing time: 0.0 lxml (xpath) selecting: 11.5
- kami8845 14y ago
  That seems more in line with my experience :) This is the stuff lxml shines at. Thank you for testing this out.

robocaptain 14y ago

I'm only upvoting for the choking analogy. That one is going into my permanent memory file.