3 ms·
BeautifulSoup is great if you don't care about the performance at all. Because it is painfully slooooooww. Lxml doesn't work well with broken html, but is an o
by polote 5y ago
BeautifulSoup is great if you don't care about the performance at all. Because it is painfully slooooooww.
Lxml doesn't work well with broken html, but is an or two orders of magnitude faster for parsing, and same for querying with xpath.
A part from that, there is also Scrapy which is used a lot, but same it is also very slow, it is just horizontally scalable easily.
There are a lot of times in which scrapping doesn't use html parsing, when you are scrapping pages which change a lot of structure, it might be better to go with full text search, and in this case, the faster the better. And in that area Python is far from the best, except when .split() and .join() are enough. Even re.match is slow because of the algorithm they use is slow
And to finish, Requests is also super slow, if you want something fast you have to use pycurl.
- traverseda 5y agoSelectolax is nice, much faster than bs4 or lxml. Not a very well known project yet though. Not sure there's anything faster on the javascript side of the fence?
- dec0dedab0de 5y agoBeautiful Soup can use lxml, and does by default for parsing xml.
- polote 5y agoThere is a big speed difference between lxml alone and lxml + bs4
- Marazan 5y agoYes, the difference is infinite with broken HTML which is checks notes a huge chunk of the Internet.
- dec0dedab0de 5y agoI was unaware, I always use bs4 with lxml for parsing xml just because I like the interface. For what I'm doing, the bottleneck is the remote system/network, so it doesn't really matter. But now I'm curious about which parts are slower and why. Maybe I'll run some experiments later.
- guskel 5y agoDoes Scrapy's slow speed actually matter much? Your main bottleneck is always going to be network calls and rate limiting. I don't know how much optimization can help there.
- sigil 5y agoIn my experience selectolax is about 10x faster than lxml, and keeps the familiar CSS selector API: https://rushter.com/blog/python-fast-html-parser/ https://rushter.com/blog/python-fast-html-parser/