3 ms·
> [...] prefer to stick with lxml for raw speed. Is parsing speed really an issue when you are scraping the data you are parsing?
by rplnt 13y ago
> [...] prefer to stick with lxml for raw speed.
Is parsing speed really an issue when you are scraping the data you are parsing?
- dikei 13y agoIt's an issue if you have high internet speed. I've had cases where the data can be downloaded faster than it can be parsed and overflow the internal queue. It's even worse if your crawler is not asynchronous, in that case slow parser slow down crawling speed as well.
- Jake232 13y agoIf you're running on a server with 1Gbps, then yes - it can be an issue. Another issue (I'm going to add a section on it) is that you can peg the CPU at 100% very easily. Parsing HTML / Running xPaths uses a lot of CPU, so if you're running a good number of threads this can quickly become a problem.