3 ms·
Python shouldn't be any slower than Node for crawling if you use the right tools.
by zepolen 10y ago
Python shouldn't be any slower than Node for crawling if you use the right tools.
- chrisabrams 10y agoOh boy. When I lived in San Francisco the Python community had this on their coffee mugs :O
- dickbasedregex 10y agoYour bottleneck should probably be managing your requests, not saving the document/assets. I'm dismissive that language speed is non-negligible in scrapping. Rate limiting and being smart about how you're fetching data should probably be your concern. ... sure, if you don't mind slamming a server with concurrent requests, your language choice might start to matter if your IP isn't blocked first.