3 ms·
I like python for the ease of use and scraping is I/O bound anyways so there's no pressure to switch to a more performant language.
by monkeybutton 6y ago
I like python for the ease of use and scraping is I/O bound anyways so there's no pressure to switch to a more performant language.
- RhodesianHunter 6y agoI'd say that really depends on your scale and what you're doing with the content you scrape. In my experience with large scale scraping you're much better off using something like Java where you can more easily have a thread pool with thousands of threads (or better yet, Kotlin coroutines) handling the crawling itself and a *NUM CORES thread pool handling CPU bound tasks like parsing.
- monkeybutton 6y agoCould you give a ballpark figure for what you mean by large scale scraping? I've only worked on a couple projects, one was a broad (100K to 500K domains) and shallow (root + 1 level of page depth, also with a low cap on the number of children pages). The other just a single domain but scraping around 50K pages from it.
- RhodesianHunter 6y agoMy experience was with e-commerce scraping. Not many domains, but a massive catalogue.
- tluyben2 6y agoI would say millions of domains regularly. That's where the pricing of most 'scraping services' falls down too compared to just doing it yourself.