4 ms·
Could you give a ballpark figure for what you mean by large scale scraping? I've only worked on a couple projects, one was a broad (100K to 500K domains) and sh
by monkeybutton 6y ago
Could you give a ballpark figure for what you mean by large scale scraping? I've only worked on a couple projects, one was a broad (100K to 500K domains) and shallow (root + 1 level of page depth, also with a low cap on the number of children pages). The other just a single domain but scraping around 50K pages from it.
- RhodesianHunter 6y agoMy experience was with e-commerce scraping. Not many domains, but a massive catalogue.
- tluyben2 6y agoI would say millions of domains regularly. That's where the pricing of most 'scraping services' falls down too compared to just doing it yourself.