5 ms·
I can confirm that, scraping Google instantly needs huge effort and money. In our best we can scrape 2500 SERP per IP. But i must say using proxy services and
by samcolvin 6y ago
I can confirm that, scraping Google instantly needs huge effort and money. In our best we can scrape 2500 SERP per IP.
But i must say using proxy services and other things did not helped us much. Because most of them were banned before we use.
- Fnoord 6y agoHow does it wotk out for IPv6?
- FreshFries 6y agoComplete /64's get blocked.
- core-questions 6y agoIPv6 is just not widely used, so when you do use it, you stick out like a sore thumb. Think like a bayesian: for Google, it's easy to just block whole /32s of IPv6 space.
- p1mrx 6y agoAt what point would you consider IPv6 "widely used"? It's currently 30% of traffic: https://www.google.com/intl/en/ipv6/statistics.html https://www.google.com/intl/en/ipv6/statistics.html
- core-questions 6y agoI bet you half of that is scrapers!
- core-questions 6y agoYeah, and then you recycle the IP back into the pool for the next guy to work with. An operation I know of was getting 6+ million SERPs a day, budget for proxies was hundreds of thousands a year.