5 ms·
SEO companies are big customers of products like this. Scraping Google at scale requires a huge budget for proxies and IP blocks, and I believe they apply ML to
by core-questions 6y ago
SEO companies are big customers of products like this. Scraping Google at scale requires a huge budget for proxies and IP blocks, and I believe they apply ML to detect people abusing their service.
- samcolvin 6y agoI can confirm that, scraping Google instantly needs huge effort and money. In our best we can scrape 2500 SERP per IP. But i must say using proxy services and other things did not helped us much. Because most of them were banned before we use.
- Fnoord 6y agoHow does it wotk out for IPv6?
- FreshFries 6y agoComplete /64's get blocked.
- core-questions 6y agoIPv6 is just not widely used, so when you do use it, you stick out like a sore thumb. Think like a bayesian: for Google, it's easy to just block whole /32s of IPv6 space.
- p1mrx 6y agoAt what point would you consider IPv6 "widely used"? It's currently 30% of traffic: https://www.google.com/intl/en/ipv6/statistics.html https://www.google.com/intl/en/ipv6/statistics.html
- core-questions 6y agoI bet you half of that is scrapers!
- core-questions 6y agoYeah, and then you recycle the IP back into the pool for the next guy to work with. An operation I know of was getting 6+ million SERPs a day, budget for proxies was hundreds of thousands a year.
- anderspitman 6y agoI actually don't think I would need to make a ridiculous amount of requests. The project is a pinning service where people submit a video URL so they can do things like manage lists of the videos they watch (think goodreads for web videos). I need to scrape the video description and tags for each submitted URL. It would be nice to just do it from the user's browser, but the same origin policy won't allow that. A browser extension is the next obvious route, but that has its own issues. I was thinking to first attempt the request via CloudFlare workers, then fall back to a residential proxy service. No idea how usable workers would end up being for this sort of scraping.
- lwn 6y agoThis sounds like your would be better of using Youtube's API https://developers.google.com/youtube/v3/getting-started https://developers.google.com/youtube/v3/getting-started
- rafaelm 6y agoYes, I was going to mention this. I worked for a small company that had a YouTube product that retrieved video descriptions, duration, thumbnail, etc, using the API. We started with 100k daily API calls and that got later increased to 1 million after requesting it.
- anderspitman 6y agoLooks like the limit is 10k/day now. What did the process look like to get an increase?
- dylz 6y agoautomated form for me.
- anderspitman 6y agoNice. And that got you from 10k to 1M?