4 ms·
There’s one technique that can be very useful in some circumstances that isn’t mentioned. Put simply, some sites try to block all bots except for those from the
by curun1r 5y ago
There’s one technique that can be very useful in some circumstances that isn’t mentioned. Put simply, some sites try to block all bots except for those from the major search engines. They don’t want their content scraped, but they want the traffic that comes from search. In those cases, it’s often possible to scrape the search engines instead using specialized queries designed to get the content you want into the blurb for each search result.
This kind of indirect scraping can be useful for getting almost all the information you want from sites like LinkedIn that do aggressive scraping detection.
- amelius 5y agoBut won't the search engines block you after some limit has been reached?
- curun1r 5y agoEventually, but they’re not very aggressive when it comes to bot detection. Simple IP rotation usually works.
- marginalia_nu 5y agoI ended up having to put my search engine behind a CDN and CAPTCHA every new IP because I got something like 30,000 search requests like this _per_hour_ from a botnet. They seem to have backed off now but they were really quite persistent for a while.
- tomrod 5y agoSome.
- deleted 5y ago[deleted]
- janmo 5y agoOr, you can spoof the google bot or Bing bot user agent and try to scrape the site that way.
- notriddle 5y agoImpersonating Googlebot is a great way to get blocked. Real Googlebot only comes from certain IP addresses.
- AlexanderTheGr8 5y agoalmost all sites uses "host" command to see if the IP belongs to the googlebot (this is recommended way by Google). And you can easily find many IPs which return googlebot when you "host" them. So you can use any of these IP addresses to spoof GoogleBot. To find more IPs, make your own website, and wait until GoogleBot eventually shows up :)