3 ms·

Sure, go join the realms of shady SEOs and malware, if I want to really stop you I'll know you're not coming from a Google IP range. https://www.incapsula.com/b

by cube00 10y ago

Sure, go join the realms of shady SEOs and malware, if I want to really stop you I'll know you're not coming from a Google IP range. https://www.incapsula.com/blog/was-that-really-a-google-bot-crawling-my-site.html https://www.incapsula.com/blog/was-that-really-a-google-bot-... However, consider what your ultimate end game is, if it's a website you expect visitors to find through Google or the Play store, good luck once web masters start reporting your misbehaving "Googlebot" crawler.

lend000 10y ago
Unless you do it from a Google Cloud instance, that is.
- mootothemax 10y ago
  >Unless you do it from a Google Cloud instance, that is. What's the reverse DNS for Google Cloud IPs? Google says to check that Googlebot's IP resolves to either a .google.com or .googlebot.com domain. https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en
  - hrrsn 10y ago
    .googleusercontent.com
  - lend000 10y ago
    Good point -- although it makes sense why it isn't frequently implemented. DNS lookups aren't cheap for this kind of thing.
    - Symbiote 10y ago
      It would be sufficient to let some requests come through from "Googlebot", and then deal with them (block, rate-limit, whatever) once the DNS check has been completed.
  - milankragujevic 10y ago
    Couldn't you use GWT Mobilizer to scrape a site then index that? Like this: http://i.imgur.com/ocR54Yq.jpg http://i.imgur.com/ocR54Yq.jpg
walshemj 10y ago
You know you have to be able to crawl your own or client sites using screaming frog or deep crawl as google to identify any crawl/ crawl budget issues.