3 ms·

How do you block their IP? Google has hundreds of numbers and AFAIK they don’t even publish all of them.

edm0nd 7y ago
You can block Googlebot by User-Agent and IP addresses.
bhartzer 7y ago
There are services that track google IPs, those doing black hat SEO and cloaking rely on them. Google is going to crawl, you just can’t stop them from crawling even if you tell them not to in robots.txt. Best option at this point is to add a noindex tag on pages and let them crawl to see it.
- LunaSea 7y ago
  Wouldn't that be considered illegal due to the same reasons as scraping?
  - hnick 7y ago
    I don't think scraping is illegal, unless you're hitting protected pages behind a login after agreeing to a Terms of Service, or hitting it so often you'd constitute a DOS attack. An implied TOS notice on a separate page or footer is not considered enforceable last time I checked, likewise for robots.txt.
  - pxtail 7y ago
    Lol, of course not - it would be (and will be!) illegal if you would be doing it, google can crawl and scrape whatever it wants.
echeese 7y ago
Verifying Googlebot: https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en
LinuxBender 7y ago
As others mentioned, you can block their user-agent. That will stop most crawling. They have some bots that use browser agents to see if you serve different content to non-google bots. To block those, you can feed their IP's to haproxy or iptables ipset. fetch_google() { for line in $(dig +short txt _cloud-netblocks.googleusercontent.com | tr " " "\n" | grep include | cut -f 2 -d :) do dig +short txt "${line}" done | tr " " "\n" | grep ip4 | cut -f 2 -d : | sort -n | uniq | xz -9ecv > _GOOGLE.netset.xz }; In HAProxy, you could include the decompressed version of that IP list with something like acl BOTS src -f bot.txt Then either redirect, reject, silent-drop.
- StudentStuff 7y ago
  Any clue if Google crawls from IPv6 addresses?
  - LinuxBender 7y ago
    They will crawl ipv4 and ipv6, whichever your DNS advertises for your sites.

You can block Googlebot by User-Agent and IP addresses.

There are services that track google IPs, those doing black hat SEO and cloaking rely on them. Google is going to crawl, you just can’t stop them from crawling even if you tell them not to in robots.txt. Best option at this point is to add a noindex tag on pages and let them crawl to see it.

LunaSea 7y ago
Wouldn't that be considered illegal due to the same reasons as scraping?
- hnick 7y ago
  I don't think scraping is illegal, unless you're hitting protected pages behind a login after agreeing to a Terms of Service, or hitting it so often you'd constitute a DOS attack. An implied TOS notice on a separate page or footer is not considered enforceable last time I checked, likewise for robots.txt.
- pxtail 7y ago
  Lol, of course not - it would be (and will be!) illegal if you would be doing it, google can crawl and scrape whatever it wants.

echeese 7y ago

Verifying Googlebot: https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en

LinuxBender 7y ago

As others mentioned, you can block their user-agent. That will stop most crawling. They have some bots that use browser agents to see if you serve different content to non-google bots. To block those, you can feed their IP's to haproxy or iptables ipset. fetch_google() { for line in $(dig +short txt _cloud-netblocks.googleusercontent.com | tr " " "\n" | grep include | cut -f 2 -d :) do dig +short txt "${line}" done | tr " " "\n" | grep ip4 | cut -f 2 -d : | sort -n | uniq | xz -9ecv > _GOOGLE.netset.xz }; In HAProxy, you could include the decompressed version of that IP list with something like acl BOTS src -f bot.txt Then either redirect, reject, silent-drop.

StudentStuff 7y ago
Any clue if Google crawls from IPv6 addresses?
- LinuxBender 7y ago
  They will crawl ipv4 and ipv6, whichever your DNS advertises for your sites.