4 ms·
Just to be clear, there is a mistake in this post. Adding a noindex tag will cause google to not display the page in their search results. They will still crawl
by bhartzer 7y ago
Just to be clear, there is a mistake in this post. Adding a noindex tag will cause google to not display the page in their search results. They will still crawl ( they have to), in order to see that directive.
Also, keep in mind that google has recently changed the way they deal with robots.txt. So they only way to do what the author is suggesting is to add the noindex meta tag and let them crawl. Once they are removed from the index, block the google IPs.
- joering2 7y agoHow do you block their IP? Google has hundreds of numbers and AFAIK they don’t even publish all of them.
- edm0nd 7y agoYou can block Googlebot by User-Agent and IP addresses.
- bhartzer 7y agoThere are services that track google IPs, those doing black hat SEO and cloaking rely on them. Google is going to crawl, you just can’t stop them from crawling even if you tell them not to in robots.txt. Best option at this point is to add a noindex tag on pages and let them crawl to see it.
- LunaSea 7y agoWouldn't that be considered illegal due to the same reasons as scraping?
- hnick 7y agoI don't think scraping is illegal, unless you're hitting protected pages behind a login after agreeing to a Terms of Service, or hitting it so often you'd constitute a DOS attack. An implied TOS notice on a separate page or footer is not considered enforceable last time I checked, likewise for robots.txt.
- pxtail 7y agoLol, of course not - it would be (and will be!) illegal if you would be doing it, google can crawl and scrape whatever it wants.
- echeese 7y agoVerifying Googlebot: https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en
- LinuxBender 7y agoAs others mentioned, you can block their user-agent. That will stop most crawling. They have some bots that use browser agents to see if you serve different content to non-google bots. To block those, you can feed their IP's to haproxy or iptables ipset. fetch_google() { for line in $(dig +short txt _cloud-netblocks.googleusercontent.com | tr " " "\n" | grep include | cut -f 2 -d :) do dig +short txt "${line}" done | tr " " "\n" | grep ip4 | cut -f 2 -d : | sort -n | uniq | xz -9ecv > _GOOGLE.netset.xz }; In HAProxy, you could include the decompressed version of that IP list with something like acl BOTS src -f bot.txt Then either redirect, reject, silent-drop.
- StudentStuff 7y agoAny clue if Google crawls from IPv6 addresses?
- LinuxBender 7y agoThey will crawl ipv4 and ipv6, whichever your DNS advertises for your sites.
- 1shooner 7y agoI've found that google will still include the page in their search index if it's been discovered by means other than crawling the resource itself (e.g. it's linked from other pages). The search result will simply not include any content from the page, since they've presumably not crawled the page.
- liberte82 7y agoif noindex == true { content = "" } Of course they still have it indexed in their own systems .... would you trust Google not to.
- dazc 7y agoShould read: 'Dear Google, I'm giving you full access to my website but I'm asking you not to index it.'
- popee 7y agoYeah, just reject any access from their subnets