3 ms·
There are services that track google IPs, those doing black hat SEO and cloaking rely on them. Google is going to crawl, you just can’t stop them from crawling
by bhartzer 7y ago
There are services that track google IPs, those doing black hat SEO and cloaking rely on them.
Google is going to crawl, you just can’t stop them from crawling even if you tell them not to in robots.txt. Best option at this point is to add a noindex tag on pages and let them crawl to see it.
- LunaSea 7y agoWouldn't that be considered illegal due to the same reasons as scraping?
- hnick 7y agoI don't think scraping is illegal, unless you're hitting protected pages behind a login after agreeing to a Terms of Service, or hitting it so often you'd constitute a DOS attack. An implied TOS notice on a separate page or footer is not considered enforceable last time I checked, likewise for robots.txt.
- pxtail 7y agoLol, of course not - it would be (and will be!) illegal if you would be doing it, google can crawl and scrape whatever it wants.