5 ms·

You're not getting "caught", and Google can fuck right off with any "terms of service" that I never agreed to about how I respond when they scrape my site witho

You're not getting "caught", and Google can fuck right off with any "terms of service" that I never agreed to about how I respond when they scrape my site without permission. I know that there are practical concerns here, but Google has done a masterful job of shaping the conversation so that many people think anything not sanctioned by Almighty Google with regard to SEO (let alone actual "blackhat" tactics) is illegal or immoral.

squeaky-clean 9y ago
It's not illegal, but if you're hiding content from Googlebot because you don't want it to derank that page, once they crawl it with an agent that hides the fact that it's Google, they will then derank your whole website. So it's not illegal, but if you think you're beating Google by doing it, they're well aware of this workaround and will punish you harder. If you don't care about ranking, then no biggie. But why would you manipulate your web server like that if you weren't trying to rank.
Karunamon 9y ago
You’re giving permission to scrape by way of having a page accessible to the general public. If googlebot is bothering you so much, install a two line robots.txt and it’ll go away.
- jiggunjer 9y ago
  robots can ignore your /robots.txt.
  - Karunamon 9y ago
    Googlebot, and every other search engine robot of note. Putting a page up on the public internet includes the risk that it may be visited.
falcor84 9y ago
>scrape my site without permission. No one needs permission to request a web page, and that includes bots. But if that still bothers you, why not create arobots.txt with your preferences?