5 ms·
Shouldn't that be caught by anti-trust laws? Google does not pay websites for scraping, so anyone should be entitled to scrap Google to their hearts content.
by intricatedetail 5y ago
Shouldn't that be caught by anti-trust laws? Google does not pay websites for scraping, so anyone should be entitled to scrap Google to their hearts content.
- namdnay 5y agoWebsites can choose whether to be scraped or not, and google respects that choice
- hire_charts 5y agoIt's nice of google to respect the choice to effectively become undiscoverable to anyone searching for your product.
- ben_w 5y agoYou sound like you have a third opinion, one which would allow a website to appear on Google in response to user searches without Google knowing anything about what the website contains?
- DangitBobby 5y agoNo, they are sarcastically indicating that you don't really have a choice, because if you aren't indexed by Google, practically speaking, you might as well not have a website.
- k4rli 5y agoTo be fair if being scraped were opt-in (Robots Inclusion Standard?) instead of opt-out, we might have a completely different technological world. Who knows.
- intricatedetail 5y agoBy creating an account and using their webmaster tools giving up even more of your data?
- mitchdoogle 5y agoNo you just have to put a noindex header on requests to your website
- intricatedetail 5y agoSo they still can use up my resources?
- Google234 5y agoHow can they know without looking? Again, you could also use the webmasters tools to block it...
- uglygoblin 5y agoRobots.txt can accomplish the same thing which means they periodically grab a single text file from you.