3 ms·
Google respects robots.txt, so it might be hard to prove that they are accessing websites without their implied consent. Also, their own robots.txt contains "D
by skybrian 7y ago
Google respects robots.txt, so it might be hard to prove that they are accessing websites without their implied consent.
Also, their own robots.txt contains "Disallow: /search". So, there is arguably no inconsistency, either.
But, what does this new ruling mean for robots.txt?
- dclusin 7y agoI think OP is getting at the nature of the relationship is kinda imbalanced. Consider basically most of their website is off limits: https://www.google.com/robots.txt https://www.google.com/robots.txt
- nojvek 7y agoyou can have your website offlimits from google by having a robots.txt too. What's the problem? People willingly want google to index them so they appear in search results and google can send them traffic.