7 ms·
> Is it legal for a government entity to issue a robots.txt like that? I may be wrong (this isn't my area), but I was under the impression that robots.txt was
by bigwavedave 6y ago
> Is it legal for a government entity to issue a robots.txt like that?
I may be wrong (this isn't my area), but I was under the impression that robots.txt was just an unofficial convention? I'm not saying people should ignore robots.txt, but are there legal ramifications if ignored? I'm not asking about techniques sites use to discourage crawlers/scrapers, I'm specifically wondering if robots.txt has any legal weight.
- banana_giraffe 6y agoYep, it's defined at https://www.robotstxt.org/ https://www.robotstxt.org/ Looks like Google is trying to turn it into an RFC though.