3 ms·
Is it legal for a government entity to issue a robots.txt like that? Maybe the line between use and abuse hasn't been delinated as well as it needs to be.
by adolph 6y ago
Is it legal for a government entity to issue a robots.txt like that? Maybe the line between use and abuse hasn't been delinated as well as it needs to be.
- bigwavedave 6y ago> Is it legal for a government entity to issue a robots.txt like that? I may be wrong (this isn't my area), but I was under the impression that robots.txt was just an unofficial convention? I'm not saying people should ignore robots.txt, but are there legal ramifications if ignored? I'm not asking about techniques sites use to discourage crawlers/scrapers, I'm specifically wondering if robots.txt has any legal weight.
- banana_giraffe 6y agoYep, it's defined at https://www.robotstxt.org/ https://www.robotstxt.org/ Looks like Google is trying to turn it into an RFC though.
- vageli 6y agoIs failure to honor a robots.txt a crime? Or rather, would it be unlawful to spoof a user agent to access this publicly available data? After the linkedin [0] case it seems reasonable to think not. [0]: https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-linkedin-protects-scraping-public-data https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l...
- Spivak 6y agoSpoofing user-agents hasn't worked in a long time for anything but small operations because search engines publish specific IP ranges their scrapers use.
- cryptonector 6y agoThe CFAA is so broad and broadly interpreted that I would assume that failure to honor any any site's robots.txt file may incur criminal liability if the U.S. government can claim American jurisdiction (e.g., because the site's owners are U.S. persons or a U.S. corporation, or because the site's servers are located in the U.S.).