3 ms·
It undoubtedly communicates some intention by the site owner about how the work on the site can be used, so probably?
by SiempreViernes 9y ago
It undoubtedly communicates some intention by the site owner about how the work on the site can be used, so probably?
- amelius 9y agoIt only communicates the intention if the reader is actually obliged to read and interpret the file as a legal document. If I put a file named "legal.txt" in an online folder, is anyone required to read it and act upon it? It might as well be a file intended for some completely unrelated purpose; e.g. a lawyer that put some drafts online, or for all the reader knows, it might even be part of a movie script.
- pseudalopex 9y agorobots.txt has been a de facto standard for over 20 years. Someone might be able to claim ignorance, but the Internet Archive has shown that they know about it. It has a specific format; if it can be parsed, it's safe to assume that it isn't part of a movie script. In most cases, copyright law requires the reader of a document not to republish it, so the robots.txt standard is actually much more permissive.