6 ms·
Robots.txt is usually explicitly permission for a robot not to crawl. But a robot crawling your site and an archive, cache, or duplicate page are all different
by pbhjpbhj 10y ago
Robots.txt is usually explicitly permission for a robot not to crawl. But a robot crawling your site and an archive, cache, or duplicate page are all different propositions.
Google and others have enhanced robots.txt to enable permission for crawling (allow, sitemap), meta tags can deny archiving and various means allow permission to be explicitly denied for caching.
To use your analogy of raising a sign: if you don't put up a 'no trespassing' sign then it doesn't make trespassing legal.
FWIW I disprove of this state of affairs and consider copyright to be hugely defective in these respects.
>but they clearly can keep a copy //
It's nuanced but permission to access a page =/= permission to keep a copy. Just as you have explicit permission to access a video on YouTube but in most jurisdictions will not have permission to download it for later (commercial) use.
- Retric 10y agoNot having robots.txt is having no sign. So if you have the file you are putting up a sign that says something. Also, it's meaningless for a bot to get permission without having permission to make a copy. There are arguments around the number of copies, but the clear implication is at lest all the routers can make a copy.
- ghaff 10y ago>So if you have the file you are putting up a sign that says something. Not really. There are often default robots.txt files that the system just puts there in the course of building a default website. The legally "right" way to do things is only archiving sites that give explicit permission to do so. But then, for all intents and purposes, you can't have a web archive. So we have the current ask forgiveness rather than permission system which works fine most of the time for organizations like the IA and AT. But it does mean that someone like the IA is inclined to err on the side of removing content if someone objects.
- matt4077 10y agoIA has every right to scrape, save, and display. See US Code 108. Robots.txt is mostly meaningless in terms of the law, because it doesn't say anything about copying or distributing–only access, which is outside the scope of copyright.
- pbhjpbhj 10y agoI think you mean 17-USC-108. It might surprise you to know that the web extend outside USA and that Fair Use and exemptions for libraries and archives (aside: where's the definition of those two terms in the USC that's being used here?) don't extend to the World. IA may be legal in USA but a whole lot of the websites of the world reside on servers in other countries. There's a minor technical problem with that USC too, it seems. It allows archives to "reproduce no more than one copy of a work". But to compare a website you make a second [admittedly transient] copy to decide whether to re-archive. That's technically not within the scope of that 17USC108 accommodation AFAICT. This may have been solved in US law; I've a feeling there was a modification of EU law to allow transient/cache copies?
- Retric 10y agoThat might hold water if the robots.text where the default on every site ever used by a company, but change even a single bit on any of them and what's left is clearly your intent on all of them. Further, setting up a physical device connected to a public IP is never default behavior so you are putting up the sign in either case. So, at best your argument is someone athorised to do something put up a sign by mistake saying something that was not intended, but your intent has little relevance at that point. Worse, your argument is based on the assumption that nobody knew what was going on so even simple coursework mentioning robots.txt would demonstrate knowledge and thus intent through willful inaction.
- ghaff 10y ago>To use your analogy of raising a sign: if you don't put up a 'no trespassing' sign then it doesn't make trespassing legal. Right. And it's actually not a bad analogy as analogies go. Not having a sign doesn't make trespassing legal but if someone sometimes walks over a corner of your property and you go to the police to try to get him arrested, the first thing they'll probably ask you is if you have your property posted and/or if you bothered to ask him to stop. If the answer is no, they'll probably tell you to go away and do so and only come back if he ignores the sign.
- pbhjpbhj 10y agoYeah, you come down to breakfast and there's a random person sat on your sofa, you ring the police and they say "meh, you didn't put a sign up"? The requirement to post a sign to make trespassing an actionable offence is a USA thing AIUI, it's not a UK thing at least, but copyright is almost universal and doesn't require even adding a (c) mark, it's automatic at the point of creation under the Berne Convention. Or in other words you've pushed your analogy too far and hit a marked legal difference between USA physical property law enforcement and international intellectual property law.
- ghaff 10y agoClearly entering a house is much different from the specific example I gave of walking across a corner of a property. Also, a number of number of European countries have various variants of right to roam that make trespassing a less actionable offense than in the US. The more general principle is that if no harm is done and the individual/organization will just stop the action if you asked, the courts are often reluctant to get involved. There are exceptions of course, especially in the vein of making an example of someone to discourage others.