5 ms·

they only ever needed to honor the robots.txt at the date of archival. archive.org fucked up by making robots retroactive, if they used the archived robots.txt

they only ever needed to honor the robots.txt at the date of archival. archive.org fucked up by making robots retroactive, if they used the archived robots.txt as a filter for a site at the relevant date, they'd have had the best of both worlds - respecting how sites work without losing how sites appeared at a date.

deleted 10y ago
[deleted]
pbhjpbhj 10y ago
They'd be the flouting copyright laws, like Google do, but nonetheless tortuously. They're making a copy, which is already an infringement, distributing it seemingly against the owners express wishes is treated as a crime in some jurisdictions.
- Retric 10y ago
  Robots.TXT is explicit permission to make a copy, otherwise crawling is meaningless and that is not nessisarily reversible. Like putting up a yard sale sign, then trying to get the poeple that show up yesterday arrested for trespassing. What the archive can do after that point is a different issue, but they clearly can keep a copy. Further, someone else is using the domain they don't nessisarily have anything to do with the archived data.
  - reitanqild 10y ago
    Anyone knows why this was downvoted?
  - pbhjpbhj 10y ago
    Robots.txt is usually explicitly permission for a robot not to crawl. But a robot crawling your site and an archive, cache, or duplicate page are all different propositions. Google and others have enhanced robots.txt to enable permission for crawling (allow, sitemap), meta tags can deny archiving and various means allow permission to be explicitly denied for caching. To use your analogy of raising a sign: if you don't put up a 'no trespassing' sign then it doesn't make trespassing legal. FWIW I disprove of this state of affairs and consider copyright to be hugely defective in these respects. >but they clearly can keep a copy // It's nuanced but permission to access a page =/= permission to keep a copy. Just as you have explicit permission to access a video on YouTube but in most jurisdictions will not have permission to download it for later (commercial) use.
    - Retric 10y ago
      Not having robots.txt is having no sign. So if you have the file you are putting up a sign that says something. Also, it's meaningless for a bot to get permission without having permission to make a copy. There are arguments around the number of copies, but the clear implication is at lest all the routers can make a copy.
      - ghaff 10y ago
        >So if you have the file you are putting up a sign that says something. Not really. There are often default robots.txt files that the system just puts there in the course of building a default website. The legally "right" way to do things is only archiving sites that give explicit permission to do so. But then, for all intents and purposes, you can't have a web archive. So we have the current ask forgiveness rather than permission system which works fine most of the time for organizations like the IA and AT. But it does mean that someone like the IA is inclined to err on the side of removing content if someone objects.
        matt4077 10y ago
        IA has every right to scrape, save, and display. See US Code 108. Robots.txt is mostly meaningless in terms of the law, because it doesn't say anything about copying or distributing–only access, which is outside the scope of copyright.
        pbhjpbhj 10y ago
        I think you mean 17-USC-108. It might surprise you to know that the web extend outside USA and that Fair Use and exemptions for libraries and archives (aside: where's the definition of those two terms in the USC that's being used here?) don't extend to the World. IA may be legal in USA but a whole lot of the websites of the world reside on servers in other countries. There's a minor technical problem with that USC too, it seems. It allows archives to "reproduce no more than one copy of a work". But to compare a website you make a second [admittedly transient] copy to decide whether to re-archive. That's technically not within the scope of that 17USC108 accommodation AFAICT. This may have been solved in US law; I've a feeling there was a modification of EU law to allow transient/cache copies?
- matt4077 10y ago
  The law has specific exemptions for archival and search, which google and IA use. Do you really think they could do what they're doing when any one of millions of people could just take them to court at any time?
  - ghaff 10y ago
    Well, they have been sued, e.g. https://arstechnica.com/uncategorized/2006/08/7634/ https://arstechnica.com/uncategorized/2006/08/7634/ and they settled. But they're not a target that you're going to collect big from, as a non-profit archive/library they're sympathetic whether or not that gives them any special legal standing, and they'll basically take down your content past and present if you ask them to. So it will almost certainly cost you money to sue them, you won't collect much in the best case, and you can get your content taken down in about as much time as it would take you to pick a lawyer out of the phone book.
  - pbhjpbhj 10y ago
    So my website is on a server outside the USA, what exemption are they relying on?

deleted 10y ago

[deleted]

pbhjpbhj 10y ago

They'd be the flouting copyright laws, like Google do, but nonetheless tortuously. They're making a copy, which is already an infringement, distributing it seemingly against the owners express wishes is treated as a crime in some jurisdictions.

Retric 10y ago
Robots.TXT is explicit permission to make a copy, otherwise crawling is meaningless and that is not nessisarily reversible. Like putting up a yard sale sign, then trying to get the poeple that show up yesterday arrested for trespassing. What the archive can do after that point is a different issue, but they clearly can keep a copy. Further, someone else is using the domain they don't nessisarily have anything to do with the archived data.
- reitanqild 10y ago
  Anyone knows why this was downvoted?
- pbhjpbhj 10y ago
  Robots.txt is usually explicitly permission for a robot not to crawl. But a robot crawling your site and an archive, cache, or duplicate page are all different propositions. Google and others have enhanced robots.txt to enable permission for crawling (allow, sitemap), meta tags can deny archiving and various means allow permission to be explicitly denied for caching. To use your analogy of raising a sign: if you don't put up a 'no trespassing' sign then it doesn't make trespassing legal. FWIW I disprove of this state of affairs and consider copyright to be hugely defective in these respects. >but they clearly can keep a copy // It's nuanced but permission to access a page =/= permission to keep a copy. Just as you have explicit permission to access a video on YouTube but in most jurisdictions will not have permission to download it for later (commercial) use.
  - Retric 10y ago
    Not having robots.txt is having no sign. So if you have the file you are putting up a sign that says something. Also, it's meaningless for a bot to get permission without having permission to make a copy. There are arguments around the number of copies, but the clear implication is at lest all the routers can make a copy.
    - ghaff 10y ago
      >So if you have the file you are putting up a sign that says something. Not really. There are often default robots.txt files that the system just puts there in the course of building a default website. The legally "right" way to do things is only archiving sites that give explicit permission to do so. But then, for all intents and purposes, you can't have a web archive. So we have the current ask forgiveness rather than permission system which works fine most of the time for organizations like the IA and AT. But it does mean that someone like the IA is inclined to err on the side of removing content if someone objects.
      - matt4077 10y ago
        IA has every right to scrape, save, and display. See US Code 108. Robots.txt is mostly meaningless in terms of the law, because it doesn't say anything about copying or distributing–only access, which is outside the scope of copyright.
        pbhjpbhj 10y ago
        I think you mean 17-USC-108. It might surprise you to know that the web extend outside USA and that Fair Use and exemptions for libraries and archives (aside: where's the definition of those two terms in the USC that's being used here?) don't extend to the World. IA may be legal in USA but a whole lot of the websites of the world reside on servers in other countries. There's a minor technical problem with that USC too, it seems. It allows archives to "reproduce no more than one copy of a work". But to compare a website you make a second [admittedly transient] copy to decide whether to re-archive. That's technically not within the scope of that 17USC108 accommodation AFAICT. This may have been solved in US law; I've a feeling there was a modification of EU law to allow transient/cache copies?
matt4077 10y ago
The law has specific exemptions for archival and search, which google and IA use. Do you really think they could do what they're doing when any one of millions of people could just take them to court at any time?
- ghaff 10y ago
  Well, they have been sued, e.g. https://arstechnica.com/uncategorized/2006/08/7634/ https://arstechnica.com/uncategorized/2006/08/7634/ and they settled. But they're not a target that you're going to collect big from, as a non-profit archive/library they're sympathetic whether or not that gives them any special legal standing, and they'll basically take down your content past and present if you ask them to. So it will almost certainly cost you money to sue them, you won't collect much in the best case, and you can get your content taken down in about as much time as it would take you to pick a lawyer out of the phone book.
- pbhjpbhj 10y ago
  So my website is on a server outside the USA, what exemption are they relying on?