3 ms·

Absolutely agreed. I work on Wikipedia articles on a specific subject where we rely on a select few web resources. Many of them have long since closed down and

Absolutely agreed. I work on Wikipedia articles on a specific subject where we rely on a select few web resources. Many of them have long since closed down and we use the archived versions instead. It's downright tragic when we've lost complete websites, perfectly usable sources for hundreds of articles with quality content, all because some domain parker brought up the URL and added a robots.txt that retroactively disabled the existing archive for the site. We need to archive everything, regardless of what the site owner thinks crawlers shouldn't see. Years down the road we might actually wish we had archives of things that many found uninteresting or not meant to be archived. (for example to see how sitemaps were set up or RSS feeds)

Walf 10y ago
Isn't this a problem with the archiving logic? Why on Earth would one apply robots.txt rules retroactively?
- deleted 10y ago
  [deleted]
- detaro 10y ago
  They need some way for people to easily hide archived content, since they have no clear right to publish their copy. You can send them an e-mail instead, but those take comparatively more resources to process and robots.txt at least sort-of indicates that you actually are in a position where you can demand the removal and can be processed automatically.
  - CM30 10y ago
    Wouldn't then the best logic be to finegrain this system a bit more? So blocking the archive blocks new content added after robots.txt is made available, and adding a certain few lines in the robots.txt file explicitly hides older content on the domain in the archive. That way, everyone wins. Sites that really want to remove content for some insane reason can do so, older sites usually aren't lost if the domain expires and domain holding page owners/cybersquatters don't accidentally cause older content to be hidden (since hey, they don't really want to block it, just stop their holding page from being archived).
dTal 10y ago
I think it's quite a clever way of dealing with the problem. Archive.org takes the long view - they don't delete data merely because someone changed a robots.txt, they merely hide it. So the data is only inaccessible as long as someone cares enough to specifically hide it. At the same time this covers their ass, copyright-wise.

Walf 10y ago

Isn't this a problem with the archiving logic? Why on Earth would one apply robots.txt rules retroactively?

deleted 10y ago
[deleted]
detaro 10y ago
They need some way for people to easily hide archived content, since they have no clear right to publish their copy. You can send them an e-mail instead, but those take comparatively more resources to process and robots.txt at least sort-of indicates that you actually are in a position where you can demand the removal and can be processed automatically.
- CM30 10y ago
  Wouldn't then the best logic be to finegrain this system a bit more? So blocking the archive blocks new content added after robots.txt is made available, and adding a certain few lines in the robots.txt file explicitly hides older content on the domain in the archive. That way, everyone wins. Sites that really want to remove content for some insane reason can do so, older sites usually aren't lost if the domain expires and domain holding page owners/cybersquatters don't accidentally cause older content to be hidden (since hey, they don't really want to block it, just stop their holding page from being archived).

dTal 10y ago

I think it's quite a clever way of dealing with the problem. Archive.org takes the long view - they don't delete data merely because someone changed a robots.txt, they merely hide it. So the data is only inaccessible as long as someone cares enough to specifically hide it. At the same time this covers their ass, copyright-wise.