4 ms·

And they are one robots away from cancellation. For all the good they do, retroactively applying robots exclusions to their crawler is a terrible thing. Luckily

And they are one robots away from cancellation. For all the good they do, retroactively applying robots exclusions to their crawler is a terrible thing. Luckily there are alternatives for going forward.

slededit 8y ago
They keep the data and just don’t display it. The last thing they need is a court order demanding they delete it. Sure the archive is useful today, but it’s primary purpose is retaining information for future generations. If that means placating copyright holders it’s worth the cost.
- rovr138 8y ago
  > They keep the data and just don’t display it. I’ve read about the robots.txt mentioned before but hadn’t seen this mentioned. Any idea if they have this somewhere on the site?
  - slededit 8y ago
    Sure see here: https://archive.org/post/133690/robotstxt-only-gives-temporary-removal-of-pages https://archive.org/post/133690/robotstxt-only-gives-tempora...
- dingaling 8y ago
  > If that means placating copyright holders it’s worth the cost. In many cases it's nothing to do with the copyright holder. In fact, the complete opposite: someone who subsequently bought the domain and unwittingly stuck a generic robots.txt on the site. If the IA hid content upon receiving a request from webmaster@domain then that'd make sense. But doing it automatically and retrospectively from robots.txt is bizarre and shows again the dangers of centralisation on the Internet. I now block IA from my sites as a protest. We need more competition and fresh ideas in digital archiving.
dredmorbius 8y ago
IA are ignoring robots.txt as of earlier this year.
- teddyh 8y ago
  Really? Reference, please!
  - duckington 8y ago
    https://boingboing.net/2017/04/22/internet-archive-to-ignore-rob.html https://boingboing.net/2017/04/22/internet-archive-to-ignore... They are gradually expanding the scope of sites where robots.txt is ignored, perhaps "testing the water".
  - dredmorbius 8y ago
    A few months ago we stopped referring to robots.txt files on U.S. government and military web sites for both crawling and displaying web pages (though we respond to removal requests sent to info@archive.org). As we have moved towards broader access it has not caused problems, which we take as a good sign. We are now looking to do this more broadly. https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

slededit 8y ago

They keep the data and just don’t display it. The last thing they need is a court order demanding they delete it. Sure the archive is useful today, but it’s primary purpose is retaining information for future generations. If that means placating copyright holders it’s worth the cost.

rovr138 8y ago
> They keep the data and just don’t display it. I’ve read about the robots.txt mentioned before but hadn’t seen this mentioned. Any idea if they have this somewhere on the site?
- slededit 8y ago
  Sure see here: https://archive.org/post/133690/robotstxt-only-gives-temporary-removal-of-pages https://archive.org/post/133690/robotstxt-only-gives-tempora...
dingaling 8y ago
> If that means placating copyright holders it’s worth the cost. In many cases it's nothing to do with the copyright holder. In fact, the complete opposite: someone who subsequently bought the domain and unwittingly stuck a generic robots.txt on the site. If the IA hid content upon receiving a request from webmaster@domain then that'd make sense. But doing it automatically and retrospectively from robots.txt is bizarre and shows again the dangers of centralisation on the Internet. I now block IA from my sites as a protest. We need more competition and fresh ideas in digital archiving.

dredmorbius 8y ago

IA are ignoring robots.txt as of earlier this year.

teddyh 8y ago
Really? Reference, please!
- duckington 8y ago
  https://boingboing.net/2017/04/22/internet-archive-to-ignore-rob.html https://boingboing.net/2017/04/22/internet-archive-to-ignore... They are gradually expanding the scope of sites where robots.txt is ignored, perhaps "testing the water".
- dredmorbius 8y ago
  A few months ago we stopped referring to robots.txt files on U.S. government and military web sites for both crawling and displaying web pages (though we respond to removal requests sent to info@archive.org). As we have moved towards broader access it has not caused problems, which we take as a good sign. We are now looking to do this more broadly. https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...