3 ms·

Any archival purposes should use the Internet Archive; Google's cache couldn't be counted on to remain present and unchanged, even before this. (The Internet A

Any archival purposes should use the Internet Archive; Google's cache couldn't be counted on to remain present and unchanged, even before this. (The Internet Archive can't be counted on to remain unchanged either, but at least the Internet Archive will either remain unchanged or disappear, never change. I wish archive.org didn't auto-nuke entire sites based on robots.txt, because when domains disappear the domain squatters often seem to use robots.txt files for some reason.)

dredmorbius 3y ago
IA have increasingly ignored robots.txt since the mid-2010s, going on at least eight years now: <https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...> <https://blog.archive.org/2016/12/17/robots-txt-gov-mil-websites/ https://blog.archive.org/2016/12/17/robots-txt-gov-mil-websi...>
- JoshTriplett 3y ago
  That's excellent news, thank you!
deleted 3y ago
[deleted]