4 ms·

Cache: directive is very useful for many circumstances, especially for information archiving purpose, what a shame Google even want to cancel it.

• JoshTriplett 3y ago
  Any archival purposes should use the Internet Archive; Google's cache couldn't be counted on to remain present and unchanged, even before this. (The Internet Archive can't be counted on to remain unchanged either, but at least the Internet Archive will either remain unchanged or disappear, never change. I wish archive.org didn't auto-nuke entire sites based on robots.txt, because when domains disappear the domain squatters often seem to use robots.txt files for some reason.)
  • dredmorbius 3y ago
    IA have increasingly ignored robots.txt since the mid-2010s, going on at least eight years now: <https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...> <https://blog.archive.org/2016/12/17/robots-txt-gov-mil-websites/ https://blog.archive.org/2016/12/17/robots-txt-gov-mil-websi...>
    • JoshTriplett 3y ago
      That's excellent news, thank you!
  • deleted 3y ago
    [deleted]