3 ms·

Your mind is gonna be blown when you discover the Internet Archive's Wayback Machine! https://web.archive.org/web/20080310221514/http://www.masnick.com/2007/09

• selfhoster11 5y ago
  Domain parking often vandalises the Archive record for a domain with their robots.txt. It's far better that it's recorded in multiple places plus archive.org
  • fouc 5y ago
    does robots.txt apply retroactively though? That would be bad..
    • selfhoster11 5y ago
      At one point, it did. I'm not sure about now.
      • bombcar 5y ago
        It was archive.org's attempt at automating "please remove X from the archive" but I don't think it was entirely successful and I think they may have stopped doing it. The idea was if you decided you didn't want /cats.html archived you'd robot.txt it, archive.org would notice, and delete the archive; but it also meant if the domain changes and robots.txt just blocked everything, everything would be lost.