2 ms·

Domain parking often vandalises the Archive record for a domain with their robots.txt. It's far better that it's recorded in multiple places plus archive.org

• fouc 6y ago
  does robots.txt apply retroactively though? That would be bad..
  • selfhoster11 6y ago
    At one point, it did. I'm not sure about now.
    • bombcar 6y ago
      It was archive.org's attempt at automating "please remove X from the archive" but I don't think it was entirely successful and I think they may have stopped doing it. The idea was if you decided you didn't want /cats.html archived you'd robot.txt it, archive.org would notice, and delete the archive; but it also meant if the domain changes and robots.txt just blocked everything, everything would be lost.