4 ms·

does robots.txt apply retroactively though? That would be bad..

• selfhoster11 6y ago
  At one point, it did. I'm not sure about now.
  • bombcar 6y ago
    It was archive.org's attempt at automating "please remove X from the archive" but I don't think it was entirely successful and I think they may have stopped doing it. The idea was if you decided you didn't want /cats.html archived you'd robot.txt it, archive.org would notice, and delete the archive; but it also meant if the domain changes and robots.txt just blocked everything, everything would be lost.