3 ms·

The web archive will retroactively remove anything they've archived from your domain if you disallow "ia_archiver" in your robots.txt file (and obviously, will

• there 18y ago
  just to clarify because i've had to deal with this before, excluding ia_archiver from your robots.txt will prevent archive.org from indexing new versions and deny access to your old versions, but it will not make them delete any archived material from your site that already exists. when you visit an archive.org page, their robot will fetch that site's robots.txt file in real-time (with some basic caching mechanism, i'd assume) and check for an exclusion. if your site ever goes down after the point you block their robot, or you let your domain name expire, or some squatter buys your domain after you're done with it and doesn't put up a robots.txt file, or anything else that would prevent their site from reading your robots.txt at the time someone tries to view the archive, they will allow full access to your site archives.