The policy the Internet Archive applies re: "robots.txt" comes from an archive policy created at U.C. Berkeley in the early 2000's (The Oakland Archive Policy -

The policy the Internet Archive applies re: "robots.txt" comes from an archive policy created at U.C. Berkeley in the early 2000's (The Oakland Archive Policy - http://www2.sims.berkeley.edu/research/conferences/aps/removal-policy.html http://www2.sims.berkeley.edu/research/conferences/aps/remov...). Jason Scott (an employee of the Internet Archive) mentioned that the Archive doesn't ever delete anything. He stated that items may be removed from public access because of changes to "robots.txt" but they're not actually deleted. (That's a little comforting, at least.)