archive.org is awesome - however there is one thing that bothers me about the indexing service, and I am not sure how it could be improved. The problem is that

archive.org is awesome - however there is one thing that bothers me about the indexing service, and I am not sure how it could be improved. The problem is that if a site currently has a robots.txt blocking indexing, the whole history of that site disappears with it. On the one hand, this makes sense - what if some webmaster forgot to prevent indexing and then adds it, with the original intention to not share any of the sites work. On the other hand, this prevents me from finding a site I worked on 15 years ago, because the current domain has it blocked.