I think it's quite a clever way of dealing with the problem. Archive.org takes the long view - they don't delete data merely because someone changed a robots.tx

I think it's quite a clever way of dealing with the problem. Archive.org takes the long view - they don't delete data merely because someone changed a robots.txt, they merely hide it. So the data is only inaccessible as long as someone cares enough to specifically hide it. At the same time this covers their ass, copyright-wise.