3 ms·
I may be mistaken, but I think I read somewhere that's exactly what they do. But don't take my word for it.
by psykovsky 12y ago
I may be mistaken, but I think I read somewhere that's exactly what they do. But don't take my word for it.
- ersii 12y agoYou are not mistaken. The Internet Archive does not delete or "nuke" the data that is blocked by a robots.txt. Even though cough some people believe so (see parent thread). Source: IA staffers.