3 ms·
Would archive.org typically honor a robots.txt for a resource it already retrieved? I never understood the intent of a robots.txt to be retroactive.
by hughw 11y ago
Would archive.org typically honor a robots.txt for a resource it already retrieved? I never understood the intent of a robots.txt to be retroactive.
- mikeash 11y agoApparently yes, it would: https://archive.org/about/exclude.php https://archive.org/about/exclude.php
- X-Istence 11y agoYes, it simply hides the content, it is still kept in their database so if the robots.txt disappears, it pops back from their archive. New pages won't be archived though.
- syncsynchalt 11y agoMy understanding is that sites like archive.org honor robots.txt retroactively not because they are required to, but to best honor the wishes of the content provider.