3 ms·
It was correct. And archive.org used this self inflicted problem to justify disregarding robots.txt. https://blog.archive.org/2017/04/17/robots-txt-meant-for-s
by pseudalopex 3y ago
It was correct. And archive.org used this self inflicted problem to justify disregarding robots.txt.
https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
- npunt 3y agoOh thank you for the update, that's great news! Agree with their position 100%.
- pseudalopex 3y ago> Agree with their position 100%. Including the part where they blamed anyone but themselves for hiding old snapshots when robots.txt changed?
- npunt 3y agono I didn't see that in there, I just agree that robots.txt makes sense for web crawlers but not archival purposes