3 ms·
IIRC archive.org pay attention to the robots.txt now rather than in 2024.
by pabs3 6y ago
IIRC archive.org pay attention to the robots.txt now rather than in 2024.
- moonchild 6y agoThey do not. They have in the past removed public access to pages due to robots.txt changes.
- pabs3 6y agoHmm, this old blog post sounds like they were planning on ending that practice: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...