4 ms·
They do not. They have in the past removed public access to pages due to robots.txt changes.
by moonchild 6y ago
They do not. They have in the past removed public access to pages due to robots.txt changes.
- pabs3 6y agoHmm, this old blog post sounds like they were planning on ending that practice: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...