3 ms·
Does that matter if they just add the whole website to their robots.txt in December of 2024?
by Rebelgecko 6y ago
Does that matter if they just add the whole website to their robots.txt in December of 2024?
- pabs3 6y agoIIRC archive.org pay attention to the robots.txt now rather than in 2024.
- moonchild 6y agoThey do not. They have in the past removed public access to pages due to robots.txt changes.
- pabs3 6y agoHmm, this old blog post sounds like they were planning on ending that practice: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...