5 ms·
"and it respects robots.txt" Not since 2017. https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://b
by tyingq 6y ago
"and it respects robots.txt"
Not since 2017. https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
They now have a clunky manual process to exclude your site. https://help.archive.org/hc/en-us/articles/360004651732-Using-The-Wayback-Machine https://help.archive.org/hc/en-us/articles/360004651732-Usin... ("How can I exclude...")
They don't spoof user agents, but blocking them actively doesn't remove their history.
- Karunamon 6y agoWhen did this change? It used to be that adding robots.txt would retroactively remove archives for a domain.
- tyingq 6y ago2017.
- pseudalopex 6y agoHide. Not remove.