4 ms·
Hmm, this old blog post sounds like they were planning on ending that practice: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-wo
by pabs3 6y ago
Hmm, this old blog post sounds like they were planning on ending that practice:
https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...