3 ms·
IA have increasingly ignored robots.txt since the mid-2010s, going on at least eight years now: <https://blog.archive.org/2017/04/17/robots-txt-meant-for-searc
by dredmorbius 3y ago
IA have increasingly ignored robots.txt since the mid-2010s, going on at least eight years now:
<https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...>
<https://blog.archive.org/2016/12/17/robots-txt-gov-mil-websites/ https://blog.archive.org/2016/12/17/robots-txt-gov-mil-websi...>