4 ms·
This policy was actually changed in 2017 to completely ignore robots.txt. [0] https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work
by TJSomething 7y ago
This policy was actually changed in 2017 to completely ignore robots.txt.
[0] https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...