I mean even honoring the opt-out of robots.txt would be fantastic. As another commenter pointed out they willfully ignore it: https://blog.archive.org/2017/04/1

I mean even honoring the opt-out of robots.txt would be fantastic. As another commenter pointed out they willfully ignore it: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea... It's fairly unethical.