I thought they might have added an entry to robots.txt but archive.org ignored those https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-don

I thought they might have added an entry to robots.txt but archive.org ignored those https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...