TIL they still follow robots.txt -- https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archiv

TIL they still follow robots.txt -- https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea... mentioned that they were planning to stop doing that (and I remember reading a news article based on it claiming that they already stopped following robots.txt, hence my confusion). Truly a shame. I get following robots.txt at collection time, I don't get following a robots.txt that was added later.