well I didn't mean to be rude. but it was an off the cuff remark that was wrong. People say things without posting a source and that's how disinformation starts

well I didn't mean to be rude. but it was an off the cuff remark that was wrong. People say things without posting a source and that's how disinformation starts... first thing that came up in search suggests archive.org also is moving away from listening to robot.text files (granted this is an old blog post) https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea... I couldn't find anything stating their policy clearly. but maybe I didn't search hard enough. I find it hard to believe news papers wouldn't block them if it was that easy