Yeah, which is the Internet Archive's biggest issue. What site uses a domain name at one time isn't what site might use the same domain name at a different date

Yeah, which is the Internet Archive's biggest issue. What site uses a domain name at one time isn't what site might use the same domain name at a different date. If the archive had more common sense, they'd note down when sites and robots.txt files change and only block pages by the site owner with the offending robots.txt file. So if say, Google went under and a new company started up, a robots.txt file from the latter would only block files added since the site switched over. That would solve 99.99% of these issues.