As someone who once helped maintain the exclusion-mechanisms, my personal opinion is: The retroactive-robots.txt policy made sense originally as a way of reduc

As someone who once helped maintain the exclusion-mechanisms, my personal opinion is: The retroactive-robots.txt policy made sense originally as a way of reducing risk from angry-rightsholders, while minimizing burdens on staff time, and had little downside when the history-of-the-web was short, and most domains were still under their original ownership. It was a toggle any webmaster could throw, with no support/maintenance effort required at IA. It obviously sucks now, more than a decade after it was adopted as a quick fix, but would take some dedicated policy and technical design to gracefully replace. For example, many rightsholders may be relying on the old behavior. But, the IA hasn't yet been able to prioritize creation of a new scheme. A new process could involve a policy where someone claiming to be the original site/content rightsholder asserts that as of some boundary date, for example when they ceded or sold the domain, later robots.txt should not affect earlier content. (Such a boundary could also be clearly-indicated in Wayback summary/calendar pages.) Then, presumption would flip to showing the earlier content, unless some other rightsholder (such as the current domain-holder) formally claims ("under penalty of perjury", etc) that it's their material and they wish the block to stay in place. It'd be a bit in overall shape like the DMCA takedown/counter-notify procedure, as if the robots.txt was a sloppy takedown request. Squatters making a false claim to ownership of older content would be forced to go on record and take some risk. Ideally any dispute could then proceed between the other two parties, leaving the IA out of it. Again, this would be similar to how the DMCA tries to leave ISPs/caches/hosters out of takedown legal battles. I doubt most new domain-owners are strongly or consciously trying to hide the past; it's usually an automatic choice made for other reasons. A few might be holding the history hostage to make ownership of the domain more valuable – "buy it to re-access the past". Some others might be embarrassed of the previous unaffiliated content – but a clearer Wayback UI indicating changes-of-management could help allay that concern. Unfortunately, for both good and bad reasons, there is no easy/reliable/canonical source of all domain-ownership info over time.