One major problem with new security technology is that testing is generally created as the result of failure, rather than preceding the failure so that it doesn

One major problem with new security technology is that testing is generally created as the result of failure, rather than preceding the failure so that it doesn't occur in the first place. Even very basic things like verifying that a new key actually fit the intended lock (ie, that a new key correspond to the published data) is a concept that people tend to implement only once they have had experienced a failure. One general solution is to have standardized tools with standardized testing. Transferring a domain should always involve testing the old data, testing the old keys, testing the new data, testing the new keys, testing the chain of delegation, testing the chain of trust and so on. Some TLD's has gone as far as doing basic sanity test before a domain owner is allowed to transfer a domain. (industry term is to delegate a domain, since transfer is strictly about legal ownership) Thankfully there are initiatives to create standard tools and standard test. Zonemaster (https://zonemaster.net https://zonemaster.net) is one I recommend for testing, which is the result of a collaboration between .se and .fr. There is also initiatives for tooling that create multi-signing (https://github.com/DNSSEC-Provisioning/music https://github.com/DNSSEC-Provisioning/music), which is to have multiple keys operating for the same domain. Redundancy in HSM's are nice, but redundancy is not a replacement for testing and backup procedures. It was one of the hard lessons learned by the IT industry, especially with data. No amount of RAID will save a company if a rogue script deletes all the files, or if the database becomes corrupt. A lot of experts learned the hard way that redundancy is mostly just a complement to sanity tests and backup procedures.