What I've heard is that MIRI has an explicit philosophy of concentrating on the more abstract & theoretical aspects of AI safety. The idea being that if AI saf

What I've heard is that MIRI has an explicit philosophy of concentrating on the more abstract & theoretical aspects of AI safety. The idea being that if AI safety is something that you can just tack on to a working design at the end, they don't have a comparative advantage there: it's difficult to predict which design will win and the design's implementor is best positioned to tack on the safety bit themselves. >...imagine a hypothetical computer security expert named Bruce. You tell Bruce that he and his team have just 3 years to modify the latest version of Microsoft Windows so that it can’t be hacked in any way, even by the smartest hackers on Earth. If he fails, Earth will be destroyed because reasons. >Bruce just stares at you and says, “Well, that’s impossible, so I guess we’re all fucked.” >The problem, Bruce explains, is that Microsoft Windows was never designed to be anything remotely like “unhackable.” It was designed to be easily useable, and compatible with lots of software, and flexible, and affordable, and just barely secure enough to be marketable, and you can’t just slap on a special Unhackability Module at the last minute. >To get a system that even has a chance at being robustly unhackable, Bruce explains, you’ve got to design an entirely different hardware + software system that was designed from the ground up to be unhackable. And that system must be designed in an entirely different way than Microsoft Windows is, and no team in the world could do everything that is required for that in a mere 3 years. So, we’re fucked. >But! By a stroke of luck, Bruce learns that some teams outside Microsoft have been working on a theoretically unhackable hardware + software system for the past several decades (high reliability is hard) — people like Greg Morrisett (SAFE) and Gerwin Klein (seL4). Bruce says he might be able to take their work and add the features you need, while preserving the strong security guarantees of the original highly secure system. Bruce sets Microsoft Windows aside and gets to work on trying to make this other system satisfy the mysterious reasons while remaining unhackable. He and his team succeed just in time to save the day. >This is an oversimplified and comically romantic way to illustrate what MIRI is trying to do in the area of long-term AI safety... http://lukemuehlhauser.com/a-reply-to-wait-but-why-on-machine-superintelligence/ http://lukemuehlhauser.com/a-reply-to-wait-but-why-on-machin...