Haha, thanks. I think: 1. That an AGI which was significantly more intelligent than humans could destroy us if it chose 2. That it's possible that such an AI

Haha, thanks. I think: 1. That an AGI which was significantly more intelligent than humans could destroy us if it chose 2. That it's possible that such an AI could be created in the next decade or two, given the current trajectory. And so, I think we definitely need to be careful, and make sure we don't blunder into the AI apocalypse. However, there are several further assertions which are often made which are part of the "we're all doomed" scenario: 3. There would be no signs of "misalignment" in not-quite-as-capable AGIs. 4. Even if there were signs of misalignment, that at least some AI research groups would continue to press on and create a mis-aligned super-intelligence 5. Even if we learned how to align not-quite-as-capable AGIs, those techniques wouldn't transfer over to the super-intelligent AGIs. It's possible all of those things are true, but a) 3 and 5 are not true of biological general intelligences b) give our experience with nuclear weapons, I think 4 is likely not to be true. So re number 3: When you have severely "mis-aligned" people -- sociopathic humans who end up performing atrocities -- there are usually signs of this tendency during development. We have far more license to perform "what-if" testing on developmental AIs; I think it very likely that if AGI-1 or AGI-2, who are "only" as good at planning as a 7-year-old, have severe mis-alignment risks, that this would be detectable if we're looking for it: that if it's likely to destroy the world, and we try to give it opportunities to destroy the world in a simulation, that it will show its colors. Re number 4: Many world leaders thought scientists were over-reacting about the risk of nuclear weapons, until they saw the effects themselves. Then everyone began to take the risk of nuclear war seriously. I think that if it's demonstrated that AGI-1 or AGI-2 would destroy the world if given a chance, then people will start to take the risk more seriously, and focus more effort on methods to "align" the existing AGI (and also further probe its alignment), rather than continuing to advance AGI capabilities until they are beyond our ability to control. Re number 5: Children go through phases where their capabilities make sudden leaps. And yet, those leaps never seem to cause otherwise well-adjusted children to suddenly murder their parents. If we learn how to do "inner alignment" on AGI-2, I think there's every reason to think that this basic level of alignment will continue to be effective (at least at the "don't destroy the world level") for AGI-3; at which point, if we've been warned by AGI-2's initial mis-alignment, researchers in general will be motivated to continue to probe alignment and hone mis-alignment techniques before going on to AGI-4 and so on. There's a lot of "if"s there, both on what humans do, and what the development of AGI looks like. We should be careful, but I think if we're careful, there's a good chance of avoiding catastrophe.