In the story the AI creators didn't even consider the hypothesis that the Earthlings were much smarter and hostile. Maybe they could have done more detailed ins

In the story the AI creators didn't even consider the hypothesis that the Earthlings were much smarter and hostile. Maybe they could have done more detailed inspection of their simulated world. But they weren't paranoid enough. I don't think many real world AI researchers are very paranoid. Many don't take AI risk very seriously. The scenario is completely realistic. Second any realistic AI is going to be a complete black box. We can barely understand what goes on in much simpler, smaller scale artificial neural nets. The article proposes a scenario where the "AIs" evolved in a simulated world and weren't understood at all by their creators. The real world equivalent would be someone evolving an AGI using genetic programming. WIth no understanding of how the output actually works. EDIT: Responding to your edit responding to my comment. >The AI researchers don't take AI risk seriously simply because we're not there yet. This wouldn't fill you with confidence if you think there's a chance AI progress might blindside us. Regardless, it's not merely an issue of timescale. I know researchers who honestly don't believe smarter than human AI is a threat. They have various fallacies like believing intelligence implies morality, or that AIs wouldn't need Earth's resources and would leave us alone, or that they are inherently superior and we deserve to be replaced. Yes very intelligent people involved in serious AI research believe these things. It's only very recently that talking about AI risk at all is not seen as a crazy crank theory. And that's in great deal due to this article and others by this author spreading those ideas. They are still quite controversial. >More so, the AI researchers trust software testing practices. If only you knew the testing practices places like Google and Facebook have developed wrt using unmonitored AI on real-world data. No one bothers to test research level work. Stuff that isn't in production. People just throw code together and iterate until it works. Nor is there any reasonable testing software that can alert you that your AI has gone superintelligent and hostile. I'm not convinced such a thing is possible (see above) but even if it was, it would require tons of work to implement.