It's not like I think the first iteration of superintelligent AGI will necessarily be an existential threat. The problem is really the law of large numbers. Whe

It's not like I think the first iteration of superintelligent AGI will necessarily be an existential threat. The problem is really the law of large numbers. When you have N separate militaries and M separate companies all with their own agenda, stretched out over X years (where X is thousands, mind you), there is a lot of scope for >= 1 of these groups going the way I described -- effectively creating a new species that's smarter than us and that can function in the real world. Many of these groups would have an incentive to do that, because such agents are useful for certain tasks. > Also, it would have to get a job to pay its AWS bill. Inference power costs are low now on agents that are better than humans at Chess and Go. It isn't going to be an issue after another 20-100 years of further R&D and optimizations. Nothing about the history of computing should tell us that this will be a big limiting factor.