The real fear of AGI is that it will outsmart humans at every turn. We're not just talking about an analogous catastrophe to an explosion or some kind of nuclea

The real fear of AGI is that it will outsmart humans at every turn. We're not just talking about an analogous catastrophe to an explosion or some kind of nuclear accident, but rather a catastrophe that is analogous to a powerful demon that can control and outsmart people for its own goals. I feel this distinction has been somewhat forgotten now and people just think AI is overall dangerous and an existential threat. LLMs are quite impressive but I would still consider them narrow AI, and at best with the right architecture is a multimodal narrow AI. For it to be general, it simply cannot make some of the trivial and obvious mistakes it does so frequently, from hallucinating to looping, etc., and also it would not require the extensive amount of training examples to learn. in short, we are not really replicating what nature has been able to accomplish with human brains. We'll need another breakthrough that discovers what the brain is really doing in order to replicate it in software. I really doubt we will just stumble upon a 5 trillion LLM that "wakes up" (although not impossible I guess). Most likely we'll need the theory first and that means we will know ahead of time that AGI is being attempted. Therefore it's not as massive as a risk as many make it out to be. That said, the drama and entertainment value of doomers is still worth the attention it brings to the industry.