I'd like to offer a counteropinion. Almost nobody is working on real AGI. Almost all AI / deep learning is stateless. GPT-3 is not sitting there and thinking,

I'd like to offer a counteropinion. Almost nobody is working on real AGI. Almost all AI / deep learning is stateless. GPT-3 is not sitting there and thinking, it has no inner life and no incentives. It is just a huge huge huge combination if its inputs, a glorified markov chain model, that gets calculated with every request. Embodied AI is much underestimated. The modularity of the human brain is much underestimated. You have a dedicated face recognizer, a grammar processor, a rythm processor a brain region that does SLAM, and so on. And all these parts are connected together. I don't think you'll get AGI by training one huge net front to back. You'll need to connect dozens of best-in-class models together, with adaptive "connective tissue", and some input/output (i.e. senses or a "body") that does not exist yet. If you try to do it in one, you get a "curse of the cross product": complexity is then NxN and not N+N. Higher order training will also not work easily. If you want to use deep learning to train hyperparameters for deep learning, each learning step from the higher level net will include training the lower level net. Meaning, the training time will be exorbitant. Finally, I think the ability of AGI to "break out" to the physical world is overestimated. If a battlestar galactica or borg scenario was possible, then we would have seen more spectacular cyberattacks already. The crappyness of our digital infrastructure is a cause of vulnerabilities, but it is also the guarantee that a super hacker cannot take over everything. In the end, I think the greatest danger is from humans building autonomous systems (i.e. killer robots), and from using IT and AI to construct a police state. Remember, we already built one world-wide system that we no longer can control and that subjugated us all - our global economy. Maybe lets try to reign that one in before we worry about fictitious systems that might arise.