Although I disagree with the conclusion, this looks like a pretty decent article in terms of detailed thinking. I just think that we not only need to work on a

Although I disagree with the conclusion, this looks like a pretty decent article in terms of detailed thinking. I just think that we not only need to work on all of the more immediate problems but also worry about less immediate problems that are existential risks. Also people always say "AGI" and they mean different things but I am guessing most now mean something like a digital living human/simulation of human when they say that. Mixed in with some blurry connotation of it automatically becoming godlike in power. That stuff is very speculative. But we don't need AI to get to that level for it to be very dangerous. Just imagine that we have an open source GPT that is something like 33% smarter than GPT-4 and less brittle. Make the output 50 times faster than human thought and suppose that we can run it very inexpensively, in a "swarm" of agents cooperating. Then you have a type of superintelligence that does not require any really speculative AI advance -- hyperspeed reasoning -- based on the current technology. If that is widely deployed for military and industrial decision-making then something like a computer virus could create existential risk. Also, realize that all these systems really need to emulate some of the core functional aspects of animals, like a self-preservation instinct or desire to control resources, is the right instruction and relaxation of guardrails. So when they can reason a bit better and at hyperspeed with collaboration between them, they don't need to be alive or anything to be dangerous. Look at the history of increases in computing efficiency. It is easy to imagine 50 or whatever times output speed increase in less than 5 years. And even though they are not human level now, GPT-4 has proven that these types of systems can have strong problem solving ability.