I lost ten years of my life to worrying about AI risk. I try not to be bitter about that as the rationality techniques I learned along the way have helped me im

I lost ten years of my life to worrying about AI risk. I try not to be bitter about that as the rationality techniques I learned along the way have helped me immensely. But I do try to caution others who appear to be falling down the same rabbit hole. It is the AI x-risk crowd that is non-mainstream, so there isn't really an easily accessible collection of opposing viewpoints, because few have bothered to write up their views. The default/null position is that AI, like nearly all other technologies ever, will have a gradual impact on society and will remain subject to human controls. And that's what most people believe, so researchers who don't buy into the x-risk scenario simply dismiss it as they do all crackpot ideas and move on with their lives. (It should be said that Goertzel has written up his response though: http://multiverseaccordingtoben.blogspot.com/2010/10/singularity-institutes-scary-idea-and.html http://multiverseaccordingtoben.blogspot.com/2010/10/singula...) "GPT-3 is AGI" is a bit of a strawman, sorry. I don't think anyone believes that. But GPT-3 is a clear demonstration that networks can be trained on todays hardware to have at least enough encoded information as a human mind, and encoded in such a way as to be generically useful. But GPT-3 is not an agent, obviously. In the mind of its creators however, it ought to be relatively simple (read: a few million dollars and a couple of years) to attach the GPT-3 knowledgebase to the kind of reinforcement learning based agent architecture that DeepMind and OpenAI are independently developing. Do that and you would have an agent-y general, human-level (or better) intelligence. Am I worried about that? No. Nor are most researchers in the field, for a variety of reasons. Some say it will be so fragile that it will require a symbiotic relationship with its human handlers, and unable to realistically act against us. Others say that its neural network architecture and human training data will make it behave similar enough to its creators in the ways which matter. Others note that initial projects will be so bug-ridden as to be unusable for word domination until decades down the line, and AI defenses will naturally be developed at a similar pace. So AI might become a geopolitical arms race, but not any different than nuclear weapons in that regard. Those are all viable arguments I've heard, and I'm sure there are others as well. Notably for AI x-risk to be a thing, all of the above arguments have to be flawed. If even one of them is true, AI will not take over the world. So we don't even have to delve deep into each of them to assess that the likely outcome is that AI does not kill us all. Edit: Also I think you're right that the AlignmentForum crowd doesn't think that "AGI" requires old-fashioned AI techniques instead of neural networks, but rather that they find those techniques easier to analyze and they're taking a maths-heavy approach to friendly-AI. Even Goertzel thinks that a neural-network approach could work; he just thinks (or thought?) that it would take too long to develop. Now it's probably the sunk cost fallacy and pride that keeps him developing OpenCog as an AGI architecture instead, even though the most basic AI-gym bots have surpassed OpenCog in capability.