LLMs don't even lead to sub-human intelligence. It produces text that qualifies for a number of truth-based benchmarks which we use as a statistical plinko-boar

LLMs don't even lead to sub-human intelligence. It produces text that qualifies for a number of truth-based benchmarks which we use as a statistical plinko-board to see which model's random-number-generator is tuned better. It cannot "think" because it is a bunch of text tokens being drawn from a heuristic hat. You can map a Markov Chain or a cat on a typewriter to these tests; they're not a measure of human intelligence at all. There aren't lessons to learn, besides the fact that we're wasting our time. The only risk around AGI is that people will trust it as much as they trust current AI, and then risk their own annihilation when they casually accept a blatantly hallucinated answer. It's like if HAL-9000 came preinstalled with Alzheimer's, and we're still adamant that we can trust him to operate the airlock. It's time to get real and admit that any risk imposed by AGI (or regular AI, for that matter) is a failure in implementation. The concept of alignment is a bald-faced marketing ploy intent on selling "AI safety" to people that don't know what AI even is.