It doesn't matter to me whether the intelligence is "really emergent" or "just a simulation." Two things are true: 1. Solving all kinds of nontrivial problems

It doesn't matter to me whether the intelligence is "really emergent" or "just a simulation." Two things are true: 1. Solving all kinds of nontrivial problems posed in text format is extremely useful, no matter how it works under the hood. This means lots of people will use it, and it will change how people work 2. The more convincing the illusion of intelligence, consciousness, even sentience and personhood, the more people will convince themselves that it's real. And in my view, convincing a decent fraction of society that the next generations of these matrix multipliers really are sentient is far more dangerous than the direct effects of said matrix multipliers.