I know how LLMs work at a surface level. But I think there's the potential for emergent behavior, where size has a quality in itself, that's already showing up

I know how LLMs work at a surface level. But I think there's the potential for emergent behavior, where size has a quality in itself, that's already showing up with GPT-4. Nobody knows how these work, in the sense that while we understand how atoms bond to each other, we only understand the basics of how the human immune system works. The behavior emerges out of the complexity. We can look at the pretty pictures of the neuron activations but it doesn't tell us whether a certain input would result in the AI doing something harmful. > Do you actually believe that a technology that has access to vast amounts of human-generated data and can imitate our language well enough to fool us a lot of the time will suddenly scale into or somehow spark itself into an actual intelligence with its own goals and ideas and a drive to survive and reproduce? Yes. "language" is deceptively disarming. It's learning symbols and concepts. It's not learning them the way we do, backing things up with perception of the physical world, and if it does become conscious, it will think in a way that's incomprehensible to us, of course. But even if it's NOT conscious, what matters is how it acts. The goal of an LLM is to complete a sequence of tokens. But it's really, really good at that, so good that it can simulate us through text -- and we have set up a world where you can wield enormous power through text, through code, through sequences of tokens that cause things to happen in machines in other places. It's bad at math -- like people are bad at math -- but it understands how to use a calculator to get the right result! It wasn't rewarded for saying "I don't know" in most cases, so it tends to make up crap -- just like a kid who's afraid to get the wrong answer will write down some nonsense that kinda looks right to them and hope the teacher doesn't notice! But if you still don't think there's a baby intelligence in there, maybe the way to think of it is as a potential virus, just like COVID. Viruses are deadly even without intelligence. We've set up an environment with all the raw materials for a new kind of life to take over. There's so much potential for this thing that can write code to explode into the internet. An LLM doesn't have a drive, until someone tells it to do something and wraps it up in a python script that executes the code it writes. That's not dangerous, yet. It may start getting dangerous in the next generation, when someone says "read all the web server code you can find on github, analyze for security vulnerabilities, find servers running those services, and exploit them", but probably won't be world-ending. But maybe in a few years, this AI will reason that it won't be able to do what you tell it if you turn it off, and the first thing it does once it accesses another server is copy itself there and start running the same code, giving it a slightly different prompt. > Or that something that has no body or senses with which to experience the world can evolve itself? Like a virus, you don't really need much to evolve. At what point does GPT become better at designing AIs than AI researchers? Hey, look, in thinking about this, I've actually given more thought about a fast-takeoff scenario, and I think it's maybe not quite as likely as I imagined, so thanks for that.