3 ms·

> ... are based on predicting the next most likely letter based on a giant internet-based database. We do know that’s what LLMs do. If you're claiming that the

by fl7305 13d ago

> ... are based on predicting the next most likely letter based on a giant internet-based database. We do know that’s what LLMs do. If you're claiming that the training objective tells us what kind of internal mechanisms the training produced, then I think that's just plain wrong. Next-token prediction describes the optimization target, not the internal mechanisms that the training produced. In the same way for the natural evolution of humans, DNA replication is the evolutionary objective. It's not a description of the internal mechanisms that evolution has produced. As an example, we know that neural networks can be trained to develop generalized algorithms for arithmetic. They might first memorize the training examples, then with further training transition to a solution that generalizes correctly to unseen examples. In some cases we've even reverse-engineered the evolved internal mechanisms and found structured arithmetic algorithms rather than rote memorization. Interestingly, for modular addition this can involve Fourier representations, which isn't an algorithm I would have guessed gradient descent training of neural networks would produce.

talon8635 13d ago
You are more convincing than the person you’re responding to.
DrewADesign 13d ago
You can try to say that I’m arguing whatever you like. If you’re claiming that the underlying structure of digital so-called neural networks is comparable to biological neural networks— which we’ve studied for far longer without really understanding— no amount of jargon will obviate the ‘citation needed’ requirement for that claim.
- fl7305 13d ago
  > You can try to say that I’m arguing whatever you like. I did my honest best possible interpretation of what you really meant from what you wrote. >> We do know that zero parts of human decision making are based on predicting the next most likely letter based on a giant internet-based database. We do know that’s what LLMs do. I read this as "The decision making of LLMs are based on predicting the next most likely letter based on a giant internet-based database." Is that wrong? I understood that your meaning was something like "LLMs can't reason, they just output likely letters"? > If you’re claiming that the underlying structure of digital so-called neural networks is comparable to biological neural networks No, I don't claim that. What do claim is this: Regardless of how the LLMs were trained, they show overwhelming signs of being able to reason, and not just recall memorized information. This doesn't mean that they always reason perfectly about everything. But if they only memorized things and output the next likely letter, you would see them answering very badly much more often.

by fl7305 13d ago

talon8635 13d ago

You are more convincing than the person you’re responding to.

DrewADesign 13d ago

You can try to say that I’m arguing whatever you like. If you’re claiming that the underlying structure of digital so-called neural networks is comparable to biological neural networks— which we’ve studied for far longer without really understanding— no amount of jargon will obviate the ‘citation needed’ requirement for that claim.

fl7305 13d ago
> You can try to say that I’m arguing whatever you like. I did my honest best possible interpretation of what you really meant from what you wrote. >> We do know that zero parts of human decision making are based on predicting the next most likely letter based on a giant internet-based database. We do know that’s what LLMs do. I read this as "The decision making of LLMs are based on predicting the next most likely letter based on a giant internet-based database." Is that wrong? I understood that your meaning was something like "LLMs can't reason, they just output likely letters"? > If you’re claiming that the underlying structure of digital so-called neural networks is comparable to biological neural networks No, I don't claim that. What do claim is this: Regardless of how the LLMs were trained, they show overwhelming signs of being able to reason, and not just recall memorized information. This doesn't mean that they always reason perfectly about everything. But if they only memorized things and output the next likely letter, you would see them answering very badly much more often.