3 ms·

In order to produce the next sentence you have to produce the next word first and then the word after it and so on. Before the model arrives at the candidates

In order to produce the next sentence you have to produce the next word first and then the word after it and so on. Before the model arrives at the candidates for the next word it first computes vectors in high dimensional space that combine every combination of words in the context and extract semantics from it. When producing the next token the model effectively has already "decided" the direction where the answer will go and that is encoded as a high dimensional vector before being reduced to the next token (and the process repeated)

Jensson 2y ago
> When producing the next token the model effectively has already "decided" the direction where the answer will go No it hasn't, if you tell it to write a random story and it starts with "A", it hasn't figured out what the next word should be, and you run it many times from that "A" you will get many different sentences. It will do some adapting to future possibilities, but it doesn't calculate the sentence once like you suggest it will, it comes up with a new sentence for every token it generates.
- ithkuil 2y ago
  If you "tell it" to make up a random story then your prompt is part of the context and if the model is fine-tunes to follow instructions it will emit a story that sounds like a random story that begins with A (since that's a constrain). If instead the prompt says it should emit 5 repetitions of the letter "A" unsurprisingly it will compete the output with " A A A A". The task is performed by emitting tokens, but on order to correctly execute the task the model has to "understand" the prompt sufficiently well in order to choose the next token (and the next etc). Now, obviously current LLM models have severe deficiencies in the ability to model the real world (which is revealed by their failures to handle common sense scenarios). This problem is completely compounded by a psychological factor in which we humans tend to ascribe more "intelligence" to an agent that "speaks well" so the dissonance of a model that sounds intelligent and yet sometimes is so hilariously stupid throws us off rails. But there is clearly some modeling and processing going on. It's not a mere stochastic parrot. We have those (Markov chains of various sorts) and they cannot maintain a coherent text for long. LLMs OTOH are objectively a phase transition in that space. All I'm trying to say is that whatever is lacking in LLMs is not just merely because they "just do next token prediction". There are other things these models should do in order to go to the next level of reasoning. It's not clear if that can be achieved just by training the model on more and more data (hoping that the models learn the trick by themselves) or whether we need to improve the architecture in order to enable the next phase.

Jensson 2y ago

> When producing the next token the model effectively has already "decided" the direction where the answer will go No it hasn't, if you tell it to write a random story and it starts with "A", it hasn't figured out what the next word should be, and you run it many times from that "A" you will get many different sentences. It will do some adapting to future possibilities, but it doesn't calculate the sentence once like you suggest it will, it comes up with a new sentence for every token it generates.

ithkuil 2y ago
If you "tell it" to make up a random story then your prompt is part of the context and if the model is fine-tunes to follow instructions it will emit a story that sounds like a random story that begins with A (since that's a constrain). If instead the prompt says it should emit 5 repetitions of the letter "A" unsurprisingly it will compete the output with " A A A A". The task is performed by emitting tokens, but on order to correctly execute the task the model has to "understand" the prompt sufficiently well in order to choose the next token (and the next etc). Now, obviously current LLM models have severe deficiencies in the ability to model the real world (which is revealed by their failures to handle common sense scenarios). This problem is completely compounded by a psychological factor in which we humans tend to ascribe more "intelligence" to an agent that "speaks well" so the dissonance of a model that sounds intelligent and yet sometimes is so hilariously stupid throws us off rails. But there is clearly some modeling and processing going on. It's not a mere stochastic parrot. We have those (Markov chains of various sorts) and they cannot maintain a coherent text for long. LLMs OTOH are objectively a phase transition in that space. All I'm trying to say is that whatever is lacking in LLMs is not just merely because they "just do next token prediction". There are other things these models should do in order to go to the next level of reasoning. It's not clear if that can be achieved just by training the model on more and more data (hoping that the models learn the trick by themselves) or whether we need to improve the architecture in order to enable the next phase.