4 ms·
Your brain doesn't solely pick the next best word. As best as I understand it, the brain has an external state of the world that constantly updates, paired to a
by tomrod 3y ago
Your brain doesn't solely pick the next best word. As best as I understand it, the brain has an external state of the world that constantly updates, paired to an internal model predicting the next best word.
Which is why we can create the counterfactual that "The Cowboys should have won last night" and it has implicit meaning.
Current LLM models don't have an external state of the world, which is why folks like LeCunn are suggesting model architectures like JEPA. Without an external, correcting state of the world, model prediction errors compound almost surely (to use a technical phrase).
- AlecSchueler 3y ago> Your brain doesn't solely pick the next best word. Wasn't the latest research shared here recently suggesting that that is actually what the brain does? And that we also predict the next token in our own brain while listening to others? Hope someone else remembers this and can share again.
- bottlepalm 3y agoChatGPT wasn't trained on only guessing 'the next word'. ChatGPT was trained on the best total output for the given input. The 'next word' is just intermediate state. Internal to the model, it knows where it is going. Each inference just revives the previous state.