4 ms·
> LLMs predict next words. That's an oversimplification. Deep learning LLMs such as ChatGPT are designed to solve natural language processing (NLP) tasks, not
by xcv123 3y ago
> LLMs predict next words.
That's an oversimplification. Deep learning LLMs such as ChatGPT are designed to solve natural language processing (NLP) tasks, not just next-word prediction. In order to "predict next words" they create a model and "reason" about it. They are not simple Markov processes. There's a lot happening in that ~100 layer neural network.
- jraph 3y agoOkay, it seems I should dig more on the topic. Thanks!
- xcv123 3y agoWell as the other commenter here clarified, you're right about transformer LLMs doing next word prediction, but my point is that's only the tip of the iceberg. The complexity of the model doing the prediction is incredible. Not basic stats.
- IshKebab 3y agoTechnically they are designed to solve next-word prediction. It just turns out that good next-word prediction requires understanding the world. (And you can equally view humans as "next word predictors" anyway.)
- xcv123 3y agoYes that's exactly right. NLP problem solving abilities are a consequence of the fact that in order to predict the next token it actually needs to "understand" all of the previous tokens and how they relate to the next predicted token, and determine that the new output is "logically" coherent in context.