3 ms·
I would read Francois Chollet's explanation of this. It's very good: https://fchollet.substack.com/p/how-i-think-about-llm-prompt-engineering https://fchollet.s
by svaha1728 3y ago
I would read Francois Chollet's explanation of this. It's very good: https://fchollet.substack.com/p/how-i-think-about-llm-prompt-engineering https://fchollet.substack.com/p/how-i-think-about-llm-prompt...
For words that are not in the model's vocabulary, like 'fluftable', the model uses a subword tokenization strategy. It breaks down the word into smaller known subunits (subwords or characters) and represents each subunit with its own vector. By understanding the context in which 'fluftable' appears and comparing it to known words with similar subunits, the model can infer a plausible meaning for the word. This is done by analyzing the vector space in which these representations exist, observing how the vectors align or differ from those of known words.
'As always, the most important principle for understanding LLMs is that you should resist the temptation of anthropomorphizing them.'
- sharemywin 3y agoisn't 'infer' another word for reason?
- svaha1728 3y agovector math in a 1536-dimensional space?
- lucubratory 3y agoI'm sorry, but that's absurd. Being able to explain the precise mechanism behind reasoning would make anything sound like it's not reasoning, because of our prior experiences. If we understood human reasoning well enough to explain exactly what happens in our brain, you would conclude that we're not really reasoning because you can provide an explanation of how we're reasoning about novel, out of distribution data. This is "God of the gaps" for thought.
- miffy900 3y agoWhat you've written does nothing to disabuse any reasonable person of the notion that LLMs cannot reason; if anything you've explained how LLM's reason, not that they cannot do it.