4 ms·
>For instance, (The) might be a token, as might (cast) and (ing), so an LLM might read "The casting" as "(The)( )(cast)(ing) Also, tokens are just numbers inte
by kgeist 3y ago
>For instance, (The) might be a token, as might (cast) and (ing), so an LLM might read "The casting" as "(The)( )(cast)(ing)
Also, tokens are just numbers internally after tokenization. So LLM sees the question as "how many letters N are there in 563 3684 11267?". It has basically no idea what you're talking about. Unless somewhere in the training data someone mentions that token number 11267 has two letters N.
- xanderlewis 3y agoThe fact that tokens are numbers doesn’t seem to have any particular meaning. They’re tokens; they’re elements of a finite set. It doesn’t matter what they ‘are’ because the model doesn’t see this external representation just as it can’t inspect its own architecture. It’s not just the latter part of that sentence that consists of ‘numbers’, all of it does.