3 ms·
> For instance you should be able to ask it how it tokenizes a phrase Can you tell me how your brain maps input to its internal representation? If not, why s
by dragonwriter 2y ago
> For instance you should be able to ask it how it tokenizes a phrase
Can you tell me how your brain maps input to its internal representation?
If not, why should we think its an essential feature of an AI system to do the equivalent?
- PaulHoule 2y agoYou can ask me questions like "How many words are in this sentence?" or to solve anagram puzzles because I have access to information at the word and the character level which the LLM doesn't have. Tokens matter because they are the bridge between the text the user sees and what the model sees and debugging of that relationship is helpful.