4 ms·
How does it know the letters in the token? It doesn't. There's literally no mapping anywhere of the letters in a token.
by fzzzy 11mo ago
How does it know the letters in the token?
It doesn't.
There's literally no mapping anywhere of the letters in a token.
- danielscrubs 11mo agoIf it did frequency analysis then I would consider it having a PhD level intelligence, not just a PhD level of knowledge (like a dictionary).
- ACCount37 11mo agoThere is a mapping. An internal, fully learned mapping that's derived from seeing misspellings and words spelled out letter by letter. Some models make it an explicit part of the training with subword regularization, but many don't. It's hard to access that mapping though. A typical LLM can semi-reliably spell common words out letter by letter - but it can't say how many of each are in a single word immediately. But spelling the word out first and THEN counting the letters? That works just fine.