2 ms·
As far as I know, that's not the case. The tokenizer takes a bunch of characters, like "berry", identifies it as a token, and what the LLM gets is the token ID.
by Al-Khwarizmi 2y ago
As far as I know, that's not the case. The tokenizer takes a bunch of characters, like "berry", identifies it as a token, and what the LLM gets is the token ID. It doesn't have access to the information about which letters that token is composed of. Here is an explanation by OpenAI themselves: https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them https://help.openai.com/en/articles/4936856-what-are-tokens-... - as you can see, "Models take the prompt, convert the input into a list of tokens, processes the prompt, and convert the predicted tokens back to the words we see in the response". And the tokens are basically IDs, without any internal structure - there are examples there.
If I'm missing something and you have a source for the claim that character information is present in the input after tokenization, please provide it. I have never implemented an LLM or fiddled with them at low level so I might be missing some detail, but from everything I have read, I'm pretty sure it doesn't work that way.
- psb217 2y agoA sequence of tokens can be converted back to the sequence of tokenized characters without loss of information. Eg, how do you think text is rendered for the user based on sequences of tokens generated by the LLM? Different tokenization schemes arrange that information differently and may make it (hand waving here) harder or easier for the model to reason about details like raw character counts that are affected by tokenization. If the training set included sufficiently many examples of character counting Q/A pairs, an LLM would have no trouble learning how to do this task.