3 ms·
Based on their tokenizer tool[1], for GPT 5.x "geschniegelt" is tokenized into three tokens: (ges)(chn)(iegelt) [1]: https://platform.openai.com/tokenizer h
by magicalhippo 7mo ago
Based on their tokenizer tool[1], for GPT 5.x "geschniegelt" is tokenized into three tokens:
(ges)(chn)(iegelt)
[1]: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer
- Tiberium 7mo agoIt's a single token in the most common usage, that is, with a space in front of it "This word is geschniegelt" is [2500, 2195, 382, 192786] Last token here is " geschniegelt"
- nialv7 7mo agoMaybe this is why? Most of the training data has the single token version, so the three tokens version was undertrained?