2 ms·
Incidentally, according to https://gpt-tokenizer.dev https://gpt-tokenizer.dev, in gpt-5, "literature" is two tokens ("liter" + "ature"), whereas "文学" is one.
by valleyer 2mo ago
Incidentally, according to https://gpt-tokenizer.dev https://gpt-tokenizer.dev, in gpt-5, "literature" is two tokens ("liter" + "ature"), whereas "文学" is one.
- argee 2mo agoMy bad, the English word I had in mind was "technology" (技術) but I misremembered due to feeling like "tech" should be its own token.