4 ms·
It's crazy that OpenAI hasn't fixed their tokenizer yet. They are leaving the door wide open for some Chinese big tech company to capture the non-Latin script p
by 0xDEF 4y ago
It's crazy that OpenAI hasn't fixed their tokenizer yet. They are leaving the door wide open for some Chinese big tech company to capture the non-Latin script parts of the world.
i18n (and accessibility) was something American tech companies were serious about in the 90s and early 2000s. That is how they captured most of the global market. US tech dropping the ball on this leaves the door wide open for Chinese competitors.
- hombre_fatal 4y agoDoes OpenAI’s tokenizer issues cash out into having worse results for Greek rather than just being more expensive for gpt-4? (gpt-3.5-turbo already costs peanuts) If not, then this response seems overblown. The competitive advantage in LLM at this point probably is not tokenizer optimizations and more about having results worth a damn.
- v4dok 4y agoThe usability is worse. Token limits are so much easier to reach. It's like using a model with dementia.
- hombre_fatal 4y agoI didn't think about that but now it's obvious. Good point.
- Fauntleroy 4y agoIt could be that they are actively working on this problem, but the product has not yet been released.