3 ms·
just curious.. are there languages that are better or more efficient to build LLM's with other than English?
by senectus1 4mo ago
just curious.. are there languages that are better or more efficient to build LLM's with other than English?
- kmori_de 4mo ago[dead]
- _jayhack_ 4mo agoFor some definitions of better, yes. Chinese is more token efficient for representing fixed text, for example, although this does not always lead to better performance on downstream tasks.
- kmori_de 4mo agoTrue. I suspect it's still hard to tell whether the bottleneck is the language itself, the tokenizer, or just the overwhelming amount of English training data.