2 ms·
For some definitions of better, yes. Chinese is more token efficient for representing fixed text, for example, although this does not always lead to better perf
by _jayhack_ 4mo ago
For some definitions of better, yes. Chinese is more token efficient for representing fixed text, for example, although this does not always lead to better performance on downstream tasks.
- kmori_de 4mo agoTrue. I suspect it's still hard to tell whether the bottleneck is the language itself, the tokenizer, or just the overwhelming amount of English training data.