3 ms·
chinese is less token efficient than english
by jatora 2mo ago
chinese is less token efficient than english
- __m 2mo agoHow so? Do you mean in practice when using mostly english trained models?
- jatora 2mo agoEven in chinese trained models too. the amount of tokens it requires to communicate the same ideas is higher. it is just less efficient to tokenize the Chinese language because logographic languages are archaic, ridiculous, and inefficient compared to latin languages.