3 ms·
Article is new but paper mentioned within is from 2019, predating LLM revolution. From abstract: >We show here, using quantitative methods on a large cross-lin
by forgotpwd16 3y ago
Article is new but paper mentioned within is from 2019, predating LLM revolution. From abstract:
>We show here, using quantitative methods on a large cross-linguistic corpus of 17 languages, that the coupling between language-level (information per syllable) and speaker-level (speech rate) properties results in languages encoding similar information rates (~39 bits/s) despite wide differences in each property individually: Languages are more similar in information rates than in Shannon information or speech rate.
Quite interesting result. Curious how constructed language Ithkuil, designed to convey information in highly compressed manner, compares.
- r14c 3y agoI would hypothesize that Ithkuil would be spoken fairly slowly similar to Cantonese and so yield a similar bit rate in line with the overall results for natural languages. That said, nobody speaks Ithkuil fluently enough to test this so maybe I'll be proven wrong some day!
- ketralnis 3y ago> predating LLM revolution So? It's not about LLMs?
- janalsncm 3y agoMaybe they were referring to tokenization? Tokens per sentence might be a useful metric although imperfect.
- d0mine 3y agoCurrent LLMs are biased (English): different bits per second for different languages (assuming constant tokens/s for a model, and the paper is true (same 39 bits/s for different languages)). Because tokens (as they are used for llms now) do not reflect information density for different languages (may degrade to token per unicode codepoint for non-English).