3 ms·
Maybe they were referring to tokenization? Tokens per sentence might be a useful metric although imperfect.
by janalsncm 3y ago
Maybe they were referring to tokenization? Tokens per sentence might be a useful metric although imperfect.
- d0mine 3y agoCurrent LLMs are biased (English): different bits per second for different languages (assuming constant tokens/s for a model, and the paper is true (same 39 bits/s for different languages)). Because tokens (as they are used for llms now) do not reflect information density for different languages (may degrade to token per unicode codepoint for non-English).