4 ms·
Detractors of UTF-8 often point to the "inefficiency" of counting characters as a reason not to use it for Unicode. I'm glad to see that this is not a valid con
by LogicHoleFlaw 18y ago
Detractors of UTF-8 often point to the "inefficiency" of counting characters as a reason not to use it for Unicode. I'm glad to see that this is not a valid concern.
- pmjordan 18y agoYep, assuming the byte sequence is guaranteed to be valid UTF-8 in any case. But if there's a chance that it isn't, then it's probably coming from disk, network, etc., so the IO will outweigh the extra validation anyway. Back-iterating works pretty well, too. Indexed code point access is the only thing that has a real penalty, as it is always O(n) in time unless you add an O(n) memory position table with O(n) preprocessing time and O(1) runtime lookup. However, code point access only gives you so much anyway, as combining code points mean you can have more than one code point per character. You have that same problem with UTF-32.
- ajross 18y agoDetractors of UTF-8 are just wrong, period. On modern machines, main memory bandwidth gives you about 10 cycles or more per byte of input of free processing. It's irrelevant. And it's important to note that "how many glyphs does this string contain" is a pretty obscure question to be asking in the first place. What's the problem being solved here? The beauty of UTF-8 is that all (all!) of the common string operations continue to work as-is without change. strlen/strdup/strcat/strcpy? Yup, same behavior. Linear search for substrings or regexes? Yup, same behavior. Alphabetic comparison? Check. Just use it, and don't mess with encodings or you will mess things up. If you don't know how it works, just trust it. If you think you know how it works and still believe it doesn't meet your requirements, you're wrong.