5 ms·

I agree about wire formats. However, programming languages that use UTF-8 as the primary string representation tend to have inferior Unicode support to those th

I agree about wire formats. However, programming languages that use UTF-8 as the primary string representation tend to have inferior Unicode support to those that use UTF-16. I think this is because with UTF-8, a lot of stuff just seems to work and so it's easy to ignore the issues, while UTF-16 forces you to come to grips with the realities of Unicode. For example, consider a function to open a file by name. If your strings are UTF-8, you can just pass your null terminated buffer to fopen() or something and things will work fine for most files. But if your strings are internally UTF-16, you have to think about which encoding to use, and you research it, and you discover that holy crap, this stuff differs across OSes, and so we better take this problem seriously.

plorkyeran 13y ago
UTF-16 has the same issue with invalid things appearing to work due to surrogate pairs. A lot of UTF-16 based applications actually only support UCS-2, because you can assume one string index equals one character and get away with it if it never occurs to you to try to use stuff outside of the BMP.
stormbrew 13y ago
> However, programming languages that use UTF-8 as the primary string representation tend to have inferior Unicode support to those that use UTF-16. I'm really curious which languages drove you to this conclusion. Right now we're in a state where the most mature implementations of unicode in general are utf-16 because of the ucs2 accident, so it wouldn't exactly surprise me, but I'd still like to see the proof.