7 ms·

> One of the advantages of UTF-16 is knowing right away it's UTF-16 as opposed to deciding if it's UTF-8/ASCII/other encoding. Sure, for transmission it's a was

by lambda 14y ago

> One of the advantages of UTF-16 is knowing right away it's UTF-16 as opposed to deciding if it's UTF-8/ASCII/other encoding. Sure, for transmission it's a waste of space (still, text for today's computer capabilities is a non issue even if using UTF-32) First of all, if you don't know the encoding, then you don't know the encoding, and you will need to figure out if it's UTF-8, UTF-16, ISO-8859-1, etc. If you happen to know that it's UTF-16, you still need to figure out if it's UTF-16BE or LE. > "It's not fixed width" But for most text, it is. This is a dangerous way of thinking. One of the big problems with UTF-16 is that for most text, it is fixed width; so many people make that assumption, and you never notice the problem until someone tries to use an obscure script or an emoji character. This means that bugs can easily be lurking under the surface; while with UTF-8, anything besides straight ASCII will break if you assume fixed width, making it much more obvious. > Sure, you can do UTF-32 and it may not be a bad idea (today) UTF-32 isn't really meaningfully fixed width either. Sure, each code point is represented in a fixed number of bytes, but code points are not necessarily the interesting unit you want to index by. A glyph could be composed of several code points. Most of the time, you actually want to deal with text in longer units such as words or tokens, which are going to be variable width anyhow. The actual width of individual code points is only really of interest to low-level text processing libraries, not most applications.

raverbashing 14y ago
"First of all, if you don't know the encoding, then you don't know the encoding" True. But as you said, you have to know if it's BE or LE on UTF16. And there are ways to determine that automatically. (or it's on the same platform so it doesn't matter). With "ASCII compatible" codes, you can't. I guess the main issue to me is that UTF-16 is not "ASCII compatible" so you know it's a different beast altogether. And don't worry, I'm not assuming UTF-16 is fixed width. One should use the libraries and not try to solve this 'manually'. About UTF-32 think: CPU registers and operations. Working with bytes is inefficient (even with the benefit of smaller size).
- luriel 14y ago
  > True. But as you said, you have to know if it's BE or LE on UTF16. Yes, with UTF-16 you need to know not just the encoding, but also the endianness. That makes UTF-16 worse, not better. UTF-16 is really the worst of all possible worlds: tons of wasted space with all the complexities of variable encoding without fixed endianness.

by lambda 14y ago

raverbashing 14y ago

"First of all, if you don't know the encoding, then you don't know the encoding" True. But as you said, you have to know if it's BE or LE on UTF16. And there are ways to determine that automatically. (or it's on the same platform so it doesn't matter). With "ASCII compatible" codes, you can't. I guess the main issue to me is that UTF-16 is not "ASCII compatible" so you know it's a different beast altogether. And don't worry, I'm not assuming UTF-16 is fixed width. One should use the libraries and not try to solve this 'manually'. About UTF-32 think: CPU registers and operations. Working with bytes is inefficient (even with the benefit of smaller size).

luriel 14y ago
> True. But as you said, you have to know if it's BE or LE on UTF16. Yes, with UTF-16 you need to know not just the encoding, but also the endianness. That makes UTF-16 worse, not better. UTF-16 is really the worst of all possible worlds: tons of wasted space with all the complexities of variable encoding without fixed endianness.