4 ms·

> There are other standards. Some older programming languages like C# and Java rely on UTF-16. In UTF-16, you use two or four bytes per character. It seemed lik

> There are other standards. Some older programming languages like C# and Java rely on UTF-16. In UTF-16, you use two or four bytes per character. It seemed like a good idea at the time ... IIUC this isn't quite right. UTF-16 never seemed like a good idea on its own. These languages started with UCS-2 which did seem like a good idea. But then USC-2 ran out of bits and UTF-16 was the least-incompatible way to adopt Unicode. So as I understand it UTF-16 was always a compromise for backwards compatibility.

TwoBit 6y ago
Right. UTF-8 was intentionally created while UTF-16 was a known hack to deal with the mistake of UCS-2. Ideally neither should exist. There's at least some argument for the existence of UCS-4.
- asveikau 6y ago
  > There's at least some argument for the existence of UCS-4. I think advocacy for 4 bytes per codepoint misses the mark. People tend to erroneously assume that this takes away all complexity and brings us back to an ASCII-like reality of random access and 1 integer per glyph. However Unicode is full of complexities that mean that is not attainable. (Joiners, combining characters, and bi-di are a few.)
- chrismorgan 6y ago
  And the crazy thing is that UTF-8 came before UTF-16. https://en.wikipedia.org/wiki/UTF-8#History https://en.wikipedia.org/wiki/UTF-8#History, published in 1993. https://en.wikipedia.org/wiki/UTF-16#History https://en.wikipedia.org/wiki/UTF-16#History, published in 1996. And yet they ruined Unicode for everyone with surrogates in order to introduce UTF-16. (Surrogates and UTF-16 are by far the worst thing about Unicode, because they’re a pain and technically completely unnecessary, only having been introduced for poor social reasons.)
br1 6y ago
And UCS-2 run out of bits because the Unicode consortium keeps adding garbage like emojis to keep their job...
- mhh__ 6y ago
  Although I think we have enough, I think the way people use emojis cements my view that they are a good thing. Nothing trivial annoys me more than people writing in "I luv u" shorthand, so if an emoji can a more emotional message in less characters I'm all for it. Even if it's a thinly veiled sexual euphemism. Emojis in official corporate communication can burn - I got one recently when applying for a relatively serious job: sends a strange message, that and it reminds me that I want to save the emojis for my friends and family (sadly not the people we most of spend our time with in)
  - johnisgood 6y ago
    Yeah, but then you should consider the effort taken to say "I love you" in that way. Such a low effort. The message of sending a heart is typically habitual for people, and has no real meaning behind it. Same could be said with "I luv u", but a bit less so, I would say. I think it has a bit more weight to it.
- ivanbakel 6y ago
  As well as this being rather closed-minded, it's also not true. The contents of the 0000-FFFE codepoints are public knowledge, and the biggest users of space are: 1. the private use area 2. the general "CJK" area The second of which has a truly mind-boggling number of characters, including every possible composite Hangul glyph used in modern Korean, despite them being constructable from the basic Hangul codepoints. Emojis and other symbols which aren't used for language appear relatively rarely. Certainly there is no reason to believe that UCS-2 would be sufficient for writing if they were removed. The number of scripts included in Unicode would exhaust even the private use area, and UTF-16 would have been invented regardless.
  - throwwwwwaway 6y ago

TwoBit 6y ago

Right. UTF-8 was intentionally created while UTF-16 was a known hack to deal with the mistake of UCS-2. Ideally neither should exist. There's at least some argument for the existence of UCS-4.

asveikau 6y ago
> There's at least some argument for the existence of UCS-4. I think advocacy for 4 bytes per codepoint misses the mark. People tend to erroneously assume that this takes away all complexity and brings us back to an ASCII-like reality of random access and 1 integer per glyph. However Unicode is full of complexities that mean that is not attainable. (Joiners, combining characters, and bi-di are a few.)
chrismorgan 6y ago
And the crazy thing is that UTF-8 came before UTF-16. https://en.wikipedia.org/wiki/UTF-8#History https://en.wikipedia.org/wiki/UTF-8#History, published in 1993. https://en.wikipedia.org/wiki/UTF-16#History https://en.wikipedia.org/wiki/UTF-16#History, published in 1996. And yet they ruined Unicode for everyone with surrogates in order to introduce UTF-16. (Surrogates and UTF-16 are by far the worst thing about Unicode, because they’re a pain and technically completely unnecessary, only having been introduced for poor social reasons.)

br1 6y ago

And UCS-2 run out of bits because the Unicode consortium keeps adding garbage like emojis to keep their job...

mhh__ 6y ago
Although I think we have enough, I think the way people use emojis cements my view that they are a good thing. Nothing trivial annoys me more than people writing in "I luv u" shorthand, so if an emoji can a more emotional message in less characters I'm all for it. Even if it's a thinly veiled sexual euphemism. Emojis in official corporate communication can burn - I got one recently when applying for a relatively serious job: sends a strange message, that and it reminds me that I want to save the emojis for my friends and family (sadly not the people we most of spend our time with in)
- johnisgood 6y ago
  Yeah, but then you should consider the effort taken to say "I love you" in that way. Such a low effort. The message of sending a heart is typically habitual for people, and has no real meaning behind it. Same could be said with "I luv u", but a bit less so, I would say. I think it has a bit more weight to it.
ivanbakel 6y ago
As well as this being rather closed-minded, it's also not true. The contents of the 0000-FFFE codepoints are public knowledge, and the biggest users of space are: 1. the private use area 2. the general "CJK" area The second of which has a truly mind-boggling number of characters, including every possible composite Hangul glyph used in modern Korean, despite them being constructable from the basic Hangul codepoints. Emojis and other symbols which aren't used for language appear relatively rarely. Certainly there is no reason to believe that UCS-2 would be sufficient for writing if they were removed. The number of scripts included in Unicode would exhaust even the private use area, and UTF-16 would have been invented regardless.
- throwwwwwaway 6y ago