9 ms·
For me, the most surprising thing is that the new PTY devices use UTF-8. Not UTF-16 or UCS-2 or weird little endian variants thereof, and not even wchar_t. Thi
by amluto 8y ago
For me, the most surprising thing is that the new PTY devices use UTF-8. Not UTF-16 or UCS-2 or weird little endian variants thereof, and not even wchar_t.
This is so un-Windows-like.
- zadjii 8y agoIt is! But this is a very un-windows like feature, isn't it? We want this to work on other platforms with as little modification as necessary, and frankly, jumping through the wchar_t<->char hoops is a _pain_. So we'll do it for you!
- cryptonector 8y agoHear hear! wchar_t is a disaster. UTF-16 is terrible. I'm not at all convinced that 2^21 codepoints will be enough, so someday it'd be nice to be able to get past UTF-16 and move to UTF-8, and Windows and ECMAScript are the biggest impediments to that. Your choice of UTF-8 will tend to place UTF-8 on a level playing field in Win32. I guess, too, that this is the end of codepages -- I doubt they'd go away, but there should be no more need to struggle with them, just use UTF-8. You'll still need a semblance of locale, for localization purposes, naturally, but all-UTF-8-all-the-time is a great simplification.
- deleted 8y ago[deleted]
- mehrdadn 8y agoConfused, where does 2^21 code points come from and how is that related to the UTF-16 vs. UTF-8 distinction? Can't both of them encode all Unicode code points? Or are you thinking of code units perhaps, and UCS-2? Although even there I'm confused where the 2^21 came from.
- speeder 8y agoUtf-16 is always 16 bit, utf-8 is variable, can go from 8 to 32 as needed... This does make coding for utf-8 harder, but when it works is really wonderful stuff.
- a3f 8y agoUCS-2 is always 2 octets, UTF-16 allows for so-called surrogate pairs which expands the bits available making it variable size.
- mehrdadn 8y agoLook up UCS-2 that I mentioned in my comment.
- AnIdiotOnTheNet 8y agoUTF-16 is not always 16b/codepoint. UCS-2, its ancestor and more or less what is actually used by Windows when it talks about "Unicode", was 16b/codepoint. It turned out that 64k codepoints wasn't enough for everybody, so they increased the range and added "surrogate pairs" to UCS-2 to make UTF-16. Codepoints in UTF-16 are, therefore, either 16b/codepoint or 32b/codepoint. Many systems that adopted UCS-2 claim UTF-16 compatability but in fact allow unpaired surrogate pairs (an error in UTF-16). The necessity of encoding such invalid UTF-16 strings in UTF-8 under certain circumstances lead to the pseudo-standard known as WTF-8[0]. UTF-8 was designed (as legend has it, on the back of a napkin during a dinner break) after the increase in range and doesn't suffer from the same problem. Additionally, it's straight-forward "continuation" mechanism isn't any more difficult to deal with than surrogate pairs, and it doesn't have any endianess issues like UTF-16/UCS-2. [0]https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/
- yuubi 8y agoUTF-16 has a limit on the size of a code point because a code point is either a single normal code unit or a pair of surrogate code units, each encoding 10 bits of the code point (I think I used the right terminology). UTF-8 has a natural extension path to up to 7-byte encodings with all the usual UTF-8 properties (first code unit indicates how many remain, other code units are recognizable as not the first).
- mehrdadn 8y agoWhere are you getting this information though? I haven't worked out the bits myself yet but Wikipedia's first sentence itself says UTF-16 can encode all 1,112,064 valid code points of Unicode, which is already more than 2^(10+10) = 1,048,576.
- jwilk 8y agoYes, UTF-16 can encode all currently valid Unicode codepoints, which is more than 2²⁰ but less than 2²¹. But cryptonector doesn't believe it will be enough in the future. OTOH, UTF-8, as originally defined, can encode 2³¹ codepoints.
- JdeBP 8y agoUnicode code point space: Was 16-bit (0000 to FFFF), then 32-bit (00000000 to FFFFFFFF), and is now 21-bit (00000000 to 0010FFFF) UTF-16: Encodes the entire 21-bit range, encoding most of the first 0000 to FFFF range as-is, and using surrogate pairs in that range to encode 00010000 to 0010FFFF. The latter range is shifted to 00000000 to 000FFFFF before encoding, which can be encoded in the 20 bits that surrogate pairs provide. This is a subtlety that one likely does not appreciate if one learns UTF-8 first and expects UTF-16 to be like it. UTF-8: Could originally encode 00000000 to 7FFFFFFF, but since the limitation to just the first 17 planes a lot of UTF-8 codecs in the real world actually no longer contain the code for handling the longer sequences. Witness things like the UTF-8 codec in MySQL, whose 32-bit support conditional compilation switch is mentioned at https://news.ycombinator.com/item?id=17311048 https://news.ycombinator.com/item?id=17311048 .
- 8y ago
- cryptonector 8y agoUTF-16 is garbage. Windows is stuck with it because it was too early an adopter of Unicode. Oh the irony. This may set Windows on a path to deprecating UTF-16 -- godspeed!
- swozey 8y agoCould you elaborate? I've been under the guise for most of my career that doubling a digit leads to huge benefits that I'm too comp-sci ignorant to understand.
- cryptonector 8y agoCan I assume that's just subtle humor on your part? If not: UTF-16 is born of UCS-2 being a very poor codeset, as it was limited to the Unicode BMP, which means 2^16 codepoints, but Unicode has many more codepoints, so users couldn't have the pile-of-poo emoticon. Something had to be done, and that something was to create a variable-length (in terms of 16-bit code units) encoding using a few then-unassigned codepoints in the BMP. The result yields only a sad, pathetic, measly 2^21 codepoints, and that's just not that much. Moreover, while many codesets play well with ASCII, UTF-16 doesn't. Also, decomposed forms of Unicode glyphs necessarily involve multiple codepoints, thus multiple code units... Many programmers hate variable length text encoding because they can't do simple array indexing operations to find the nth character in a string, but with UTF-8, UTF-16, and just plain decomposition, that's a fact of life anyways. If you're going to have a variable-length codeset encoding, you might as well use UTF-8 and get all its plays-nice-with-ASCII benefits. For Latin-mostly text UTF-8 also is more efficient than UTF-16, so there is a slight benefit there. Much of the rest of the non-Windows, non-ECMAScript world has settled on UTF-8, and that's a very very good thing.
- swozey 8y agoNo humor whatsoever. Thank you for the explanation! I'm an ops person who knows python/golang to a dangerous extent and have never gone out of my way to understand the UTF reasonings. Your post intrigued me and made me want to ask why you felt that way. This will make me sound horrendously ignorant to someone of the likes of someone such as yourself but I'm here to learn.