5 ms·

I am kinda surprised how much Windows likes its wchar's. Windows had support for multi-byte codepages since early days, and it has ways to indicate compatibilit

by theamk 4y ago

I am kinda surprised how much Windows likes its wchar's. Windows had support for multi-byte codepages since early days, and it has ways to indicate compatibility level for individual apps. It seems the solution would be simple: once UTF-8 introduced (back in 1996!), add new "UTF-8" codepage, add new "SetDefaultCodepageToUnicode()" function, and tell everyone that wchar_t is legacy and should not be used anymore. They could probably do it in time for Windows XP! Instead, UTF-8 codepage was only introduced in Windows 7 (but had incomplete support), and finally got the proper support in Windows 10. That's a lot of time Windows programmers were forced to use UCS-2.

Arnavion 4y ago
Maybe they didn't want to be early adopters of what could turn out to be a second fad like UCS-2 was the first time.
- manholio 4y ago
  What's depressing about the saga of utf8 adoption is that it's such an obvious solution. The ISO2022 standards used the principle since the 70s, and programmers of the era were widely accustomed to extending 8 bit instruction sets to multibyte instructions. It's one of those cases where you need to hit your head hard and repeatedly until you bite the bullet and accept that the cost of multibyte (for example, strlen() different from character count, need to parse to seek etc.) are trivial compared to the benefits.
  - aew4ytasghe5 4y ago
    UTF8 is only superior in encoding the american keyboard (7 bits of ascii). All the latin letters map to 1 byte encodings. This is why UTF8 wasn't universally hailed to begin with. Countries like Japan had already solved encoding differently while US was still using ASCII. (See JIS encoding). As time passed tho, the byte savings matter less. Also combined with transfer compression makes the encoding size differences negligible.
    - Beltalowda 4y ago
      > UTF8 is only superior in encoding the american keyboard (7 bits of ascii). All the latin letters map to 1 byte encodings. In practice, basic 7-bit ASCII characters dominate a lot of texts, even those in East-Asian languages, as all the markup and such around the actual text is almost always in plain ASCII. For example, if I request a Japanese document from a HTTP server then the HTTP headers, HTML tags, CSS is all in single-byte characters. Of course, there are still many scenarios where UTF-16 "wins" in terms of size – especially with plain text – but real-world size comparisons tend to be a bit more tricky. For example, for the current Wikipedia homepage of ja, zh, and ko: % file * ja-16.htm: HTML document, Unicode text, UTF-16, little-endian text, with very long lines (2709) ja-8.htm: HTML document, Unicode text, UTF-8 text, with very long lines (2709) ko-16.htm: HTML document, Unicode text, UTF-16, little-endian text, with very long lines (2503) ko-8.htm: HTML document, Unicode text, UTF-8 text, with very long lines (2656) zh-16.htm: HTML document, Unicode text, UTF-16, little-endian text, with very long lines (5589) zh-8.htm: HTML document, Unicode text, UTF-8 text, with very long lines (5589) % ls -lh 210K ja-16.htm 118K ja-8.htm 193K ko-16.htm 106K ko-8.htm 189K zh-16.htm 104K zh-8.htm The UTF-16 versions are all larger (and that's just the HTML, excluding the CSS, JS, etc). I wrote a small script to count the number of "wide" multibyte characters: ja-8.htm 100,808 7-bit characters; 19,026 wide characters ko-8.htm 93,888 7-bit characters; 14,149 wide characters zh-8.htm 91,589 7-bit characters; 14,360 wide characters It was higher than I expected, although not that surprising when looking at the source when you have things like: <div class="vector-page-toolbar"> <div class="vector-page-toolbar-container"> <div id="left-navigation"> <nav aria-label="名前空間">
      - knolax 4y ago
        Your argument uses a website whose code was written by English speakers. There would still be ASCII but verbose element names like "vector-page-toolbar-container" would definitely be shorter both in utf-8 and in utf-16 if they weren't written in English
        Beltalowda 4y ago
  - fomine3 4y ago
    ISO 2022 isn't beautiful solution like UTF-8 self sync encoding. Backward compatibility is also great (so called UTF FSS). It was designed by smart Plan9 people.
    - manholio 4y ago
      If Ken and Pike could design it in an afternoon, I'm sure even a standards committee could have done it in less than 5 years or so, the ideas are straight forward. I would also argue that the fast sync with at most one character consumed is a good feature but not essential for adoption. Any byte-oriented Unicode packing scheme that can gracefully consume ASCII is better than multibyte, codepages sent out-of-band etc. etc.
zweifuss 4y ago
I have asked myself the same question. However, I observed that the switch from UCS-2 to true UTF-16 was already very bumpy, with strange bugs and tons of edge cases that only got significantly better in Windows 8. A forced switch to UTF-8 would likely have broken ANSI strings. Languages such as PHP didn't get true UTF-8 support until 2015, Python in 2008 (and that switch took a decade), and Java and JavaScript continue to use UTF-16 as default strings. The wastefulness of UTF-16 is mostly remedied by Unicode Compression (SCSU), which can be much shorter than UTF-8 in languages like Hindi and Chinese.
- wizofaus 4y ago
  How many Windows users do benefit significantly from using UCS-2 encoding, I wonder? Chinese and Hindi (along with Japanese, South Korean and various Arabic language) speakers would surely come close to outnumbering those of us using European alphabets (I gather that Cyrillic & Greek have single-byte encodings, but maybe they mostly use UCS-2 these days too?). On that basis it seems a reasonable choice as the default storage method for human readable strings. But if course a high percentage of strings don't exist for that purpose at all, at least not primarily - using UCS-2 for storing/ transmitting CSS or configuration files or of course source code for most languages is less justifiable.
  - zweifuss 4y ago
    Microsoft software supports UTF-16 (not only UCS-2) everywhere these days (yes, even in COM). But you can probably still find errors here and there. I have no problem with saving text files in UTF-8 and normally do. I just wish SCSU was handled by COM by default as well and not only sometimes.
  - ripe 4y ago
    > Chinese and Hindi (along with Japanese, South Korean and various Arabic language) speakers would surely come close to outnumbering those of us using European alphabets You are probably right. Just slightly off topic: Hindi speakers who use computers mostly seem happy enough with English-language forms, instructions, and technical documents. This is true of most Indian users who speak languages that use one of the Indic scripts. Actual non-English text is useful mainly for publications like newspapers, magazines, and books, which native speakers do prefer to consume in their own language. The above is, of course, not counting the significant numbers of Indians who are quite comfortable using English for daily communication. These Anglophones are just like US speakers in their preferences and have no use for internationalization.
Mikhail_Edoshin 4y ago
UTF-16 is simpler and more efficient to handle than UTF-8.
- netheril96 4y ago
  > UTF-16 is simpler and more efficient to handle than UTF-8. Only if you never go out of the basic multilingual plane. And then find out so many bugs when people start using emojis.
- ainar-g 4y ago
  I feel like you're confusing it with UTF-32, which is fixed-length. UTF-16 is neither fixed-length, like UTF-32, nor compatible with the old ASCII encoding, like UTF-8. The historical reasons of why UTF-16 exists are fully understandable, but by now it is clear that it's the worst of both worlds.
  - Mikhail_Edoshin 4y ago
    No, I'm fully aware of it being a variable-length encoding. Yet as such it is still simpler to work with than UTF-8. So if we are to use a variable-length encoding anyway then UTF-16 is a reasonable choice. It is not compatible with ASCII and old APIs, that's true, but I cannot judge how important this is. I guess that for some projects the efficiency of UTF-16 should outweigh the inconveniences of that incompatibility.
    - heinrich5991 4y ago
      Can you explain in what way it is easier to handle than UTF-8? Most code that I've seen and looks easier just mishandles characters outside the basic multilingual plane. Thankfully, due to emojis being a thing, a lot of developers woke up to the fact that Unicode specifies more than 65536 characters.
      - Const-me 4y ago
        UTF-16 is cheaper to decode. All CPUs have branch predictors, 99.99% of all UTF-16 strings don't contain any surrogate pairs. For East Asian languages UTF-16 uses 50% less memory, these characters take 3 bytes in UTF-8, but only 2 bytes in UTF-16.
fredoralive 4y ago
IIRC the attitude was that the "ASCII" versions of functions largely existed only for backwards compatibility, after all on NT[1] they're basically just wrappers that do character set conversion before passing through to the real Unicode (UCS2/UTF16) functions. The view was that if you want Unicode, you should use the real functions directly, which means 16 bit strings. I think there was also some compatibility concerns, as Windows for a while only had a single global "ASCII" character set used by all apps[2], so you'd have issues with actual legacy apps expecting a real legacy code page breaking / having scrambled text if you tried to use UTF8 as the "legacy" encoding. Or the fun idea of an app that is clever enough to know that multibyte exists, but dumb enough to know they can only be one or two bytes long, and allocating buffers accordingly. I think there might have been some issues with how things like the clipboard handled text as well that assumed all "ASCII" apps used the same encoding. (Disclaimed: old memories of reading blogs, may be bollocks). At some point things must have been fixed so multiple character sets could be used though. [1] But not on 9x, which was "ASCII" only, at least until late in the day when a Unicode compatibility library was created. [2] The "Language for non-Unicode programs" as the Control Panel calls it.
- nikbackm 4y ago
  The "A" stands for "ANSI", not "ASCII". Otherwise largely correct.
debugnik 4y ago
> and finally got the proper support in Windows 10 Not quite, you still can't use some features like longPathAware with the "ANSI" APIs. The preferred APIs for file paths are still the ones using wchar, although the UTF8 codepage is apparently the preferred method for non-path APIs and console IO, at least on the GDK.
cesarb 4y ago
> It seems the solution would be simple: once UTF-8 introduced (back in 1996!), add new "UTF-8" codepage, add new "SetDefaultCodepageToUnicode()" function, and tell everyone that wchar_t is legacy and should not be used anymore. They could probably do it in time for Windows XP! IIRC, the problem was that a MBCS codepage could have a maximum of 2 bytes per character, while UTF-8 could need 3 or even 4 bytes per character. Legacy applications could have fixed-size buffers with space for only 2 bytes per character, and would break if the default codepage needed more than that. > Instead, UTF-8 codepage was only introduced in Windows 7 (but had incomplete support), and finally got the proper support in Windows 10. That's a lot of time Windows programmers were forced to use UCS-2. It appears to me that lately Windows hasn't been as obsessed with backwards compatibility as it had been in the past. They are also gradually allowing for more use of paths longer than MAX_PATH (260 characters), which are not accessible to legacy applications, and as everyone knows, the compatibility with 16-bit applications has been dropped some time ago.
- theamk 4y ago
  That's a good point... so don't make it default! Make "SetThreadLocale" or something work with UTF-8, so apps could switch one-by-one (this may not work with hooks but still better than nothing...). Add UTF-8 support to MSVC runtime, so at least fopen can be safe (people have been asking for it forever). Add FILE_FLAG_UTF_8 to CreateFileA. Have Notepad be able to edit UTF-8. But I suspect the real reason is there was a strong internal opposition to UTF-8 -- maybe it was seen as inferior to UTF-16, maybe it was NIH, maybe they hoped the world will change to follow Windows ways and to put BOM marks in each document... Oh well, I am glad they have at least changed now.
vintermann 4y ago
They couldn't just jump on every technically superior idea, because compatibility is a thing too. And, by the time everyone agreed that utf-8 was preferable, compatibility with their own stuff, which is always especially painful. It's not as if they were the only ones. I had a lot of utf-related pain in the Linux terminal back in the day. Vi and Emacs took a long time to handle variable length encodings well. My old Unix editor of choice, NEdit, never managed the switch at all. I got a mail from my boss in utf-7 once. God knows how that happened.

by theamk 4y ago

Arnavion 4y ago

Maybe they didn't want to be early adopters of what could turn out to be a second fad like UCS-2 was the first time.

manholio 4y ago
What's depressing about the saga of utf8 adoption is that it's such an obvious solution. The ISO2022 standards used the principle since the 70s, and programmers of the era were widely accustomed to extending 8 bit instruction sets to multibyte instructions. It's one of those cases where you need to hit your head hard and repeatedly until you bite the bullet and accept that the cost of multibyte (for example, strlen() different from character count, need to parse to seek etc.) are trivial compared to the benefits.
- aew4ytasghe5 4y ago
  UTF8 is only superior in encoding the american keyboard (7 bits of ascii). All the latin letters map to 1 byte encodings. This is why UTF8 wasn't universally hailed to begin with. Countries like Japan had already solved encoding differently while US was still using ASCII. (See JIS encoding). As time passed tho, the byte savings matter less. Also combined with transfer compression makes the encoding size differences negligible.
  - Beltalowda 4y ago
    > UTF8 is only superior in encoding the american keyboard (7 bits of ascii). All the latin letters map to 1 byte encodings. In practice, basic 7-bit ASCII characters dominate a lot of texts, even those in East-Asian languages, as all the markup and such around the actual text is almost always in plain ASCII. For example, if I request a Japanese document from a HTTP server then the HTTP headers, HTML tags, CSS is all in single-byte characters. Of course, there are still many scenarios where UTF-16 "wins" in terms of size – especially with plain text – but real-world size comparisons tend to be a bit more tricky. For example, for the current Wikipedia homepage of ja, zh, and ko: % file * ja-16.htm: HTML document, Unicode text, UTF-16, little-endian text, with very long lines (2709) ja-8.htm: HTML document, Unicode text, UTF-8 text, with very long lines (2709) ko-16.htm: HTML document, Unicode text, UTF-16, little-endian text, with very long lines (2503) ko-8.htm: HTML document, Unicode text, UTF-8 text, with very long lines (2656) zh-16.htm: HTML document, Unicode text, UTF-16, little-endian text, with very long lines (5589) zh-8.htm: HTML document, Unicode text, UTF-8 text, with very long lines (5589) % ls -lh 210K ja-16.htm 118K ja-8.htm 193K ko-16.htm 106K ko-8.htm 189K zh-16.htm 104K zh-8.htm The UTF-16 versions are all larger (and that's just the HTML, excluding the CSS, JS, etc). I wrote a small script to count the number of "wide" multibyte characters: ja-8.htm 100,808 7-bit characters; 19,026 wide characters ko-8.htm 93,888 7-bit characters; 14,149 wide characters zh-8.htm 91,589 7-bit characters; 14,360 wide characters It was higher than I expected, although not that surprising when looking at the source when you have things like: <div class="vector-page-toolbar"> <div class="vector-page-toolbar-container"> <div id="left-navigation"> <nav aria-label="名前空間">
    - knolax 4y ago
      Your argument uses a website whose code was written by English speakers. There would still be ASCII but verbose element names like "vector-page-toolbar-container" would definitely be shorter both in utf-8 and in utf-16 if they weren't written in English
      - Beltalowda 4y ago
- fomine3 4y ago
  ISO 2022 isn't beautiful solution like UTF-8 self sync encoding. Backward compatibility is also great (so called UTF FSS). It was designed by smart Plan9 people.
  - manholio 4y ago
    If Ken and Pike could design it in an afternoon, I'm sure even a standards committee could have done it in less than 5 years or so, the ideas are straight forward. I would also argue that the fast sync with at most one character consumed is a good feature but not essential for adoption. Any byte-oriented Unicode packing scheme that can gracefully consume ASCII is better than multibyte, codepages sent out-of-band etc. etc.

zweifuss 4y ago

I have asked myself the same question. However, I observed that the switch from UCS-2 to true UTF-16 was already very bumpy, with strange bugs and tons of edge cases that only got significantly better in Windows 8. A forced switch to UTF-8 would likely have broken ANSI strings. Languages such as PHP didn't get true UTF-8 support until 2015, Python in 2008 (and that switch took a decade), and Java and JavaScript continue to use UTF-16 as default strings. The wastefulness of UTF-16 is mostly remedied by Unicode Compression (SCSU), which can be much shorter than UTF-8 in languages like Hindi and Chinese.

wizofaus 4y ago
How many Windows users do benefit significantly from using UCS-2 encoding, I wonder? Chinese and Hindi (along with Japanese, South Korean and various Arabic language) speakers would surely come close to outnumbering those of us using European alphabets (I gather that Cyrillic & Greek have single-byte encodings, but maybe they mostly use UCS-2 these days too?). On that basis it seems a reasonable choice as the default storage method for human readable strings. But if course a high percentage of strings don't exist for that purpose at all, at least not primarily - using UCS-2 for storing/ transmitting CSS or configuration files or of course source code for most languages is less justifiable.
- zweifuss 4y ago
  Microsoft software supports UTF-16 (not only UCS-2) everywhere these days (yes, even in COM). But you can probably still find errors here and there. I have no problem with saving text files in UTF-8 and normally do. I just wish SCSU was handled by COM by default as well and not only sometimes.
- ripe 4y ago
  > Chinese and Hindi (along with Japanese, South Korean and various Arabic language) speakers would surely come close to outnumbering those of us using European alphabets You are probably right. Just slightly off topic: Hindi speakers who use computers mostly seem happy enough with English-language forms, instructions, and technical documents. This is true of most Indian users who speak languages that use one of the Indic scripts. Actual non-English text is useful mainly for publications like newspapers, magazines, and books, which native speakers do prefer to consume in their own language. The above is, of course, not counting the significant numbers of Indians who are quite comfortable using English for daily communication. These Anglophones are just like US speakers in their preferences and have no use for internationalization.

Mikhail_Edoshin 4y ago

UTF-16 is simpler and more efficient to handle than UTF-8.

netheril96 4y ago
> UTF-16 is simpler and more efficient to handle than UTF-8. Only if you never go out of the basic multilingual plane. And then find out so many bugs when people start using emojis.
ainar-g 4y ago
I feel like you're confusing it with UTF-32, which is fixed-length. UTF-16 is neither fixed-length, like UTF-32, nor compatible with the old ASCII encoding, like UTF-8. The historical reasons of why UTF-16 exists are fully understandable, but by now it is clear that it's the worst of both worlds.
- Mikhail_Edoshin 4y ago
  No, I'm fully aware of it being a variable-length encoding. Yet as such it is still simpler to work with than UTF-8. So if we are to use a variable-length encoding anyway then UTF-16 is a reasonable choice. It is not compatible with ASCII and old APIs, that's true, but I cannot judge how important this is. I guess that for some projects the efficiency of UTF-16 should outweigh the inconveniences of that incompatibility.
  - heinrich5991 4y ago
    Can you explain in what way it is easier to handle than UTF-8? Most code that I've seen and looks easier just mishandles characters outside the basic multilingual plane. Thankfully, due to emojis being a thing, a lot of developers woke up to the fact that Unicode specifies more than 65536 characters.
    - Const-me 4y ago
      UTF-16 is cheaper to decode. All CPUs have branch predictors, 99.99% of all UTF-16 strings don't contain any surrogate pairs. For East Asian languages UTF-16 uses 50% less memory, these characters take 3 bytes in UTF-8, but only 2 bytes in UTF-16.

fredoralive 4y ago

IIRC the attitude was that the "ASCII" versions of functions largely existed only for backwards compatibility, after all on NT[1] they're basically just wrappers that do character set conversion before passing through to the real Unicode (UCS2/UTF16) functions. The view was that if you want Unicode, you should use the real functions directly, which means 16 bit strings. I think there was also some compatibility concerns, as Windows for a while only had a single global "ASCII" character set used by all apps[2], so you'd have issues with actual legacy apps expecting a real legacy code page breaking / having scrambled text if you tried to use UTF8 as the "legacy" encoding. Or the fun idea of an app that is clever enough to know that multibyte exists, but dumb enough to know they can only be one or two bytes long, and allocating buffers accordingly. I think there might have been some issues with how things like the clipboard handled text as well that assumed all "ASCII" apps used the same encoding. (Disclaimed: old memories of reading blogs, may be bollocks). At some point things must have been fixed so multiple character sets could be used though. [1] But not on 9x, which was "ASCII" only, at least until late in the day when a Unicode compatibility library was created. [2] The "Language for non-Unicode programs" as the Control Panel calls it.

nikbackm 4y ago
The "A" stands for "ANSI", not "ASCII". Otherwise largely correct.

debugnik 4y ago

> and finally got the proper support in Windows 10 Not quite, you still can't use some features like longPathAware with the "ANSI" APIs. The preferred APIs for file paths are still the ones using wchar, although the UTF8 codepage is apparently the preferred method for non-path APIs and console IO, at least on the GDK.

cesarb 4y ago

> It seems the solution would be simple: once UTF-8 introduced (back in 1996!), add new "UTF-8" codepage, add new "SetDefaultCodepageToUnicode()" function, and tell everyone that wchar_t is legacy and should not be used anymore. They could probably do it in time for Windows XP! IIRC, the problem was that a MBCS codepage could have a maximum of 2 bytes per character, while UTF-8 could need 3 or even 4 bytes per character. Legacy applications could have fixed-size buffers with space for only 2 bytes per character, and would break if the default codepage needed more than that. > Instead, UTF-8 codepage was only introduced in Windows 7 (but had incomplete support), and finally got the proper support in Windows 10. That's a lot of time Windows programmers were forced to use UCS-2. It appears to me that lately Windows hasn't been as obsessed with backwards compatibility as it had been in the past. They are also gradually allowing for more use of paths longer than MAX_PATH (260 characters), which are not accessible to legacy applications, and as everyone knows, the compatibility with 16-bit applications has been dropped some time ago.

theamk 4y ago
That's a good point... so don't make it default! Make "SetThreadLocale" or something work with UTF-8, so apps could switch one-by-one (this may not work with hooks but still better than nothing...). Add UTF-8 support to MSVC runtime, so at least fopen can be safe (people have been asking for it forever). Add FILE_FLAG_UTF_8 to CreateFileA. Have Notepad be able to edit UTF-8. But I suspect the real reason is there was a strong internal opposition to UTF-8 -- maybe it was seen as inferior to UTF-16, maybe it was NIH, maybe they hoped the world will change to follow Windows ways and to put BOM marks in each document... Oh well, I am glad they have at least changed now.

vintermann 4y ago

They couldn't just jump on every technically superior idea, because compatibility is a thing too. And, by the time everyone agreed that utf-8 was preferable, compatibility with their own stuff, which is always especially painful. It's not as if they were the only ones. I had a lot of utf-related pain in the Linux terminal back in the day. Vi and Emacs took a long time to handle variable length encodings well. My old Unix editor of choice, NEdit, never managed the switch at all. I got a mail from my boss in utf-7 once. God knows how that happened.