5 ms·

Disappointed to see the continued use of wide strings in new projects.

twoodfin 14y ago
Has Microsoft given any indication of planning to abandon two byte strings in their future Windows C/C++ frameworks? That design seems pretty baked into their platform at this point, and would be presumably hard to change.
- abcd_f 14y ago
  You can't abandon them if you want to have fast code. It is possible to write code in a portable fashion, accepting utf-8 as a standard string encoding, but this will then require converting to/from utf-16 for virtually every Windows API call that takes in a string variable. Not hard to do (MultiByteToWideChar), but there's a non-negligible performance penalty.
plorkyeran 14y ago
As long as the Win32 API continues to not support UTF-8, using wide strings will remain the least bad option on Windows.
- marshray 14y ago
  But this isn't using the Win32 API. At least not directly, and certainly not on Linux. Unrelated: Where is this "unsignedchar" keyword coming from?
  - plorkyeran 14y ago
    It does use both the Win32 API and WinRT (which also wants wide strings) directly. It appears to use narrow strings on Linux. The "unsignedchar" appears to be a bug in the syntax highlighter. It was parsed as two tokens, so either it's insane enough to split tokens or it dropped the space.
    - marshray 14y ago
      By "this" I meant the example code. On the whole I hate to nitpick, it looks like a darn good piece of work.
- apaprocki 14y ago
  Windows chooses to use UTF-16 rather than UTF-8, but is it really that much of a concern?
  - plorkyeran 14y ago
    It makes writing portable code a pain; either you have to use different string types on different platforms or do a bunch of converting between utf-8 and utf-16. Many libraries support only narrow strings, so even in a Windows-only application you often end up having to convert between utf-8 and utf-16 quite a bit.
- asveikau 14y ago
  I don't know what you're talking about, Win32 supports UTF-8. I can pass CP_UTF8 into WideCharToMultiByte() just fine. :-) Kidding aside, I don't really see the issue. Do you get upset about in-memory representation of strings often? How about when using Java or Python? Is this not why there is an entire programming practice called "serialization"? Windows started supporting UCS-2 before UTF-8 existed, and so the internal representation on Windows remains 16 bits per char.
  - zurn 14y ago
    Python supports full Unicode on all platforms now (x). Who cares about in-memory representation as long as the user doesn't have to suffer from surrogates and all that horror. (x) pre-PEP393 there was a build option to use the limited 16-bit Unicode, was unfortunately popular on Windows
    - asveikau 14y ago
      Hm. I picked Python because I knew it was an outlier, but I don't know all the details. Does Python still suffer from the horrors of comparison between diacritics made up of combining characters and the same glyph as a pre-composed character? Seems like even if you expose strings as UTF-32 you'd still have that issue.
      - zurn 14y ago
        Afaik the combining character problem hasn't changed, you still have to use unicodedata.normalize() for that. But at least you can pass through Unicode strings cleanly. Are there other languages that handle this better?
        grey-area 14y ago
        The same problem occurs in Ruby, at least when dealing with the Mac OS X file system (which uses decomposed strings internally), you may receive NFC or NFD and don't know which you're getting.
        14y ago

Has Microsoft given any indication of planning to abandon two byte strings in their future Windows C/C++ frameworks? That design seems pretty baked into their platform at this point, and would be presumably hard to change.

abcd_f 14y ago
You can't abandon them if you want to have fast code. It is possible to write code in a portable fashion, accepting utf-8 as a standard string encoding, but this will then require converting to/from utf-16 for virtually every Windows API call that takes in a string variable. Not hard to do (MultiByteToWideChar), but there's a non-negligible performance penalty.

plorkyeran 14y ago

As long as the Win32 API continues to not support UTF-8, using wide strings will remain the least bad option on Windows.

marshray 14y ago
But this isn't using the Win32 API. At least not directly, and certainly not on Linux. Unrelated: Where is this "unsignedchar" keyword coming from?
- plorkyeran 14y ago
  It does use both the Win32 API and WinRT (which also wants wide strings) directly. It appears to use narrow strings on Linux. The "unsignedchar" appears to be a bug in the syntax highlighter. It was parsed as two tokens, so either it's insane enough to split tokens or it dropped the space.
  - marshray 14y ago
    By "this" I meant the example code. On the whole I hate to nitpick, it looks like a darn good piece of work.
apaprocki 14y ago
Windows chooses to use UTF-16 rather than UTF-8, but is it really that much of a concern?
- plorkyeran 14y ago
  It makes writing portable code a pain; either you have to use different string types on different platforms or do a bunch of converting between utf-8 and utf-16. Many libraries support only narrow strings, so even in a Windows-only application you often end up having to convert between utf-8 and utf-16 quite a bit.
asveikau 14y ago
I don't know what you're talking about, Win32 supports UTF-8. I can pass CP_UTF8 into WideCharToMultiByte() just fine. :-) Kidding aside, I don't really see the issue. Do you get upset about in-memory representation of strings often? How about when using Java or Python? Is this not why there is an entire programming practice called "serialization"? Windows started supporting UCS-2 before UTF-8 existed, and so the internal representation on Windows remains 16 bits per char.
- zurn 14y ago
  Python supports full Unicode on all platforms now (x). Who cares about in-memory representation as long as the user doesn't have to suffer from surrogates and all that horror. (x) pre-PEP393 there was a build option to use the limited 16-bit Unicode, was unfortunately popular on Windows
  - asveikau 14y ago
    Hm. I picked Python because I knew it was an outlier, but I don't know all the details. Does Python still suffer from the horrors of comparison between diacritics made up of combining characters and the same glyph as a pre-composed character? Seems like even if you expose strings as UTF-32 you'd still have that issue.
    - zurn 14y ago
      Afaik the combining character problem hasn't changed, you still have to use unicodedata.normalize() for that. But at least you can pass through Unicode strings cleanly. Are there other languages that handle this better?
      - grey-area 14y ago
        The same problem occurs in Ruby, at least when dealing with the Mac OS X file system (which uses decomposed strings internally), you may receive NFC or NFD and don't know which you're getting.
      - 14y ago