4 ms·

Even on Windows it's best to keep your text in UTF-8 and convert it to and from UTF-16 when interacting with win32 APIs. Java, dotNet and JavaScript are the wor

by Asooka 7y ago

Even on Windows it's best to keep your text in UTF-8 and convert it to and from UTF-16 when interacting with win32 APIs. Java, dotNet and JavaScript are the worst of all worlds because you're both stuck with wide characters (in their native string types) and have the intricacies of UTF-16 to consider. I guess the advice might have been better phrased as "Unless you're forced to, or have very special needs, stay away from UTF-16".

Const-me 7y ago
> it's best to keep your text in UTF-8 and convert it to and from UTF-16 when interacting with win32 APIs It’s extra source code to write and then support, extra machine code to execute, and likely extra memory to malloc/free. Too slow, in my book automatically means “not best”. > Java, dotNet and JavaScript are the worst of all worlds because you're both stuck with wide characters (in their native string types) and have the intricacies of UTF-16 to consider. Just a normal UTF-16, like in WinAPI and many other popular languages, frameworks and libraries. E.g. QT is used a lot in the wild. > the advice might have been better phrased as It says exactly the opposite, “Use the encoding preferred by your library and convert to/from UTF-8 at the edges of the program.”
- fwip 7y ago
  "Too" slow depends on a lot of factors.
  - Const-me 7y ago
    When you’re writing code that you 100% sure won’t ever become a performance bottleneck, you still care about time of development. Very often, unless it’s a throwaway code, also about cost of support. Writing any code at all when that code is not needed is always too slow, this is regardless of any technical factors.
    - fwip 7y ago
      Very little code in this world is needed. Much of it is, however, useful. The person you replied to obviously isn't advocating for something they find useless. Perhaps you could have instead asked "Why do you recommend doing this? I don't understand the benefit." But instead, you decided that they're advocating to do something useless for no reason.
      - Const-me 7y ago
        > you decided that they're advocating to do something useless for no reason. No, I decided they’re advocating to do something harmful for no reason. They're advocating to waste hardware resources (as a developer I don’t like doing that), waste development time (as a manager I don’t like when developers do that). But the worst of all, UTF8 on Windows and converting to/from UTF16 at WinAPI boundary is a source of bugs, the kernel doesn’t guarantee the bytes you get from these APIs are valid UTF16, quite the opposite, it guarantees to treat them as opaque chunk of words. UTF-8 has it’s place even on Windows, e.g. it makes sense for some network services, and even for RAM data when you know it’ll be 99% English so it saves resources, and that data never hits WinAPI. But as soon as you’re consuming WinAPI, COM, UWP, windows shell, any other native stuff, UTF-8 is just not good.
- vardump 7y ago
  Spot on. When coding against "raw" win32 API (or NT kernel APIs and perhaps rare native usermode NT API), using UTF-16 is the only way to keep your sanity. Converting strings back and forth between UTF-8 and UTF-16 in that kind of case is just senseless waste of CPU cycles. One API call might take multiple strings and each conversion often means memory allocation and freeing — something you usually try to avoid as much as possible if it's something that's going to run most of the time the system is powered on. The situation can be different in cross-platform code. In those cases, UTF-8 is a preferable abstraction. Just don't use it for filenames. Filenames are just bags of bytes on at least on Windows (well, 16-bit WCHARs, but the idea is same) and Linux, and considering them anything else is not a great idea.
ChrisSD 7y ago
That very much depends on what you're doing. Constantly reencoding between UTF-16 and UTF-8 would be pointless. Not to mention that "UTF-16" on Windows usually means UCS-2, so you risk losing information if you reencode. But if your application's strings are mostly independent of the WinAPI then sure, use UTF-8 and only convert when absolutely necessary.
- Const-me 7y ago
  > "UTF-16" on Windows usually means UCS-2 Wikipedia says it's UTF-16 since Windows 2000: https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows Supporting Windows NT4 or Windows 95 in 2019 is what I would call "very special needs".
  - ChrisSD 7y ago
    Wikipedia is wrong on the technical details. E.g. the filesystem accepts any sequence of WCHARs, whether or not they're valid UTF-16: https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file https://docs.microsoft.com/en-us/windows/desktop/FileIO/nami... > the file system treats path and file names as an opaque sequence of WCHARs. The same is true more generally, there's no validation so anything goes.
    - Const-me 7y ago
      From your comment I was replying to: > "UTF-16" on Windows usually means UCS-2, so you risk losing information if you reencode. On Windows, you normally call this API to convert UTF-8 to UTF-16: https://docs.microsoft.com/en-us/windows/desktop/api/stringapiset/nf-stringapiset-multibytetowidechar https://docs.microsoft.com/en-us/windows/desktop/api/stringa... As you see, the documentation says it converts to UTF-16, not UCS-2, so no information is lost re-encoding. And the article you’ve linked says “file system treats path and file names as an opaque sequence of WCHARs.” This means no information is lost in the kernel, either. Indeed, kernel doesn’t validate nor normalize these WCHARs, but should it? I would be very surprised if I ask an OS kernel to create a file, and it silently changed the name doing some Unicode normalization. Linux kernel doesn’t do that either, https://www.kernel.org/doc/html/latest/admin-guide/ext4.html https://www.kernel.org/doc/html/latest/admin-guide/ext4.html says “the file name provided by userspace is a byte-per-byte match to what is actually written in the disk”

by Asooka 7y ago

Const-me 7y ago

> it's best to keep your text in UTF-8 and convert it to and from UTF-16 when interacting with win32 APIs It’s extra source code to write and then support, extra machine code to execute, and likely extra memory to malloc/free. Too slow, in my book automatically means “not best”. > Java, dotNet and JavaScript are the worst of all worlds because you're both stuck with wide characters (in their native string types) and have the intricacies of UTF-16 to consider. Just a normal UTF-16, like in WinAPI and many other popular languages, frameworks and libraries. E.g. QT is used a lot in the wild. > the advice might have been better phrased as It says exactly the opposite, “Use the encoding preferred by your library and convert to/from UTF-8 at the edges of the program.”

fwip 7y ago
"Too" slow depends on a lot of factors.
- Const-me 7y ago
  When you’re writing code that you 100% sure won’t ever become a performance bottleneck, you still care about time of development. Very often, unless it’s a throwaway code, also about cost of support. Writing any code at all when that code is not needed is always too slow, this is regardless of any technical factors.
  - fwip 7y ago
    Very little code in this world is needed. Much of it is, however, useful. The person you replied to obviously isn't advocating for something they find useless. Perhaps you could have instead asked "Why do you recommend doing this? I don't understand the benefit." But instead, you decided that they're advocating to do something useless for no reason.
    - Const-me 7y ago
      > you decided that they're advocating to do something useless for no reason. No, I decided they’re advocating to do something harmful for no reason. They're advocating to waste hardware resources (as a developer I don’t like doing that), waste development time (as a manager I don’t like when developers do that). But the worst of all, UTF8 on Windows and converting to/from UTF16 at WinAPI boundary is a source of bugs, the kernel doesn’t guarantee the bytes you get from these APIs are valid UTF16, quite the opposite, it guarantees to treat them as opaque chunk of words. UTF-8 has it’s place even on Windows, e.g. it makes sense for some network services, and even for RAM data when you know it’ll be 99% English so it saves resources, and that data never hits WinAPI. But as soon as you’re consuming WinAPI, COM, UWP, windows shell, any other native stuff, UTF-8 is just not good.
vardump 7y ago
Spot on. When coding against "raw" win32 API (or NT kernel APIs and perhaps rare native usermode NT API), using UTF-16 is the only way to keep your sanity. Converting strings back and forth between UTF-8 and UTF-16 in that kind of case is just senseless waste of CPU cycles. One API call might take multiple strings and each conversion often means memory allocation and freeing — something you usually try to avoid as much as possible if it's something that's going to run most of the time the system is powered on. The situation can be different in cross-platform code. In those cases, UTF-8 is a preferable abstraction. Just don't use it for filenames. Filenames are just bags of bytes on at least on Windows (well, 16-bit WCHARs, but the idea is same) and Linux, and considering them anything else is not a great idea.

ChrisSD 7y ago

That very much depends on what you're doing. Constantly reencoding between UTF-16 and UTF-8 would be pointless. Not to mention that "UTF-16" on Windows usually means UCS-2, so you risk losing information if you reencode. But if your application's strings are mostly independent of the WinAPI then sure, use UTF-8 and only convert when absolutely necessary.

Const-me 7y ago
> "UTF-16" on Windows usually means UCS-2 Wikipedia says it's UTF-16 since Windows 2000: https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows Supporting Windows NT4 or Windows 95 in 2019 is what I would call "very special needs".
- ChrisSD 7y ago
  Wikipedia is wrong on the technical details. E.g. the filesystem accepts any sequence of WCHARs, whether or not they're valid UTF-16: https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file https://docs.microsoft.com/en-us/windows/desktop/FileIO/nami... > the file system treats path and file names as an opaque sequence of WCHARs. The same is true more generally, there's no validation so anything goes.
  - Const-me 7y ago
    From your comment I was replying to: > "UTF-16" on Windows usually means UCS-2, so you risk losing information if you reencode. On Windows, you normally call this API to convert UTF-8 to UTF-16: https://docs.microsoft.com/en-us/windows/desktop/api/stringapiset/nf-stringapiset-multibytetowidechar https://docs.microsoft.com/en-us/windows/desktop/api/stringa... As you see, the documentation says it converts to UTF-16, not UCS-2, so no information is lost re-encoding. And the article you’ve linked says “file system treats path and file names as an opaque sequence of WCHARs.” This means no information is lost in the kernel, either. Indeed, kernel doesn’t validate nor normalize these WCHARs, but should it? I would be very surprised if I ask an OS kernel to create a file, and it silently changed the name doing some Unicode normalization. Linux kernel doesn’t do that either, https://www.kernel.org/doc/html/latest/admin-guide/ext4.html https://www.kernel.org/doc/html/latest/admin-guide/ext4.html says “the file name provided by userspace is a byte-per-byte match to what is actually written in the disk”