4 ms·

I wonder how long Windows can hold UTF-16 as the default native encodings. They recently added UTF-8 locale.

by ezoe 8y ago

ygra 8y ago
CP65001 has existed for quite a while. There just have been plenty of places where it never really worked (I think the console was one of them). Nonetheless, UTF-16 will most likely stay. They might add wrappers around all API functions that convert arguments first, so we'd get CreateWindow8 alongside CreateWindowW and CreateWindowA, but I wouldn't hold my breath. The duality of API functions was to enable programs to be built for NT and 9x with the same codebase. It was never really intended for people to call them directly to enable non-Unicode-aware applications on NT. The file system cannot migrate from UTF-16 anyway without breaking things, so that will stay as is. IMHO there's pretty much nothing to gain, except a lot of risk in trying to change Windows to UTF-8 internally and the current interface might be an inconvenience to some developers, but not as much as to justify changing the whole OS.
- repolfx 8y ago
  It's even harder than that - APIs have expected performance characteristics, and APIs that involve string manipulation can change their performance when needing to work directly with UTF-8, e.g. working with file paths might become slower.
- vbezhenar 8y ago
  When the world uses UTF-8 and you have to constantly recode from/to with every WinAPI call, that's definitely an unnecessary overhead.
  - badestrand 8y ago
    Obviously it is unnecessary but I doubt it affects the perceived speed of the average desktop application in any way.
- ChrisSD 8y ago
  To be clear, the Windows filesystem isn't even UTF-16. It's UCS-2, which is simply a sequence of 16 bit values. The difference is that not all sequences are valid UTF-16 (see surrogate pairs). Thus you have to be very careful when handling filesystem paths.
- ezoe 8y ago
  The problem is, the rest of the world decided to settle on UTF-8. We have some normalization disagreements but it's UTF-8 anyway. Now we have surrogate pairs and the IVS, there is no advantage of using UTF-16 at all. I will not surprise the Microsoft someday convert everything including kernel to filesystem to UTF-8 someday... or more likely they lost the market share.

I wonder how long Windows can hold UTF-16 as the default native encodings. They recently added UTF-8 locale.

by ezoe 8y ago

I wonder how long Windows can hold UTF-16 as the default native encodings. They recently added UTF-8 locale.

ygra 8y ago

CP65001 has existed for quite a while. There just have been plenty of places where it never really worked (I think the console was one of them). Nonetheless, UTF-16 will most likely stay. They might add wrappers around all API functions that convert arguments first, so we'd get CreateWindow8 alongside CreateWindowW and CreateWindowA, but I wouldn't hold my breath. The duality of API functions was to enable programs to be built for NT and 9x with the same codebase. It was never really intended for people to call them directly to enable non-Unicode-aware applications on NT. The file system cannot migrate from UTF-16 anyway without breaking things, so that will stay as is. IMHO there's pretty much nothing to gain, except a lot of risk in trying to change Windows to UTF-8 internally and the current interface might be an inconvenience to some developers, but not as much as to justify changing the whole OS.

repolfx 8y ago
It's even harder than that - APIs have expected performance characteristics, and APIs that involve string manipulation can change their performance when needing to work directly with UTF-8, e.g. working with file paths might become slower.
vbezhenar 8y ago
When the world uses UTF-8 and you have to constantly recode from/to with every WinAPI call, that's definitely an unnecessary overhead.
- badestrand 8y ago
  Obviously it is unnecessary but I doubt it affects the perceived speed of the average desktop application in any way.
ChrisSD 8y ago
To be clear, the Windows filesystem isn't even UTF-16. It's UCS-2, which is simply a sequence of 16 bit values. The difference is that not all sequences are valid UTF-16 (see surrogate pairs). Thus you have to be very careful when handling filesystem paths.
ezoe 8y ago
The problem is, the rest of the world decided to settle on UTF-8. We have some normalization disagreements but it's UTF-8 anyway. Now we have surrogate pairs and the IVS, there is no advantage of using UTF-16 at all. I will not surprise the Microsoft someday convert everything including kernel to filesystem to UTF-8 someday... or more likely they lost the market share.