10 ms·

Even Microsoft is finally giving up UTF-16! They recommend now to use the UTF-8 "code page" in new code.

by jfkebwjsbx 7y ago

nathanaldensr 7y ago
Do you have a source for this? AFAIK the .NET Framework CLR and CoreCLR both still store strings internally as UTF-16.
- ChrisSD 7y ago
  The closest I could find to a recommendation for UTF-8 is in UWP design guidelines: https://docs.microsoft.com/en-us/windows/uwp/design/globalizing/use-utf8-code-page https://docs.microsoft.com/en-us/windows/uwp/design/globaliz... However it's not quite unequivocal. Windows still uses UTF-16 in the kernel (or actually an array of 16bit integers, but UTF-16 is a very strong convention). The code page will often allow the Win32 API to perform the conversion back and forth instead of your application doing it.
- mormegil 7y ago
  AFAICT, it's not only "internal representation". .NET strings are defined as a sequence of UTF-16 units, including the definition of the Char type representing a single UTF-16 code unit. I can't imagine how such a change could be implemented (other than changing the internal representation but converting on all accesses which would be nonsense, I think).
  - leosarev 7y ago
    Current plan is: https://github.com/dotnet/corefxlab/issues/2350 https://github.com/dotnet/corefxlab/issues/2350
- leosarev 7y ago
  CoreCLR actively discussing introducing Utf8String type. https://github.com/dotnet/corefxlab/issues/2350 https://github.com/dotnet/corefxlab/issues/2350
gpvos 7y ago
Have they fixed all the bugs with that pseudocodepage?
- xeeeeeeeeeeenu 7y ago
  Bugs like WriteFile() reporting the wrong number of bytes written with 65001 codepage were fixed years ago.
  - buckminster 7y ago
    That's good news. Last time I looked, more than a decade ago admittedly, that bug was WONTFIX. In fact I was so surprised I just wrote a test program. They have fixed it! It was the dumbest bug I ever saw in Windows. It was special case code in the console output code path of the user mode part of WriteFile. It only existed to make utf8 work, and it didn't even do that.
  - gpvos 7y ago
    Ah, that's surprising, Microsoft was very stubbornly not doing that for at least a decade and a half. In fact, the FAQ in TFA (questions 9 and 20) mentions that there are still problems with CP_UTF8 (65001). Is the article out of date? Can someone respond to those statements?
    - xeeeeeeeeeeenu 7y ago
      The article is outdated, it's from 2012. Not only they fixed the problems but in Windows 10 1803 they also added an option to globally and permanently set both OEM and ANSI(!) codepages to 65001. It can be enabled by checking "Beta: Use Unicode UTF-8 for worldwide language support" checkbox in region settings.
snazz 7y ago
Is java.lang.String still UTF-16? Is there any plan to fix that? Once Windows and Java take care of it, I can't think of any other major UTF-16 uses left. Are there any that I've forgotten about? Edit: Still looks like UTF-16, according to the Oracle documentation page: https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/lang/String.html https://docs.oracle.com/en/java/javase/14/docs/api/java.base... Edit 2: JavaScript too. See my reply to someone else below.
- lokedhs 7y ago
  I think it will be hard to change that. But it's not alone. Javascript also uses UTF-16.
  - snazz 7y ago
    You’re right! I’m surprised I didn’t know that. It looks like it can also be UCS-2, going by the spec: > A conforming implementation of this International standard shall interpret characters in conformance with the Unicode Standard, Version 3.0 or later and ISO/IEC 10646-1 with either UCS-2 or UTF-16 as the adopted encoding form, implementation level 3. If the adopted ISO/IEC 10646-1 subset is not otherwise specified, it is presumed to be the BMP subset, collection 300. If the adopted encoding form is not otherwise specified, it is presumed to be the UTF-16 encoding form.
    - im3w1l 7y ago
      USC-2 is an old version of UTF-16 that lacks support for surrogate pairs, which means that rare symbols and emoji don't work.
- diroussel 7y ago
  Compact Strings were added in Java 9; https://openjdk.java.net/jeps/254 https://openjdk.java.net/jeps/254 So they can now be stored as one byte per character.
  - kllrnohj 7y ago
jdsully 7y ago
Are you sure? That will result in a conversion every time a string is passed to the kernel. Windows can handle utf-8 but it is not the native character set for the platform.
- JdeBP 7y ago
  There's a conversion in every ...A() function. Conversion between UTF-8 and WTF-16 is just more of the same, but without codepage lookup tables. (-:
  - mark-r 7y ago
    They probably still do a codepage lookup just for consistency.
  - Shebanator 7y ago
    WTF-16? I like it...
    - ekimekim 7y ago
      WTF-8 and WTF-16 are a thing: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ Basically WTF-16 is any sequence of 16-bit integers, and is thus a superset of UTF-16 (because UTF-16 doesn't allow certain combinations of integers, mainly surrogate code points that exist outside of surrogate pairs). Then WTF-8 is what you get if you naively transform invalid UTF-16 into UTF-8. It is a superset of UTF-8. This is very useful when dealing with applications like Java and Javascript that treat strings as sequences of 16-bit code points, even though not all such strings are valid UTF-16.
      - masklinn 7y ago
        > Basically WTF-16 is any sequence of 16-bit integers, and is thus a superset of UTF-16 (because UTF-16 doesn't allow certain combinations of integers, mainly surrogate code points that exist outside of surrogate pairs). If WTF-16 is the ability in potentia to store and return invalid UTF-16 without signalling errors, I don't know that there's any actual UTF-16 system out there to the possible exception of… HFS+ maybe?.
bostonvaulter2 7y ago
I hope they update the Language Server Protocol to use UTF-8 then! Nearly every language supports UTF-8 well, which is not so for UTF-16

Even Microsoft is finally giving up UTF-16! They recommend now to use the UTF-8 "code page" in new code.

nathanaldensr 7y ago

Do you have a source for this? AFAIK the .NET Framework CLR and CoreCLR both still store strings internally as UTF-16.

ChrisSD 7y ago
The closest I could find to a recommendation for UTF-8 is in UWP design guidelines: https://docs.microsoft.com/en-us/windows/uwp/design/globalizing/use-utf8-code-page https://docs.microsoft.com/en-us/windows/uwp/design/globaliz... However it's not quite unequivocal. Windows still uses UTF-16 in the kernel (or actually an array of 16bit integers, but UTF-16 is a very strong convention). The code page will often allow the Win32 API to perform the conversion back and forth instead of your application doing it.
mormegil 7y ago
AFAICT, it's not only "internal representation". .NET strings are defined as a sequence of UTF-16 units, including the definition of the Char type representing a single UTF-16 code unit. I can't imagine how such a change could be implemented (other than changing the internal representation but converting on all accesses which would be nonsense, I think).
- leosarev 7y ago
  Current plan is: https://github.com/dotnet/corefxlab/issues/2350 https://github.com/dotnet/corefxlab/issues/2350
leosarev 7y ago
CoreCLR actively discussing introducing Utf8String type. https://github.com/dotnet/corefxlab/issues/2350 https://github.com/dotnet/corefxlab/issues/2350

gpvos 7y ago

Have they fixed all the bugs with that pseudocodepage?

xeeeeeeeeeeenu 7y ago
Bugs like WriteFile() reporting the wrong number of bytes written with 65001 codepage were fixed years ago.
- buckminster 7y ago
  That's good news. Last time I looked, more than a decade ago admittedly, that bug was WONTFIX. In fact I was so surprised I just wrote a test program. They have fixed it! It was the dumbest bug I ever saw in Windows. It was special case code in the console output code path of the user mode part of WriteFile. It only existed to make utf8 work, and it didn't even do that.
- gpvos 7y ago
  Ah, that's surprising, Microsoft was very stubbornly not doing that for at least a decade and a half. In fact, the FAQ in TFA (questions 9 and 20) mentions that there are still problems with CP_UTF8 (65001). Is the article out of date? Can someone respond to those statements?
  - xeeeeeeeeeeenu 7y ago
    The article is outdated, it's from 2012. Not only they fixed the problems but in Windows 10 1803 they also added an option to globally and permanently set both OEM and ANSI(!) codepages to 65001. It can be enabled by checking "Beta: Use Unicode UTF-8 for worldwide language support" checkbox in region settings.

snazz 7y ago

Is java.lang.String still UTF-16? Is there any plan to fix that? Once Windows and Java take care of it, I can't think of any other major UTF-16 uses left. Are there any that I've forgotten about? Edit: Still looks like UTF-16, according to the Oracle documentation page: https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/lang/String.html https://docs.oracle.com/en/java/javase/14/docs/api/java.base... Edit 2: JavaScript too. See my reply to someone else below.

lokedhs 7y ago
I think it will be hard to change that. But it's not alone. Javascript also uses UTF-16.
- snazz 7y ago
  You’re right! I’m surprised I didn’t know that. It looks like it can also be UCS-2, going by the spec: > A conforming implementation of this International standard shall interpret characters in conformance with the Unicode Standard, Version 3.0 or later and ISO/IEC 10646-1 with either UCS-2 or UTF-16 as the adopted encoding form, implementation level 3. If the adopted ISO/IEC 10646-1 subset is not otherwise specified, it is presumed to be the BMP subset, collection 300. If the adopted encoding form is not otherwise specified, it is presumed to be the UTF-16 encoding form.
  - im3w1l 7y ago
    USC-2 is an old version of UTF-16 that lacks support for surrogate pairs, which means that rare symbols and emoji don't work.
diroussel 7y ago
Compact Strings were added in Java 9; https://openjdk.java.net/jeps/254 https://openjdk.java.net/jeps/254 So they can now be stored as one byte per character.
- kllrnohj 7y ago

jdsully 7y ago

Are you sure? That will result in a conversion every time a string is passed to the kernel. Windows can handle utf-8 but it is not the native character set for the platform.

JdeBP 7y ago
There's a conversion in every ...A() function. Conversion between UTF-8 and WTF-16 is just more of the same, but without codepage lookup tables. (-:
- mark-r 7y ago
  They probably still do a codepage lookup just for consistency.
- Shebanator 7y ago
  WTF-16? I like it...
  - ekimekim 7y ago
    WTF-8 and WTF-16 are a thing: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ Basically WTF-16 is any sequence of 16-bit integers, and is thus a superset of UTF-16 (because UTF-16 doesn't allow certain combinations of integers, mainly surrogate code points that exist outside of surrogate pairs). Then WTF-8 is what you get if you naively transform invalid UTF-16 into UTF-8. It is a superset of UTF-8. This is very useful when dealing with applications like Java and Javascript that treat strings as sequences of 16-bit code points, even though not all such strings are valid UTF-16.
    - masklinn 7y ago
      > Basically WTF-16 is any sequence of 16-bit integers, and is thus a superset of UTF-16 (because UTF-16 doesn't allow certain combinations of integers, mainly surrogate code points that exist outside of surrogate pairs). If WTF-16 is the ability in potentia to store and return invalid UTF-16 without signalling errors, I don't know that there's any actual UTF-16 system out there to the possible exception of… HFS+ maybe?.

bostonvaulter2 7y ago

I hope they update the Language Server Protocol to use UTF-8 then! Nearly every language supports UTF-8 well, which is not so for UTF-16