4 ms·

Is java.lang.String still UTF-16? Is there any plan to fix that? Once Windows and Java take care of it, I can't think of any other major UTF-16 uses left. Are t

by snazz 7y ago

Is java.lang.String still UTF-16? Is there any plan to fix that? Once Windows and Java take care of it, I can't think of any other major UTF-16 uses left. Are there any that I've forgotten about? Edit: Still looks like UTF-16, according to the Oracle documentation page: https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/lang/String.html https://docs.oracle.com/en/java/javase/14/docs/api/java.base... Edit 2: JavaScript too. See my reply to someone else below.

lokedhs 7y ago
I think it will be hard to change that. But it's not alone. Javascript also uses UTF-16.
- snazz 7y ago
  You’re right! I’m surprised I didn’t know that. It looks like it can also be UCS-2, going by the spec: > A conforming implementation of this International standard shall interpret characters in conformance with the Unicode Standard, Version 3.0 or later and ISO/IEC 10646-1 with either UCS-2 or UTF-16 as the adopted encoding form, implementation level 3. If the adopted ISO/IEC 10646-1 subset is not otherwise specified, it is presumed to be the BMP subset, collection 300. If the adopted encoding form is not otherwise specified, it is presumed to be the UTF-16 encoding form.
  - im3w1l 7y ago
    USC-2 is an old version of UTF-16 that lacks support for surrogate pairs, which means that rare symbols and emoji don't work.
diroussel 7y ago
Compact Strings were added in Java 9; https://openjdk.java.net/jeps/254 https://openjdk.java.net/jeps/254 So they can now be stored as one byte per character.
- kllrnohj 7y ago
  Only for ASCII text. There is still no UTF-8 support (it's even called out as a non-goal in the JEP: "It is not a goal to use alternate encodings such as UTF-8 in the internal representation of strings.")
rimunroe 7y ago
JavaScript: https://www.ecma-international.org/ecma-262/5.1/#sec-2 https://www.ecma-international.org/ecma-262/5.1/#sec-2 > A conforming implementation of this Standard shall interpret characters in conformance with the Unicode Standard, Version 3.0 or later and ISO/IEC 10646-1 with either UCS-2 or UTF-16 as the adopted encoding form, implementation level 3. If the adopted ISO/IEC 10646-1 subset is not otherwise specified, it is presumed to be the BMP subset, collection 300. If the adopted encoding form is not otherwise specified, it presumed to be the UTF-16 encoding form. https://www.ecma-international.org/ecma-262/5.1/#sec-4.3.16 https://www.ecma-international.org/ecma-262/5.1/#sec-4.3.16 > A String value is a member of the String type. Each integer value in the sequence usually represents a single 16-bit unit of UTF-16 text. However, ECMAScript does not place any restrictions or requirements on the values except that they must be 16-bit unsigned integers.
projektfu 7y ago
I don't think it's a big deal for Java because it's always easy to transfer in from and out to UTF-8. Very few Java programs use UTF-16 as a persistence format, and Java-native applications can directly marshal strings around as they are a first-class datatype.
josefx 7y ago
I don't think they can fix that without completely breaking backwards compatibility. The basic char type in Java is defined as a 16 bit wide unsigned integer value and String doesn't abstract over that.
masklinn 7y ago
> Is java.lang.String still UTF-16? Yes. > Is there any plan to fix that? That's not really possible as strings are defined in terms of char and guarantee O(1) access to UTF16 code units. They might try to switch to "indexed UTF8" (as pypy did in the Python ecosystem whereas "CPython proper" refused to switch to UTF8 with the Python 3 upheaval and went with the death trap that is PEP 393 instead).

by snazz 7y ago

lokedhs 7y ago

I think it will be hard to change that. But it's not alone. Javascript also uses UTF-16.

snazz 7y ago
You’re right! I’m surprised I didn’t know that. It looks like it can also be UCS-2, going by the spec: > A conforming implementation of this International standard shall interpret characters in conformance with the Unicode Standard, Version 3.0 or later and ISO/IEC 10646-1 with either UCS-2 or UTF-16 as the adopted encoding form, implementation level 3. If the adopted ISO/IEC 10646-1 subset is not otherwise specified, it is presumed to be the BMP subset, collection 300. If the adopted encoding form is not otherwise specified, it is presumed to be the UTF-16 encoding form.
- im3w1l 7y ago
  USC-2 is an old version of UTF-16 that lacks support for surrogate pairs, which means that rare symbols and emoji don't work.

diroussel 7y ago

Compact Strings were added in Java 9; https://openjdk.java.net/jeps/254 https://openjdk.java.net/jeps/254 So they can now be stored as one byte per character.

kllrnohj 7y ago
Only for ASCII text. There is still no UTF-8 support (it's even called out as a non-goal in the JEP: "It is not a goal to use alternate encodings such as UTF-8 in the internal representation of strings.")

rimunroe 7y ago

JavaScript: https://www.ecma-international.org/ecma-262/5.1/#sec-2 https://www.ecma-international.org/ecma-262/5.1/#sec-2 > A conforming implementation of this Standard shall interpret characters in conformance with the Unicode Standard, Version 3.0 or later and ISO/IEC 10646-1 with either UCS-2 or UTF-16 as the adopted encoding form, implementation level 3. If the adopted ISO/IEC 10646-1 subset is not otherwise specified, it is presumed to be the BMP subset, collection 300. If the adopted encoding form is not otherwise specified, it presumed to be the UTF-16 encoding form. https://www.ecma-international.org/ecma-262/5.1/#sec-4.3.16 https://www.ecma-international.org/ecma-262/5.1/#sec-4.3.16 > A String value is a member of the String type. Each integer value in the sequence usually represents a single 16-bit unit of UTF-16 text. However, ECMAScript does not place any restrictions or requirements on the values except that they must be 16-bit unsigned integers.

projektfu 7y ago

I don't think it's a big deal for Java because it's always easy to transfer in from and out to UTF-8. Very few Java programs use UTF-16 as a persistence format, and Java-native applications can directly marshal strings around as they are a first-class datatype.

josefx 7y ago

I don't think they can fix that without completely breaking backwards compatibility. The basic char type in Java is defined as a 16 bit wide unsigned integer value and String doesn't abstract over that.

masklinn 7y ago

> Is java.lang.String still UTF-16? Yes. > Is there any plan to fix that? That's not really possible as strings are defined in terms of char and guarantee O(1) access to UTF16 code units. They might try to switch to "indexed UTF8" (as pypy did in the Python ecosystem whereas "CPython proper" refused to switch to UTF8 with the Python 3 upheaval and went with the death trap that is PEP 393 instead).