4 ms·

I struggle to see why you'd ever want UTF-16. If you're using a variable length encoding, might as well stick to UTF-8. If you want predictable sizes, there's U

by pdpi 6y ago

I struggle to see why you'd ever want UTF-16. If you're using a variable length encoding, might as well stick to UTF-8. If you want predictable sizes, there's UTF-32 instead. ~Also, DOM Strings are not UTF-16, they're UCS-16.~ EDIT: UCS-2, not UCS-16. Also, I'm confusing the DOM with EcmaScript, and even that hasn't been true in a while.

ChrisSD 6y ago
> Also, DOM Strings are not UTF-16, they're UCS-16. Hm, according to the spec they should be interpreted as UTF-16 but this isn't enforced by the language so it can contain unpaired surrogates: From https://heycam.github.io/webidl/#idl-DOMString https://heycam.github.io/webidl/#idl-DOMString > Such sequences are commonly interpreted as UTF-16 encoded strings [RFC2781] although this is not required... Nothing in this specification requires a DOMString value to be a valid UTF-16 string. From https://262.ecma-international.org/11.0/#sec-ecmascript-language-types-string-type https://262.ecma-international.org/11.0/#sec-ecmascript-lang... > The String type is the set of all ordered sequences of zero or more 16-bit unsigned integer values (“elements”) up to a maximum length of 253 - 1 elements. The String type is generally used to represent textual data in a running ECMAScript program, in which case each element in the String is treated as a UTF-16 code unit value... Operations that do interpret String values treat each element as a single UTF-16 code unit. However, ECMAScript does not restrict the value of or relationships between these code units, so operations that further interpret String contents as sequences of Unicode code points encoded in UTF-16 must account for ill-formed subsequences.
bikeshaving 6y ago
UTF-32 is a bit like giving up and saying code units equal code points right? I’m more interested in the comparison between UTF-8 and UTF-16, where UTF-8 requires 1 to 3 bytes just in the BMP, with 3 bytes for CJK characters. I’m saying that as a quick measure of the actual length of a string, UTF-16 is much more predictable and provides a nice intuitive estimation of how long text actually is, as well as providing a fair encoding for most commonly used languages. RE the UTF-16 vs UCS-2 stuff, that’s probably a distinction which has technical meaning but will collapse at some point because no one actually cares, much like the distinction between URI and URL.
- masklinn 6y ago
  > I’m saying that as a quick measure of the actual length of a string, UTF-16 is much more predictable Meaning the software will deal much less well when it's wrong. > and provides a nice intuitive estimation of how long text actually is Not really due to combining codepoints, which make it not useful. > as well as providing a fair encoding for most commonly used languages. Which we know effectively doesn't matter: it's essentially only a gain for pure CJK text being stored at rest, because otherwise the waste on ASCII will more than compensate for the gain.
josephg 6y ago
Yes UTF-16 / UCS-2 is a silly encoding and in retrospect it was a mistake. It matters because for awhile it was believed that 2 bytes would be enough to encode any Unicode character. During this time, lots of important languages appeared which figured a 2 byte fixed size encoding was better than a variable length encoding. So Java, javascript, C# and others all use UCS-2. Now they suffer the downsides of both using a variable length encodings and being memory inefficient. The string.length property in all these languages is almost totally meaningless. UTF-8 is the encoding you should generally always reach for when designing new systems, or when implementing a network protocol. Rust, Go and other newer languages all use UTF-8 internally because it’s better. Well, and in Go’s case because it’s author, Rob Pike also had a hand in inventing UTF-8. Ironically C and UNIX, which (mostly) stubbornly stuck with single byte character encodings generally works better with UTF-8 than a lot of newer languages.

by pdpi 6y ago

ChrisSD 6y ago

> Also, DOM Strings are not UTF-16, they're UCS-16. Hm, according to the spec they should be interpreted as UTF-16 but this isn't enforced by the language so it can contain unpaired surrogates: From https://heycam.github.io/webidl/#idl-DOMString https://heycam.github.io/webidl/#idl-DOMString > Such sequences are commonly interpreted as UTF-16 encoded strings [RFC2781] although this is not required... Nothing in this specification requires a DOMString value to be a valid UTF-16 string. From https://262.ecma-international.org/11.0/#sec-ecmascript-language-types-string-type https://262.ecma-international.org/11.0/#sec-ecmascript-lang... > The String type is the set of all ordered sequences of zero or more 16-bit unsigned integer values (“elements”) up to a maximum length of 253 - 1 elements. The String type is generally used to represent textual data in a running ECMAScript program, in which case each element in the String is treated as a UTF-16 code unit value... Operations that do interpret String values treat each element as a single UTF-16 code unit. However, ECMAScript does not restrict the value of or relationships between these code units, so operations that further interpret String contents as sequences of Unicode code points encoded in UTF-16 must account for ill-formed subsequences.

bikeshaving 6y ago

UTF-32 is a bit like giving up and saying code units equal code points right? I’m more interested in the comparison between UTF-8 and UTF-16, where UTF-8 requires 1 to 3 bytes just in the BMP, with 3 bytes for CJK characters. I’m saying that as a quick measure of the actual length of a string, UTF-16 is much more predictable and provides a nice intuitive estimation of how long text actually is, as well as providing a fair encoding for most commonly used languages. RE the UTF-16 vs UCS-2 stuff, that’s probably a distinction which has technical meaning but will collapse at some point because no one actually cares, much like the distinction between URI and URL.

masklinn 6y ago
> I’m saying that as a quick measure of the actual length of a string, UTF-16 is much more predictable Meaning the software will deal much less well when it's wrong. > and provides a nice intuitive estimation of how long text actually is Not really due to combining codepoints, which make it not useful. > as well as providing a fair encoding for most commonly used languages. Which we know effectively doesn't matter: it's essentially only a gain for pure CJK text being stored at rest, because otherwise the waste on ASCII will more than compensate for the gain.

josephg 6y ago

Yes UTF-16 / UCS-2 is a silly encoding and in retrospect it was a mistake. It matters because for awhile it was believed that 2 bytes would be enough to encode any Unicode character. During this time, lots of important languages appeared which figured a 2 byte fixed size encoding was better than a variable length encoding. So Java, javascript, C# and others all use UCS-2. Now they suffer the downsides of both using a variable length encodings and being memory inefficient. The string.length property in all these languages is almost totally meaningless. UTF-8 is the encoding you should generally always reach for when designing new systems, or when implementing a network protocol. Rust, Go and other newer languages all use UTF-8 internally because it’s better. Well, and in Go’s case because it’s author, Rob Pike also had a hand in inventing UTF-8. Ironically C and UNIX, which (mostly) stubbornly stuck with single byte character encodings generally works better with UTF-8 than a lot of newer languages.