4 ms·

I don't agree with the section about JavaScript strings. Those are proper strings, just encoded in UTF-16. > JavaScript’s string type is backed by a sequence o

I don't agree with the section about JavaScript strings. Those are proper strings, just encoded in UTF-16. > JavaScript’s string type is backed by a sequence of unsigned 16-bit integers, so it can’t hold any codepoint higher than U+FFFF and instead splits them into surrogate pairs. You just contradicted yourself. Surrogate pairs is exactly what allows UTF-16 to encode any codepoint. Once you start talking about in-memory representation, you need to agree on an encoding. UTF-8, UTF-16 being the most common. wchar_t could be UTF-16 or UCS-2.

masklinn 11y ago
Javascript strings are not UTF-16, you'd only ever see codepoints if that were the case. Javascript "strings" are UCS2, it's trivial to demonstrate: "\ud83c" is a valid Javascript string, it's not valid UTF-16. Here's the relevant section of the Unicode FAQ on the subject: > UCS-2 does not describe a data format distinct from UTF-16, because both use exactly the same 16-bit code unit representations. However, UCS-2 does not interpret surrogate code points, and thus cannot be used to conformantly represent supplementary characters. A correct UTF-16 implementation would interpret surrogate code point, validate that they're paired and prevent access to either surrogate via string operations.
- wereHamster 11y ago
  ES6 did get some new functions to correctly deal with surrogate pairs in strings. In then end, JS strings are just an sequence of 16 bit values, with the unfortunate case that many string functions interpret those as UCS-2 and only some new functions as UTF-16. When you come across an invalid sequence while decoding particular input (like "\ud83c") then you generally have three choices: throw an exception, skip the invalid part, or replace it with a replacement character. The default JavaScript behaviors is to be lenient. But if you need more control over the decoding behavior then you can use StringView or TextDecoder which is part of this spec: https://encoding.spec.whatwg.org/ https://encoding.spec.whatwg.org/
  - masklinn 11y ago
    > ES6 did get some new functions to correctly deal with surrogate pairs in strings. In then end, JS strings are just an sequence of 16 bit values Which is exactly why they are not and can not be UTF-16. > The default JavaScript behaviors is to be lenient. The javascript behaviour is to have UCS2 "strings".
Dylan16807 11y ago
But they're not a sequence of codepoints. They're a series of 16 bit values that can be set to anything, even invalid unicode. JavaScript has byte strings, not character strings.
- ygra 11y ago
  For very wide bytes ;) I'd a sequence of code units instead of code points. Sadly that holds true for many string implementations in programming languages, often for history, compatibility, or efficiency reasons (e.g. C#could have done it right, being designed after the UCS-2/UTF-16 split, but they didn't for various reasons). So you get code unit sequences with a few functions tacked on top that add code point support.
  - Dylan16807 11y ago
    Huh, I was wrong, there's no such thing as an invalid code unit. Even U+FFFE and U+FFFF are allowed.

masklinn 11y ago

Javascript strings are not UTF-16, you'd only ever see codepoints if that were the case. Javascript "strings" are UCS2, it's trivial to demonstrate: "\ud83c" is a valid Javascript string, it's not valid UTF-16. Here's the relevant section of the Unicode FAQ on the subject: > UCS-2 does not describe a data format distinct from UTF-16, because both use exactly the same 16-bit code unit representations. However, UCS-2 does not interpret surrogate code points, and thus cannot be used to conformantly represent supplementary characters. A correct UTF-16 implementation would interpret surrogate code point, validate that they're paired and prevent access to either surrogate via string operations.

wereHamster 11y ago
ES6 did get some new functions to correctly deal with surrogate pairs in strings. In then end, JS strings are just an sequence of 16 bit values, with the unfortunate case that many string functions interpret those as UCS-2 and only some new functions as UTF-16. When you come across an invalid sequence while decoding particular input (like "\ud83c") then you generally have three choices: throw an exception, skip the invalid part, or replace it with a replacement character. The default JavaScript behaviors is to be lenient. But if you need more control over the decoding behavior then you can use StringView or TextDecoder which is part of this spec: https://encoding.spec.whatwg.org/ https://encoding.spec.whatwg.org/
- masklinn 11y ago
  > ES6 did get some new functions to correctly deal with surrogate pairs in strings. In then end, JS strings are just an sequence of 16 bit values Which is exactly why they are not and can not be UTF-16. > The default JavaScript behaviors is to be lenient. The javascript behaviour is to have UCS2 "strings".

Dylan16807 11y ago

But they're not a sequence of codepoints. They're a series of 16 bit values that can be set to anything, even invalid unicode. JavaScript has byte strings, not character strings.

ygra 11y ago
For very wide bytes ;) I'd a sequence of code units instead of code points. Sadly that holds true for many string implementations in programming languages, often for history, compatibility, or efficiency reasons (e.g. C#could have done it right, being designed after the UCS-2/UTF-16 split, but they didn't for various reasons). So you get code unit sequences with a few functions tacked on top that add code point support.
- Dylan16807 11y ago
  Huh, I was wrong, there's no such thing as an invalid code unit. Even U+FFFE and U+FFFF are allowed.