3 ms·

Plenty of applications just need to take strings in one format and pass them along in another format without doing feats of string processing to them.

int_19h 4y ago
To elaborate: UTF-8 is the most common I/O encoding, but UTF-16 is often what you need to work with the OS APIs (Win32, macOS) or popular frameworks (Qt).
- TillE 4y ago
  The funny thing is that C++ sort of has a flexible string class which uses system-dependent conversions, std::filesystem::path It's super convenient for working with Windows file APIs, but everything else is still a pain.
- Ferrotin 4y ago
  Yep. There are still some annoyances like invalid UTF-16, so maybe there is more is needed than just the basic transformations — some precise handling or parameters specifying how to handle these edge cases could be necessary. Handling UTF-8 versus WTF-8, and properly round-tripping these representations, is probably a bigger issue than graphemes and canonical forms for most users.
- josephg 4y ago
  Also unfortunately the web browser. Javascript strings are UTF-16 (Well, UCS2).
  - Yoric 4y ago
    This used to be the case when JS was created. However, I don't think that this is true anymore. It's definitely not under the hood. Nor, if I recall correctly, if you attempt to convert them from/to typed arrays. Maybe if you use `fromCharCode`?
    - Ferrotin 4y ago
      JS strings are exposed as UTF-16 in the language — surrogate pairs have length 2 and you can index into half of them.
    - erutuon 4y ago
      A JavaScript string is conceptually an array of 16-bit unsigned integers (https://tc39.es/ecma262/multipage/ecmascript-data-types-and-values.html#sec-ecmascript-language-types-string-type https://tc39.es/ecma262/multipage/ecmascript-data-types-and-...). String indexing and substring operations treat strings this way, and they couldn't change the model easily without breaking code. If you index a string containing an emoji with a code point greater than U+FFFF, you get string containing an unpaired surrogate code unit, which makes the string UCS-2, not UTF-16. I don't know how it's implemented memory-wise in the various JavaScript engines and am not familiar with typed arrays.
    - josephg 4y ago
      > It's definitely not under the hood. Really? I'd love to see some references to that. I know there's some operations to convert a javascript string to UTF8, but everything which measures a string counts using 2-byte UCS2 items. For example, array-style indexing, string.length, str.charCodeAt(), split(), slice(), indexOf(), etc. Also lots of DOM APIs use those weird string lengths too. For example, input element selectionStart / selectionEnd fields measure the selection range by counting surrogate pairs, not characters.
      - Yoric 4y ago
        > Really? I'd love to see some references to that. I remember that SpiderMonkey shifted to multiple encodings around 2019 and v8 had done that earlier. Source: I was part of the SpiderMonkey team when the shift happened.

int_19h 4y ago

To elaborate: UTF-8 is the most common I/O encoding, but UTF-16 is often what you need to work with the OS APIs (Win32, macOS) or popular frameworks (Qt).

TillE 4y ago
The funny thing is that C++ sort of has a flexible string class which uses system-dependent conversions, std::filesystem::path It's super convenient for working with Windows file APIs, but everything else is still a pain.
Ferrotin 4y ago
Yep. There are still some annoyances like invalid UTF-16, so maybe there is more is needed than just the basic transformations — some precise handling or parameters specifying how to handle these edge cases could be necessary. Handling UTF-8 versus WTF-8, and properly round-tripping these representations, is probably a bigger issue than graphemes and canonical forms for most users.
josephg 4y ago
Also unfortunately the web browser. Javascript strings are UTF-16 (Well, UCS2).
- Yoric 4y ago
  This used to be the case when JS was created. However, I don't think that this is true anymore. It's definitely not under the hood. Nor, if I recall correctly, if you attempt to convert them from/to typed arrays. Maybe if you use `fromCharCode`?
  - Ferrotin 4y ago
    JS strings are exposed as UTF-16 in the language — surrogate pairs have length 2 and you can index into half of them.
  - erutuon 4y ago
    A JavaScript string is conceptually an array of 16-bit unsigned integers (https://tc39.es/ecma262/multipage/ecmascript-data-types-and-values.html#sec-ecmascript-language-types-string-type https://tc39.es/ecma262/multipage/ecmascript-data-types-and-...). String indexing and substring operations treat strings this way, and they couldn't change the model easily without breaking code. If you index a string containing an emoji with a code point greater than U+FFFF, you get string containing an unpaired surrogate code unit, which makes the string UCS-2, not UTF-16. I don't know how it's implemented memory-wise in the various JavaScript engines and am not familiar with typed arrays.
  - josephg 4y ago
    > It's definitely not under the hood. Really? I'd love to see some references to that. I know there's some operations to convert a javascript string to UTF8, but everything which measures a string counts using 2-byte UCS2 items. For example, array-style indexing, string.length, str.charCodeAt(), split(), slice(), indexOf(), etc. Also lots of DOM APIs use those weird string lengths too. For example, input element selectionStart / selectionEnd fields measure the selection range by counting surrogate pairs, not characters.
    - Yoric 4y ago
      > Really? I'd love to see some references to that. I remember that SpiderMonkey shifted to multiple encodings around 2019 and v8 had done that earlier. Source: I was part of the SpiderMonkey team when the shift happened.