3 ms·
To elaborate: UTF-8 is the most common I/O encoding, but UTF-16 is often what you need to work with the OS APIs (Win32, macOS) or popular frameworks (Qt).
by int_19h 4y ago
To elaborate: UTF-8 is the most common I/O encoding, but UTF-16 is often what you need to work with the OS APIs (Win32, macOS) or popular frameworks (Qt).
- TillE 4y agoThe funny thing is that C++ sort of has a flexible string class which uses system-dependent conversions, std::filesystem::path It's super convenient for working with Windows file APIs, but everything else is still a pain.
- Ferrotin 4y agoYep. There are still some annoyances like invalid UTF-16, so maybe there is more is needed than just the basic transformations — some precise handling or parameters specifying how to handle these edge cases could be necessary. Handling UTF-8 versus WTF-8, and properly round-tripping these representations, is probably a bigger issue than graphemes and canonical forms for most users.
- josephg 4y agoAlso unfortunately the web browser. Javascript strings are UTF-16 (Well, UCS2).
- Yoric 4y agoThis used to be the case when JS was created. However, I don't think that this is true anymore. It's definitely not under the hood. Nor, if I recall correctly, if you attempt to convert them from/to typed arrays. Maybe if you use `fromCharCode`?
- Ferrotin 4y agoJS strings are exposed as UTF-16 in the language — surrogate pairs have length 2 and you can index into half of them.
- erutuon 4y agoA JavaScript string is conceptually an array of 16-bit unsigned integers (https://tc39.es/ecma262/multipage/ecmascript-data-types-and-values.html#sec-ecmascript-language-types-string-type https://tc39.es/ecma262/multipage/ecmascript-data-types-and-...). String indexing and substring operations treat strings this way, and they couldn't change the model easily without breaking code. If you index a string containing an emoji with a code point greater than U+FFFF, you get string containing an unpaired surrogate code unit, which makes the string UCS-2, not UTF-16. I don't know how it's implemented memory-wise in the various JavaScript engines and am not familiar with typed arrays.
- josephg 4y ago> It's definitely not under the hood. Really? I'd love to see some references to that. I know there's some operations to convert a javascript string to UTF8, but everything which measures a string counts using 2-byte UCS2 items. For example, array-style indexing, string.length, str.charCodeAt(), split(), slice(), indexOf(), etc. Also lots of DOM APIs use those weird string lengths too. For example, input element selectionStart / selectionEnd fields measure the selection range by counting surrogate pairs, not characters.
- Yoric 4y ago> Really? I'd love to see some references to that. I remember that SpiderMonkey shifted to multiple encodings around 2019 and v8 had done that earlier. Source: I was part of the SpiderMonkey team when the shift happened.