4 ms·
The needs to convert character encodings should be gradually going away. There's not going to be more call for a standard library to address this in 2026 than t
by tialaramex 4y ago
The needs to convert character encodings should be gradually going away. There's not going to be more call for a standard library to address this in 2026 than there is today.
It's more crazy that C++ was so slow to mandate UTF-8 support. You have the situation where your modern C++ environment might have several distinct "string" types none of which is guaranteed to just be UTF-8.
- vvanders 4y agoA lot of the contexts where C++ has been deployed predate the shift to UTF-8 so I think there's actually more of a reason to have the support.
- nine_k 4y agoSay, for Japanese texts Shift-JIS is vastly more economical than UTF-8, so a number of systems uses it. Likely this is the case for some other writing systems. UTF-16 should work in there cases eventually though.
- astrange 4y agoMany of those texts include enough ASCII characters (like HTML tags) that UTF-8 is more efficient even then.
- tialaramex 4y agoFor ASCII (except the blackslash and tilde) Shift-JIS and UTF-8 are the same, one byte. So your HTML tags work just fine in Shift-JIS. For small Japanese-only text, Shift-JIS really will be smaller, but it's just not usually enough to care, and the price is you can't do Unicode. It occupies a similar space to 8859-1 / Windows 1252 where older software uses it but you should just transition.
- astrange 4y agoOh, I meant UTF-8 would be more efficient than UTF-16.
- int_19h 4y agoThe need to convert UTF-8 <-> UTF-16 is not going to go away anytime soon. Many existing formats or APIs specify one or the other.
- mlindner 4y agoThere are only two types of APIs, those that support UTF-8 and broken APIs. Users should actively file bug reports with any system or product that insists on UTF-16 or abandon use of such legacy systems if they refuse to change.
- flohofwoe 4y agoUTF-16 is only a pain when working with Windows APIs though. Most other 'legacy APIs' I've seen so far provide easy ways to get UTF-8 encoded data in and out, even though they use UTF-16 under the hood (like [NSString stringWithUTF8String:] on macOS).
- int_19h 4y agoWin32 also has MultiByteToWideChar and friends, although being a C API, it's not particularly pleasant to use. For newer HSTRING-based WinRT APIs, if you're calling them from C++, you're probably going to use C++/WinRT, which provides conversion functions that operate on std::string_view and std::wstring_view. So, you're right, I need to amend that to "mostly needed for file formats and protocols".
- anonymoushn 4y agoit seems like we are stuck with UTF-16-alikes until we manage to get rid of JavaScript, Java, and Windows
- Yoric 4y agoWait, JavaScript? Under the hood, JavaScript strings have a variety of encodings, but I can't think of any API that forces you to use a specific encoding.
- anonymoushn 4y agothe length property and the charAt method for example are defined to use UTF-16. length is the length in UTF-16 code units and charAt returns a UTF-16 code unit at a certain index.
- mlindner 4y agoThat's something that should be addressed with the javascript steering committees (if such things exist) so that the standard is updated to support UTF-8.