3 ms·
> However, we have two popular Unicode formats: UTF-8 and UTF-16 The fact we are still using UTF-16 still irks me to this day. UTF-16 (which is actually two di
by qalmakka 4y ago
> However, we have two popular Unicode formats: UTF-8 and UTF-16
The fact we are still using UTF-16 still irks me to this day. UTF-16 (which is actually two different encodings, not one, hence the need for a BOM) is basically a way to salvage all those platforms that hurried on the UCS-2 (aka, the original "UNICODE") bandwagon in the '90s hoping that by just doing s/char/wchar_t/g all their internationalization problems would be solved.
It did not go well, to say the least. UTF-16 and 32 are objectively worse than UTF-8, because they still are multibyte encodings, you still have to do normalization, ... while also having to deal with "char16_t" and the likes (I will not enter into the whole "TCHAR" fiasco).
Spoiler alert, the world is still full of allegedly "UTF-16 compliant" platforms out there are not, indeed, UTF-16 compliant, they just use 16 bit chars and hope for the best.
The whole idea "1 char = 1 character" is arguably a terrible idea in Unicode, though. You not only can have multibyte characters, but you can also have multirune characters, where multiple codepoints are normalized into a single displayed character (just think about ` + e = è). It's a mess and it's bound to be broken, and there's no real way to "fix that up" - ASCII's assumption of "1 value = 1 char" was the broken concept here, and it unfortunately flawed how every developer (me included) thinks about strings. Sigh.
- dralley 4y agoUCS-32 is at least directly indexable, even though it's ludicrously space-inefficient.
- natanbc 4y agoOnly in codepoints, but it still has the problem GP mentions of ` + e = è being two codepoints (so two elements in UCS-32), but being logically one character https://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning-to-unicode-code-points/ https://manishearth.github.io/blog/2017/01/14/stop-ascribing...
- qalmakka 4y agoThis, it's pointless to have char32_t if you still need to pull several megabytes of ICU to normalize the string first in order to remove characters spanning over multiple codepoints. UTF32 is arguably dangerous because of this, it's yet another attempt to replicate ASCII but with Unicode. The only sane encoding out there is UTF-8, and that's it. If you have to always assume your string is not really splittable without a library, you won't do dangerous stuff such as assuming `wcslen(L"menù") == 4`.
- snvzz 4y ago>The fact we are still using UTF-16 still irks me to this day. Windows, unfortunately.