7 ms·
> UTF-16 is still lurking here and there As someone currently stuck in the windows world, this hurts. Every single Windows API is still stuck with using UTF-16
by roblabla 6y ago
> UTF-16 is still lurking here and there
As someone currently stuck in the windows world, this hurts. Every single Windows API is still stuck with using UTF-16/UCS2 as the string encoding.
Also fun fact, on the Nintendo Switch, various subsystems use different kind of encoding. The filsystem submodule uses Shift-JIS, most of the other modules use UTF-8, but some others yet use UTF-16 (like the virtual keyboard, IIRC). A brilliant mess.
- ChrisSD 6y agoPersonally I don't find UTF-16 to be too bad. It's a simple encoding and very easy to convert to/from UTF-8. So your program can be written in UTF-8 and your WinAPI wrappers can convert as/when needed.
- jeltz 6y agoThe bad thing with UTF-16 is that so much software assumes that one code point always is 16 bits.
- cygx 6y agoMost of the hard stuff is there no matter the encoding (normalization, user-perceived characters spanning multiple code units, paths vs strings, ...).
- jrochkind1 6y agoWhich is not UTF-16 at all, UTF-16 standard clearly says this is not so. So why do they do that? It's actually a leftover of the earlier UCS-2 standard, before it was realized we'd need more codepoints than that, and that it was a mistake to limit to 16-bit space for codepoints in any encoding. Software written for UCS-2 can mostly work compatibly with UTF-16, but there are some problems, encoding the 'higher' codepoints is only one of several. Another is how right-to-left scripts are handled. http://www.differencebetween.net/technology/software-technology/difference-between-ucs-2-and-utf-16/ http://www.differencebetween.net/technology/software-technol... https://unicode.org/faq/utf_bom.html#utf16-11 https://unicode.org/faq/utf_bom.html#utf16-11
- flohofwoe 6y agoWasn't UTF-16 explicitly created as a "backward compatibility hack" for UCS-2 when it became clear that 16 bits per code point isn't enough? They should have ditched 16-bit encodings back then instead of combining the disadvantages of UTF-8 (variable length-encoding) and UTF-32 (not endian-agnostic).
- jrochkind1 6y agoPerhaps unicode wouldn't be nearly as successfully adopted as it is, if they had left UCS-2 adopters hanging instead of providing them a "backward compatibility hack" path. The UCS-2 adopters after all had been faithfully trying to implement the standard at that time. Among other things, showing implementers that if they choose to adopt, you aren't going to leave them hanging out to dry when you realize you made a mistake in the standard, will give other people more confidence to adopt. But also, just generally I think a lesson of unicode's success -- as illustrated by UTF-8 in particular -- is, you have to give people a feasible path from where they are to adoption, this is a legitimate part of the design goals of a standard.
- jhasse 6y agoWindows finally added support for UTF-8 2 years ago: https://docs.microsoft.com/en-us/windows/uwp/design/globalizing/use-utf8-code-page https://docs.microsoft.com/en-us/windows/uwp/design/globaliz...
- ChrisSD 6y agoTechnically it's had support since (IIRC) Windows 7. What this does is call the translation functions for you instead of having to do it yourself.
- huhtenberg 6y agoConversion functions - MultiByteToWideChar & co. - were in since Windows 2000 and the UTF8 codepage was supported as early as XP if not in W2K as well.
- ChrisSD 6y agoAh thanks! It's funny because I can't recall ever using code page 65001 before 7. Maybe there was a reason for that or maybe I simply didn't know it existed until then. Or maybe I thought it simpler to just use UTF-16. I can't remember.
- gpvos 6y agoIt existed in W2K and maybe even earlier, but there were bugs in the console regarding codepage 65001, so you couldn't use it as the default. This was not fixed yet in XP, maybe in 7 though.
- vbezhenar 6y agoIt is not default configuration and it's marked as "experimental" in UI. I would never enable it for my PC, that's just absurd.
- iudqnolq 6y agoYeah, the "might" here is doing a lot of work > As Windows operates natively in UTF-16 (WCHAR), you might need to convert UTF-8 data to UTF-16 (or vice versa) to interoperate with Windows APIs.
- coliveira 6y agoJava is still using UTF-16, it is the internal format used since its creation. I don't know exactly how much this is a problem or not, but it shows that UTF-16 is still an important thing.
- rcoveson 6y agoI think its a huge problem for Java. Try doing proper string collation (standard library or ICU4J), or regular expression matching, in a context where your strings are all UTF-8 and your output should also be UTF-8. Operations that shouldn't require allocation do, because you have to transcode to UTF-16. Not to mention that in some cases, that transcoding is the most expensive part of the operation. All the core Java APIs are built around String or CharSequence (more the latter in releases post-Java 8). CharSequence is a terrible interface for supporting UTF-8 or any encoding besides latin1 or UTF-16. If Java's interfaces had been designed around Unicode codepoint iteration rather than char random access, then the coupling to UTF-16 wouldn't have been so tight. But as things stand, you aren't doing anything interesting to text in Java without either (1) re-implementing everything from scratch, from integer parsing to regexp, or (2) paying the transcode cost on everything your program consumes and emits.
- cryptonector 6y agoIt's a huge problem. UTF-16 is a big big pain. JavaScript (ECMAScript) too has this problem.