11 ms·

>Even though this is 2018, occasionally someone will try to claim in conversation with me that UTF-16 is better than UTF-8. I'm curious, what are their argumen

• beatgammit 8y ago
  I think it's also more space efficient when dealing with foreign languages primarily. I agree that UTF-8 is the better default, but that's one of the most compelling arguments in favor of it (aside from it being the default on Windows).
  • jchw 8y ago
    Even this argument isn't great, though, because a lot of the time it doesn't matter. Like, for HTML, the majority of the time the UTF-8 version will be shorter regardless because of all of the single-byte characters used in the markup.
  • danieldk 8y ago
    I think it's also more space efficient when dealing with foreign languages primarily. For plain text, yes. UTF-16 covers the Basic Multilingual Plane in 2 bytes per character, UTF-8 covers the BMP in at most 3 bytes per character. However, since most (web) content is dominated by markup in ASCII, which is 1 byte per character in UTF-8 and 2 bytes per character in UTF-16, UTF-8 typically wins even with Asian scripts. Taking some random zh Wikipedia page in UTF-8: curl "https://zh.wikipedia.org/wiki/威蘭運河" | wc -c 78543 curl "https://zh.wikipedia.org/wiki/威蘭運河" | wc -m 71003 So, that's 1.11 bytes per character on average, compared to at least 2 bytes per character if it was encoded in UTF-16. (Of course, adding compression is going to diminish the differences anyway. A very primitive compression algorithm in such cases would be recoding your UTF-16 to UTF-8 :p.)
    • Symbiote 8y ago
      curl -Ss "https://zh.wikipedia.org/wiki/威蘭運河" | iconv -t utf-16 | wc -c 142008 There are only 3799 non-ASCII characters on that page: curl -Ss "https://zh.wikipedia.org/wiki/威蘭運河" | tr -d '\1-\176' | wc -m 3799 Compared with 67204 ASCII ones (tr -cd).
      • danieldk 8y ago
        The 142008 is to be expected. The page has 71003 code points, 2 bytes per code point (I guess that zh.wikipedia.org is simplified Chinese and normally only uses characters from the BMP) gives 142006 bytes. I'd guess that the two remaining bytes are the byte order mark.
  • Const-me 8y ago
    > I agree that UTF-8 is the better default Better default for what? I agree UTF-8 is better default for HTML sent over network. For different context, the better default can be different. > side from it being the default on Windows Also default in Java (including Android userland), C#, iOS & OSX, python 3 (unless compiled with UTF32 support but the default is UTF16), Symbian, OpenOffice, QT, most XML parsers (e.g. Xerces), and many other languages, libraries and frameworks. Even JavaScript uses UTF16 strings.
    • outworlder 8y ago
      Better default for everything. If you have particular needs, then you change from the default. This is why 'default values' exist.
      • Const-me 8y ago
        > Better default for everything. Software industry is huge and the requirements are dramatically different in different areas. What’s good default for a bash script is not necessarily a good default for OSX desktop app, or Linux web server. When you’re working on native rich GUI, UTF16 is often better because majority of higher-level languages, libraries and frameworks use it, even on Linux (QT, Android). If you’ll pick something that’s not native, you’ll waste substantial time, both development and CPU, converting these strings back and forth for absolutely no value. Sometimes you have to because you’re integrating different things together, e.g. UTF16 Android’s Java with UTF8 Android’s C++: http://banachowski.com/deprogramming/2012/02/working-around-jni-utf-8-strings/ http://banachowski.com/deprogramming/2012/02/working-around-... But quite often you are good just picking whatever’s the simplest option and not doing any of that.
    • Kwpolska 8y ago
      > python 3 (unless compiled with UTF32 support but the default is UTF16) Not true since Python 3.3. It now uses latin1/ucs-2/ucs-4 based on string contents (smallest encoding that can represent all codepoints).
    • int_19h 8y ago
      > python 3 (unless compiled with UTF32 support but the default is UTF16) The default is UTF-32 on Linux, and UTF-16 on Windows and macOS - simply so as to match the platform convention.
• cryptonector 8y ago
  People who don't understand surrogate pairs and think that UTF-16 == UCS-2 tend to argue that UTF-16 is better because (in their minds) it's a fixed-sized character encoding (it's not). Even UTF-32 is not a fixed-sized character encoding!! It's a fixed-sized codepoint encoding. Characters can be composed of more than one codepoint. Even if you think "hey, I'll use pre-composed codepoints", you'll fail because not every legitimate, canonically decomposed character has a single-codepoint pre-composition. The disadvantages to UTF-16 far outweigh its one advantage (the one you mentioned). (There is one more advantage to UTF-16, and it's that Win32 uses it all over the place. But that's not much of an advantage, and Windows does seem to be making progress towards putting UTF-8 on an equal footing (if not better).)
  • ivanbakel 8y ago
    Does Win32 use UTF-16 or WTF-16?
    • vardump 8y ago
      For path and object names, it's the latter. WTF-16 "standard" is just UTF-16 that allows invalid surrogate pairs. Windows kernel style handling. It's a shame you were voted down for a perfectly legitimate (if misleading) question. This is a very common gotcha most developers ignore until they encounter their first invalid surrogate pair file path... UTF-16 can't represent all possible Windows file names or paths.
      • ivanbakel 8y ago
        Was it misleading? I was just trying to understand if Win32 used WTF-16 everywhere (and some strings mostly happened to be valid UTF-16), or if there was some place where it used UTF instead.
        • vardump 8y ago
          I think object names and paths are WTF-16, that's where NT kernel is involved. Note that Microsoft doesn't call it that way, but it's a useful naming distinction regardless. Applications can of course use whatever they please. Just need to be careful with the possible invalid object or file name surrogate pairs, not to fail on those and to pass them with alteration. Well, misleading in that sense you did get the downvotes... That said I find WTF-16 pretty appropriate name for what Windows does internally on the kernel level.
• kiriakasis 8y ago
  > AFAIK the only advantage of UTF-16 is that it's a little more efficient when encoding text written in certain scripts. and IIRC east Asian languages (which benefit the most) almost only use UTF-8
  • gsnedders 8y ago
    And almost all actual uses are cases where markup (be it HTML, OOXML, ODF) are sufficiently common that the total file size ends up being larger using UTF-16 than UTF-8.
  • jcranmer 8y ago
    I don't know if this is still the case, but as of a few years ago, there was clear evidence that the Japanese would rather have their text corrupted via mojibake than silently use UTF-8. Of course, that's about avoiding Unicode entirely rather than preferring UTF-16 over UTF-8.
    • pmarreck 8y ago
      TIL about "mojibake" (I visited Japan for 2 weeks for the first time about a month ago; had a great time)
    • simias 8y ago
      Do you know why that is? Why don't they like Unicode?
      • rspeer 8y ago
        A couple reasons I'm aware of: - Han unification. Chinese and Japanese characters that have the same etymology but are written differently are sometimes assigned to the same codepoint, and it's up to the font to distinguish. And application designers... tend not to detect the language and change the font accordingly. This leads to Japanese text being rendered in a Chinese font. - Some Han de-unification has happened in Unicode now. Great. Which version should fonts and encodings support? - Nobody actually knows the complete de facto mapping between Shift-JIS and Unicode. Yes, there are standards and ICU and Python modules and stuff; they're incomplete. This leads to data loss surrounding rare characters. (Tell me you've got such a mapping and I'll give you some strings I found in the wild to decode with it.)
• Const-me 8y ago
  > what are their arguments? It’s often faster. When you process non-English text in UTF8, branch prediction fails all the time because all characters have random size from 1 to 3 bytes. When you process non-English text in UTF16, branch prediction predicts all the time: surrogate pairs are extremely rare so the 99.9% of the stream is 2 bytes/character. > it's not compatible with ascii and C-strings like UTF-8 In modern C++, wchar_t is a built-in fundamental type: http://www.cplusplus.com/reference/cwchar/wchar_t/ http://www.cplusplus.com/reference/cwchar/wchar_t/
  • jmgao 8y ago
    > In modern C++, wchar_t is a built-in fundamental type: http://www.cplusplus.com/reference/cwchar/wchar_t/ http://www.cplusplus.com/reference/cwchar/wchar_t/ But with the slight problem that it actually means UTF-32 on everything on non-Windows (there's char16_t (C++11) and char32_t (C++20), but somewhat obnoxiously, they're new types, so you need explicit casts for pointer conversion)
    • int_19h 8y ago
      It's even worse - it means "wide character", which is not necessarily Unicode. E.g. on all BSDs, if your locale is something like zh_TW-Big5, then wchar_t stores Big5 codepoints! You can use it to store UTF-32, of course, but all the wcs* standard functions won't treat it as such. Now, there's __STDC_ISO_10646__, which, if it is defined by an implementation, means that wchar_t is guaranteed to be a Unicode codepoint. So e.g. glibc defines that, and on Linux you can therefore assume wchar_t as UTF-32 regardless of locale. But BSDs don't. And on Windows, nothing that uses 16-bit wchar_t can use that define, since 16 bits aren't wide enough to be conforming.
  • kupiakos 8y ago
    Wow, branch prediction is actually a pretty good point I hadn't thought about.
    • wahern 8y ago
      It's common, though, to decode to an internal 16- or even 32-bit representation. Combined with clever UTF-8 decoding state machines the overhead is minimal, whether you do this all at once or in a streaming, segmented fashion. Rare is the circumstance where you can just load UTF-16 directly into an array and go to town without preprocessing. In most application scenarios you need to translate to a normalization form (i.e. NFC, NFD, etc), which you can simply combine with your UTF-8 decoding. Such tight loops of arithmetic code are blazingly fast, and performance can improve dramatically by batching these steps separately rather than intermixing different phases of text processing in your application. Perl6 decodes to NFG form which provides a 1:1 mapping between characters and scalar values.[1] That makes Unicode string munging almost as easy as ASCII.[2] But it requires always preprocessing and intern'ing your strings. [1] Multi-codepoint graphemes are dynamically assigned a unique internal codepoint. [2] I say almost because some Unicode text rules, like those for word and paragraph breaks, can never be implemented as simple as in ASCII.