5 ms·

I think it's also more space efficient when dealing with foreign languages primarily. I agree that UTF-8 is the better default, but that's one of the most compe

I think it's also more space efficient when dealing with foreign languages primarily. I agree that UTF-8 is the better default, but that's one of the most compelling arguments in favor of it (aside from it being the default on Windows).

jchw 8y ago
Even this argument isn't great, though, because a lot of the time it doesn't matter. Like, for HTML, the majority of the time the UTF-8 version will be shorter regardless because of all of the single-byte characters used in the markup.
danieldk 8y ago
I think it's also more space efficient when dealing with foreign languages primarily. For plain text, yes. UTF-16 covers the Basic Multilingual Plane in 2 bytes per character, UTF-8 covers the BMP in at most 3 bytes per character. However, since most (web) content is dominated by markup in ASCII, which is 1 byte per character in UTF-8 and 2 bytes per character in UTF-16, UTF-8 typically wins even with Asian scripts. Taking some random zh Wikipedia page in UTF-8: curl "https://zh.wikipedia.org/wiki/威蘭運河" | wc -c 78543 curl "https://zh.wikipedia.org/wiki/威蘭運河" | wc -m 71003 So, that's 1.11 bytes per character on average, compared to at least 2 bytes per character if it was encoded in UTF-16. (Of course, adding compression is going to diminish the differences anyway. A very primitive compression algorithm in such cases would be recoding your UTF-16 to UTF-8 :p.)
- Symbiote 8y ago
  curl -Ss "https://zh.wikipedia.org/wiki/威蘭運河" | iconv -t utf-16 | wc -c 142008 There are only 3799 non-ASCII characters on that page: curl -Ss "https://zh.wikipedia.org/wiki/威蘭運河" | tr -d '\1-\176' | wc -m 3799 Compared with 67204 ASCII ones (tr -cd).
  - danieldk 8y ago
    The 142008 is to be expected. The page has 71003 code points, 2 bytes per code point (I guess that zh.wikipedia.org is simplified Chinese and normally only uses characters from the BMP) gives 142006 bytes. I'd guess that the two remaining bytes are the byte order mark.
Const-me 8y ago
> I agree that UTF-8 is the better default Better default for what? I agree UTF-8 is better default for HTML sent over network. For different context, the better default can be different. > side from it being the default on Windows Also default in Java (including Android userland), C#, iOS & OSX, python 3 (unless compiled with UTF32 support but the default is UTF16), Symbian, OpenOffice, QT, most XML parsers (e.g. Xerces), and many other languages, libraries and frameworks. Even JavaScript uses UTF16 strings.
- outworlder 8y ago
  Better default for everything. If you have particular needs, then you change from the default. This is why 'default values' exist.
  - Const-me 8y ago
    > Better default for everything. Software industry is huge and the requirements are dramatically different in different areas. What’s good default for a bash script is not necessarily a good default for OSX desktop app, or Linux web server. When you’re working on native rich GUI, UTF16 is often better because majority of higher-level languages, libraries and frameworks use it, even on Linux (QT, Android). If you’ll pick something that’s not native, you’ll waste substantial time, both development and CPU, converting these strings back and forth for absolutely no value. Sometimes you have to because you’re integrating different things together, e.g. UTF16 Android’s Java with UTF8 Android’s C++: http://banachowski.com/deprogramming/2012/02/working-around-jni-utf-8-strings/ http://banachowski.com/deprogramming/2012/02/working-around-... But quite often you are good just picking whatever’s the simplest option and not doing any of that.
- Kwpolska 8y ago
  > python 3 (unless compiled with UTF32 support but the default is UTF16) Not true since Python 3.3. It now uses latin1/ucs-2/ucs-4 based on string contents (smallest encoding that can represent all codepoints).
- int_19h 8y ago
  > python 3 (unless compiled with UTF32 support but the default is UTF16) The default is UTF-32 on Linux, and UTF-16 on Windows and macOS - simply so as to match the platform convention.

jchw 8y ago

Even this argument isn't great, though, because a lot of the time it doesn't matter. Like, for HTML, the majority of the time the UTF-8 version will be shorter regardless because of all of the single-byte characters used in the markup.

danieldk 8y ago

I think it's also more space efficient when dealing with foreign languages primarily. For plain text, yes. UTF-16 covers the Basic Multilingual Plane in 2 bytes per character, UTF-8 covers the BMP in at most 3 bytes per character. However, since most (web) content is dominated by markup in ASCII, which is 1 byte per character in UTF-8 and 2 bytes per character in UTF-16, UTF-8 typically wins even with Asian scripts. Taking some random zh Wikipedia page in UTF-8: curl "https://zh.wikipedia.org/wiki/威蘭運河" | wc -c 78543 curl "https://zh.wikipedia.org/wiki/威蘭運河" | wc -m 71003 So, that's 1.11 bytes per character on average, compared to at least 2 bytes per character if it was encoded in UTF-16. (Of course, adding compression is going to diminish the differences anyway. A very primitive compression algorithm in such cases would be recoding your UTF-16 to UTF-8 :p.)

Symbiote 8y ago
curl -Ss "https://zh.wikipedia.org/wiki/威蘭運河" | iconv -t utf-16 | wc -c 142008 There are only 3799 non-ASCII characters on that page: curl -Ss "https://zh.wikipedia.org/wiki/威蘭運河" | tr -d '\1-\176' | wc -m 3799 Compared with 67204 ASCII ones (tr -cd).
- danieldk 8y ago
  The 142008 is to be expected. The page has 71003 code points, 2 bytes per code point (I guess that zh.wikipedia.org is simplified Chinese and normally only uses characters from the BMP) gives 142006 bytes. I'd guess that the two remaining bytes are the byte order mark.

Const-me 8y ago

> I agree that UTF-8 is the better default Better default for what? I agree UTF-8 is better default for HTML sent over network. For different context, the better default can be different. > side from it being the default on Windows Also default in Java (including Android userland), C#, iOS & OSX, python 3 (unless compiled with UTF32 support but the default is UTF16), Symbian, OpenOffice, QT, most XML parsers (e.g. Xerces), and many other languages, libraries and frameworks. Even JavaScript uses UTF16 strings.

outworlder 8y ago
Better default for everything. If you have particular needs, then you change from the default. This is why 'default values' exist.
- Const-me 8y ago
  > Better default for everything. Software industry is huge and the requirements are dramatically different in different areas. What’s good default for a bash script is not necessarily a good default for OSX desktop app, or Linux web server. When you’re working on native rich GUI, UTF16 is often better because majority of higher-level languages, libraries and frameworks use it, even on Linux (QT, Android). If you’ll pick something that’s not native, you’ll waste substantial time, both development and CPU, converting these strings back and forth for absolutely no value. Sometimes you have to because you’re integrating different things together, e.g. UTF16 Android’s Java with UTF8 Android’s C++: http://banachowski.com/deprogramming/2012/02/working-around-jni-utf-8-strings/ http://banachowski.com/deprogramming/2012/02/working-around-... But quite often you are good just picking whatever’s the simplest option and not doing any of that.
Kwpolska 8y ago
> python 3 (unless compiled with UTF32 support but the default is UTF16) Not true since Python 3.3. It now uses latin1/ucs-2/ucs-4 based on string contents (smallest encoding that can represent all codepoints).
int_19h 8y ago
> python 3 (unless compiled with UTF32 support but the default is UTF16) The default is UTF-32 on Linux, and UTF-16 on Windows and macOS - simply so as to match the platform convention.