5 ms·

UTF-16 is way better for many Asian languages. You know, languages billions of people use.

BoppreH 7y ago
Space-wise, yes, but the size of our user-entered strings is rarely a concern. On the other hand, UTF-16 is often mixed with UCS-2, which is not really Unicode, and lulls developers into a false sense of security by almost never needing two code units for a given code point. It's a partial fix that makes bugs harder to catch. There's also the BOM issue, and not being ASCII compatible...
- camgunz 7y ago
  Space is always a concern, and encodings should largely be transparent to your application unless you're writing something like a rendering engine. Consider things like databases (relational/document/key-value/etc.), JSON payloads, compression algorithms, battery life, etc. etc. There is a problem with lots of developers assuming UTF-16 is fixed-width (just look at this thread), but that's a mistake for any UTF encoding, and it's almost always a mistake in any encoding--you shouldn't be indexing into strings. People say, "just use UTF-8, you never have to worry about it and it's ASCII compatible", but you shouldn't be using the ASCII compatibility (arguably this is an anti-feature) and you shouldn't have to worry about the details like BOM because you're using your language's built-in string support or a library right?
jhanschoo 7y ago
What's your reasoning? Where there is significant markup, UTF-16 encoding is rarely more space-efficient than UTF-8. When compression is involved, there is rarely a difference.
- thaumasiotes 7y ago
  UTF-16 may not be great, but UTF-8 is an obviously bad choice for Asian languages. UTF-8 is a variable-width encoding on the following system: ASCII: 1 byte European characters: 2 bytes Asian characters: 3 bytes There is no reason you'd want this in Asia. China uses GB2312, which flips it around: ASCII: 1 byte Asian characters: 2 bytes European characters: 3 bytes
  - jhanschoo 7y ago
    This doesn't address my points. That where there is markup, the additional space required by the ASCII markup on UTF16 may very well offset the space savings of Asian text in UTF8. And also, that with compression, these size arguments become negligible for any sufficiently long document written in one language.
    - thaumasiotes 7y ago
      I conceded that with "UTF-16 may not be great". I'm responding to the idea that UTF-8 is such a good solution to the problem that everyone should just use UTF-8. It isn't.
      - hsivonen 7y ago
        For writing globally appropriate software, UTF-8 is the best option that exists in practice. Note that “Asian” is quite a gamut in terms of UTF-8 size: Chinese is at the very compact end of the spectrum and the Burmese script at the other end: https://hsivonen.fi/string-length/#counts https://hsivonen.fi/string-length/#counts
        camgunz 7y ago
        Wow what a resource, I love this. I'm a little skeptical about that table though, because as you point out, there's not a great way of figuring "meaning per character". For some more (real flawed) comparison, the English version of The Tale of Genji is ~60k words and 224 pages, and the Chinese version is ~75k words and 300 pages. [1] [2] So yeah, I'm skeptical about the claim that some languages have more meaning per character, and as a result you'll end up storing less text overall. I think it'd be cool to look at more data about this though, like (for example) stats from Treasure Data [3]. But regardless, it's pretty indisputable that UTF-16 is a lot better space-wise for languages that tend to be multibyte in UTF-8. Mostly what I'm trying (Quixotically) to say is "UTF-8 the world" ignores a lot of the world. [1]: https://www.readinglength.com/book/isbn-4805314648 https://www.readinglength.com/book/isbn-4805314648 [2]: https://www.readinglength.com/book/isbn-7544717275 https://www.readinglength.com/book/isbn-7544717275 [3]: https://www.treasuredata.com/ https://www.treasuredata.com/
- camgunz 7y ago
  There's a lot of text out there that isn't markup, or where markup is a sparse minority. Consider anything binary, ebooks, PDFs, various document formats, almost anything in a database, almost all messages sent via messaging apps, almost all emails, any JSON, anything in a protobuf, etc. etc. The vast majority of "text" in the world isn't in HTML/XML. > When compression is involved, there is rarely a difference. Rare is the situation where smaller input to a compression algorithm leads to larger output, and UTF-16 is smaller in lots of languages. It might rarely make a difference you, but that's not the same as rarely making a difference.
lopsidedBrain 7y ago
I think I'll disagree, and I'm someone with a native Asian language. A character doesn't mean anything in Asian languages, and any attempt to use a fixed-length encoding is pointless. The concepts you want instead are either a code point or a glyph. The concept of a code point is useful as a part (and not the whole) of Unicode-validating, encoding, and decoding. A glyph is useful mostly in rendering engines (i.e. webkit-internals and UI rendering frameworks). Okay, maybe they are also kind of useful for sorting and collating. But fixed-width character encodings are almost never useful, and invite programmer to make assumptions about how strings can be sliced. Most server-side applications should never have to know what these concepts even are. Or any library that is not user-facing. They get bytes from the UI layer, and they can keep them as opaque bytes. For user-facing apps, you can ask your renderer library for a pixel-width or similar for a string, and let them handle how to parse it. Very little code ever needs to know about unicode. Any kind of input-sanitization is vastly simplified in utf8, and that makes it worth it for me. For me the really troubling trends are conventions like Rust Utf8Error, where they can cause what I'd consider a UI-related exception in code that had no business even interpreting what those bytes are. Unfortunately, every API uses strings, so they are kind of hard to avoid. It introduces what I'd consider a software layering problem. Maybe others here with more experience with internationalization can chime in and tell me I'm wrong.
- Freak_NL 7y ago
  Also, UTF-16 is not fixed width in modern usage. As soon as someone uses an emoji, boom!, surrogate pairs. So unless UTF-32 is used at a glorious four bytes per character, you won't get fixed width. > For me the really troubling trends are conventions like Rust Utf8Error,[…] Interesting. Isn't that only returned when the input bytes contain a non-UTF-8 byte sequence? How is Rust's approach different from other languages?
  - atoav 7y ago
    > Interesting. Isn't that only returned when the input bytes contain a non-UTF-8 byte sequence? As a Rust user: this is what your function returns if the user inputs non-UTF-8 bytes into something that expects UTF-8 bytes and the programmer explicitly choses not to handle that error. I don’t see anything wrong with that. Sometimes you might be interested in receiving, processing, storing valid UTF-8 strings rather than arbitrary byte sequences, that may or may not be able to be translated back into something valid that you can display. I always hated to do encoding related work with a passion before I started using Rust. Rust forced me to do it the right way and actually understand why I am doing it a certain way. When it comes to encoding I feel safer in Rust then e.g. in Python, despite having used it for four times as long.

Space-wise, yes, but the size of our user-entered strings is rarely a concern. On the other hand, UTF-16 is often mixed with UCS-2, which is not really Unicode, and lulls developers into a false sense of security by almost never needing two code units for a given code point. It's a partial fix that makes bugs harder to catch. There's also the BOM issue, and not being ASCII compatible...

camgunz 7y ago
Space is always a concern, and encodings should largely be transparent to your application unless you're writing something like a rendering engine. Consider things like databases (relational/document/key-value/etc.), JSON payloads, compression algorithms, battery life, etc. etc. There is a problem with lots of developers assuming UTF-16 is fixed-width (just look at this thread), but that's a mistake for any UTF encoding, and it's almost always a mistake in any encoding--you shouldn't be indexing into strings. People say, "just use UTF-8, you never have to worry about it and it's ASCII compatible", but you shouldn't be using the ASCII compatibility (arguably this is an anti-feature) and you shouldn't have to worry about the details like BOM because you're using your language's built-in string support or a library right?

jhanschoo 7y ago

What's your reasoning? Where there is significant markup, UTF-16 encoding is rarely more space-efficient than UTF-8. When compression is involved, there is rarely a difference.

thaumasiotes 7y ago
UTF-16 may not be great, but UTF-8 is an obviously bad choice for Asian languages. UTF-8 is a variable-width encoding on the following system: ASCII: 1 byte European characters: 2 bytes Asian characters: 3 bytes There is no reason you'd want this in Asia. China uses GB2312, which flips it around: ASCII: 1 byte Asian characters: 2 bytes European characters: 3 bytes
- jhanschoo 7y ago
  This doesn't address my points. That where there is markup, the additional space required by the ASCII markup on UTF16 may very well offset the space savings of Asian text in UTF8. And also, that with compression, these size arguments become negligible for any sufficiently long document written in one language.
  - thaumasiotes 7y ago
    I conceded that with "UTF-16 may not be great". I'm responding to the idea that UTF-8 is such a good solution to the problem that everyone should just use UTF-8. It isn't.
    - hsivonen 7y ago
      For writing globally appropriate software, UTF-8 is the best option that exists in practice. Note that “Asian” is quite a gamut in terms of UTF-8 size: Chinese is at the very compact end of the spectrum and the Burmese script at the other end: https://hsivonen.fi/string-length/#counts https://hsivonen.fi/string-length/#counts
      - camgunz 7y ago
        Wow what a resource, I love this. I'm a little skeptical about that table though, because as you point out, there's not a great way of figuring "meaning per character". For some more (real flawed) comparison, the English version of The Tale of Genji is ~60k words and 224 pages, and the Chinese version is ~75k words and 300 pages. [1] [2] So yeah, I'm skeptical about the claim that some languages have more meaning per character, and as a result you'll end up storing less text overall. I think it'd be cool to look at more data about this though, like (for example) stats from Treasure Data [3]. But regardless, it's pretty indisputable that UTF-16 is a lot better space-wise for languages that tend to be multibyte in UTF-8. Mostly what I'm trying (Quixotically) to say is "UTF-8 the world" ignores a lot of the world. [1]: https://www.readinglength.com/book/isbn-4805314648 https://www.readinglength.com/book/isbn-4805314648 [2]: https://www.readinglength.com/book/isbn-7544717275 https://www.readinglength.com/book/isbn-7544717275 [3]: https://www.treasuredata.com/ https://www.treasuredata.com/
camgunz 7y ago
There's a lot of text out there that isn't markup, or where markup is a sparse minority. Consider anything binary, ebooks, PDFs, various document formats, almost anything in a database, almost all messages sent via messaging apps, almost all emails, any JSON, anything in a protobuf, etc. etc. The vast majority of "text" in the world isn't in HTML/XML. > When compression is involved, there is rarely a difference. Rare is the situation where smaller input to a compression algorithm leads to larger output, and UTF-16 is smaller in lots of languages. It might rarely make a difference you, but that's not the same as rarely making a difference.

lopsidedBrain 7y ago

I think I'll disagree, and I'm someone with a native Asian language. A character doesn't mean anything in Asian languages, and any attempt to use a fixed-length encoding is pointless. The concepts you want instead are either a code point or a glyph. The concept of a code point is useful as a part (and not the whole) of Unicode-validating, encoding, and decoding. A glyph is useful mostly in rendering engines (i.e. webkit-internals and UI rendering frameworks). Okay, maybe they are also kind of useful for sorting and collating. But fixed-width character encodings are almost never useful, and invite programmer to make assumptions about how strings can be sliced. Most server-side applications should never have to know what these concepts even are. Or any library that is not user-facing. They get bytes from the UI layer, and they can keep them as opaque bytes. For user-facing apps, you can ask your renderer library for a pixel-width or similar for a string, and let them handle how to parse it. Very little code ever needs to know about unicode. Any kind of input-sanitization is vastly simplified in utf8, and that makes it worth it for me. For me the really troubling trends are conventions like Rust Utf8Error, where they can cause what I'd consider a UI-related exception in code that had no business even interpreting what those bytes are. Unfortunately, every API uses strings, so they are kind of hard to avoid. It introduces what I'd consider a software layering problem. Maybe others here with more experience with internationalization can chime in and tell me I'm wrong.

Freak_NL 7y ago
Also, UTF-16 is not fixed width in modern usage. As soon as someone uses an emoji, boom!, surrogate pairs. So unless UTF-32 is used at a glorious four bytes per character, you won't get fixed width. > For me the really troubling trends are conventions like Rust Utf8Error,[…] Interesting. Isn't that only returned when the input bytes contain a non-UTF-8 byte sequence? How is Rust's approach different from other languages?
- atoav 7y ago
  > Interesting. Isn't that only returned when the input bytes contain a non-UTF-8 byte sequence? As a Rust user: this is what your function returns if the user inputs non-UTF-8 bytes into something that expects UTF-8 bytes and the programmer explicitly choses not to handle that error. I don’t see anything wrong with that. Sometimes you might be interested in receiving, processing, storing valid UTF-8 strings rather than arbitrary byte sequences, that may or may not be able to be translated back into something valid that you can display. I always hated to do encoding related work with a passion before I started using Rust. Rust forced me to do it the right way and actually understand why I am doing it a certain way. When it comes to encoding I feel safer in Rust then e.g. in Python, despite having used it for four times as long.