6 ms·

The problem is now deeply entrenched, so I don't have perfect answers. But here are my guesses: > Who (process-wise) is responsible for converting bytes to pix

The problem is now deeply entrenched, so I don't have perfect answers. But here are my guesses: > Who (process-wise) is responsible for converting bytes to pixels? The operating system, with minor exceptions (word processors, for example). Rendering logic is too complicated to be embedded into every application. And you get better accessibility and consistency. > How do users on social media put in their name? How is it stored? Keyboards, or their preferred input methods, and stored in UTF-8. I'm not saying to get rid of all strings, just don't use it for infrastructure. > How do users get urls with specific usernames? You don't, because that's how you get little Bobby FRACTION-SLASH. Also, if you have a valuable namespace like URLs, people will hack each other to get valuable names, but that's only tangentially related. > Now take all that and multiply by the complexity of world languages, many which don't even map to one glyph == one morpheme. The ol' apple message crash bug was due to the property of some Arabic not being monotonic in rendering space vs string length. That's exactly my point! You get this multiplied complexity when people try to peek into string contents instead of treating them like black boxes. Stop with the dangerous string operations and you now support usernames with zalgo-ed hieroglyphs if that's what users want. > I think we could have skipped utf8 and just gone to 4byte runes. But even then, that would not have avoided the above bug. > Utf16 is a hot mess though, worst of all worlds. Agreed with UTF-16. I like UTF-8, and I honestly think it solved our encoding problems for non-legacy applications. Everyone should be using it, as long as the contents are for human consumption only.

camgunz 7y ago
UTF-16 is way better for many Asian languages. You know, languages billions of people use.
- BoppreH 7y ago
  Space-wise, yes, but the size of our user-entered strings is rarely a concern. On the other hand, UTF-16 is often mixed with UCS-2, which is not really Unicode, and lulls developers into a false sense of security by almost never needing two code units for a given code point. It's a partial fix that makes bugs harder to catch. There's also the BOM issue, and not being ASCII compatible...
  - camgunz 7y ago
    Space is always a concern, and encodings should largely be transparent to your application unless you're writing something like a rendering engine. Consider things like databases (relational/document/key-value/etc.), JSON payloads, compression algorithms, battery life, etc. etc. There is a problem with lots of developers assuming UTF-16 is fixed-width (just look at this thread), but that's a mistake for any UTF encoding, and it's almost always a mistake in any encoding--you shouldn't be indexing into strings. People say, "just use UTF-8, you never have to worry about it and it's ASCII compatible", but you shouldn't be using the ASCII compatibility (arguably this is an anti-feature) and you shouldn't have to worry about the details like BOM because you're using your language's built-in string support or a library right?
- jhanschoo 7y ago
  What's your reasoning? Where there is significant markup, UTF-16 encoding is rarely more space-efficient than UTF-8. When compression is involved, there is rarely a difference.
  - thaumasiotes 7y ago
    UTF-16 may not be great, but UTF-8 is an obviously bad choice for Asian languages. UTF-8 is a variable-width encoding on the following system: ASCII: 1 byte European characters: 2 bytes Asian characters: 3 bytes There is no reason you'd want this in Asia. China uses GB2312, which flips it around: ASCII: 1 byte Asian characters: 2 bytes European characters: 3 bytes
    - jhanschoo 7y ago
      This doesn't address my points. That where there is markup, the additional space required by the ASCII markup on UTF16 may very well offset the space savings of Asian text in UTF8. And also, that with compression, these size arguments become negligible for any sufficiently long document written in one language.
      - thaumasiotes 7y ago
        I conceded that with "UTF-16 may not be great". I'm responding to the idea that UTF-8 is such a good solution to the problem that everyone should just use UTF-8. It isn't.
        hsivonen 7y ago
        For writing globally appropriate software, UTF-8 is the best option that exists in practice. Note that “Asian” is quite a gamut in terms of UTF-8 size: Chinese is at the very compact end of the spectrum and the Burmese script at the other end: https://hsivonen.fi/string-length/#counts https://hsivonen.fi/string-length/#counts
- lopsidedBrain 7y ago
  I think I'll disagree, and I'm someone with a native Asian language. A character doesn't mean anything in Asian languages, and any attempt to use a fixed-length encoding is pointless. The concepts you want instead are either a code point or a glyph. The concept of a code point is useful as a part (and not the whole) of Unicode-validating, encoding, and decoding. A glyph is useful mostly in rendering engines (i.e. webkit-internals and UI rendering frameworks). Okay, maybe they are also kind of useful for sorting and collating. But fixed-width character encodings are almost never useful, and invite programmer to make assumptions about how strings can be sliced. Most server-side applications should never have to know what these concepts even are. Or any library that is not user-facing. They get bytes from the UI layer, and they can keep them as opaque bytes. For user-facing apps, you can ask your renderer library for a pixel-width or similar for a string, and let them handle how to parse it. Very little code ever needs to know about unicode. Any kind of input-sanitization is vastly simplified in utf8, and that makes it worth it for me. For me the really troubling trends are conventions like Rust Utf8Error, where they can cause what I'd consider a UI-related exception in code that had no business even interpreting what those bytes are. Unfortunately, every API uses strings, so they are kind of hard to avoid. It introduces what I'd consider a software layering problem. Maybe others here with more experience with internationalization can chime in and tell me I'm wrong.
  - Freak_NL 7y ago
    Also, UTF-16 is not fixed width in modern usage. As soon as someone uses an emoji, boom!, surrogate pairs. So unless UTF-32 is used at a glorious four bytes per character, you won't get fixed width. > For me the really troubling trends are conventions like Rust Utf8Error,[…] Interesting. Isn't that only returned when the input bytes contain a non-UTF-8 byte sequence? How is Rust's approach different from other languages?
    - atoav 7y ago
      > Interesting. Isn't that only returned when the input bytes contain a non-UTF-8 byte sequence? As a Rust user: this is what your function returns if the user inputs non-UTF-8 bytes into something that expects UTF-8 bytes and the programmer explicitly choses not to handle that error. I don’t see anything wrong with that. Sometimes you might be interested in receiving, processing, storing valid UTF-8 strings rather than arbitrary byte sequences, that may or may not be able to be translated back into something valid that you can display. I always hated to do encoding related work with a passion before I started using Rust. Rust forced me to do it the right way and actually understand why I am doing it a certain way. When it comes to encoding I feel safer in Rust then e.g. in Python, despite having used it for four times as long.

camgunz 7y ago

UTF-16 is way better for many Asian languages. You know, languages billions of people use.

BoppreH 7y ago
Space-wise, yes, but the size of our user-entered strings is rarely a concern. On the other hand, UTF-16 is often mixed with UCS-2, which is not really Unicode, and lulls developers into a false sense of security by almost never needing two code units for a given code point. It's a partial fix that makes bugs harder to catch. There's also the BOM issue, and not being ASCII compatible...
- camgunz 7y ago
  Space is always a concern, and encodings should largely be transparent to your application unless you're writing something like a rendering engine. Consider things like databases (relational/document/key-value/etc.), JSON payloads, compression algorithms, battery life, etc. etc. There is a problem with lots of developers assuming UTF-16 is fixed-width (just look at this thread), but that's a mistake for any UTF encoding, and it's almost always a mistake in any encoding--you shouldn't be indexing into strings. People say, "just use UTF-8, you never have to worry about it and it's ASCII compatible", but you shouldn't be using the ASCII compatibility (arguably this is an anti-feature) and you shouldn't have to worry about the details like BOM because you're using your language's built-in string support or a library right?
jhanschoo 7y ago
What's your reasoning? Where there is significant markup, UTF-16 encoding is rarely more space-efficient than UTF-8. When compression is involved, there is rarely a difference.
- thaumasiotes 7y ago
  UTF-16 may not be great, but UTF-8 is an obviously bad choice for Asian languages. UTF-8 is a variable-width encoding on the following system: ASCII: 1 byte European characters: 2 bytes Asian characters: 3 bytes There is no reason you'd want this in Asia. China uses GB2312, which flips it around: ASCII: 1 byte Asian characters: 2 bytes European characters: 3 bytes
  - jhanschoo 7y ago
    This doesn't address my points. That where there is markup, the additional space required by the ASCII markup on UTF16 may very well offset the space savings of Asian text in UTF8. And also, that with compression, these size arguments become negligible for any sufficiently long document written in one language.
    - thaumasiotes 7y ago
      I conceded that with "UTF-16 may not be great". I'm responding to the idea that UTF-8 is such a good solution to the problem that everyone should just use UTF-8. It isn't.
      - hsivonen 7y ago
        For writing globally appropriate software, UTF-8 is the best option that exists in practice. Note that “Asian” is quite a gamut in terms of UTF-8 size: Chinese is at the very compact end of the spectrum and the Burmese script at the other end: https://hsivonen.fi/string-length/#counts https://hsivonen.fi/string-length/#counts
lopsidedBrain 7y ago
I think I'll disagree, and I'm someone with a native Asian language. A character doesn't mean anything in Asian languages, and any attempt to use a fixed-length encoding is pointless. The concepts you want instead are either a code point or a glyph. The concept of a code point is useful as a part (and not the whole) of Unicode-validating, encoding, and decoding. A glyph is useful mostly in rendering engines (i.e. webkit-internals and UI rendering frameworks). Okay, maybe they are also kind of useful for sorting and collating. But fixed-width character encodings are almost never useful, and invite programmer to make assumptions about how strings can be sliced. Most server-side applications should never have to know what these concepts even are. Or any library that is not user-facing. They get bytes from the UI layer, and they can keep them as opaque bytes. For user-facing apps, you can ask your renderer library for a pixel-width or similar for a string, and let them handle how to parse it. Very little code ever needs to know about unicode. Any kind of input-sanitization is vastly simplified in utf8, and that makes it worth it for me. For me the really troubling trends are conventions like Rust Utf8Error, where they can cause what I'd consider a UI-related exception in code that had no business even interpreting what those bytes are. Unfortunately, every API uses strings, so they are kind of hard to avoid. It introduces what I'd consider a software layering problem. Maybe others here with more experience with internationalization can chime in and tell me I'm wrong.
- Freak_NL 7y ago
  Also, UTF-16 is not fixed width in modern usage. As soon as someone uses an emoji, boom!, surrogate pairs. So unless UTF-32 is used at a glorious four bytes per character, you won't get fixed width. > For me the really troubling trends are conventions like Rust Utf8Error,[…] Interesting. Isn't that only returned when the input bytes contain a non-UTF-8 byte sequence? How is Rust's approach different from other languages?
  - atoav 7y ago
    > Interesting. Isn't that only returned when the input bytes contain a non-UTF-8 byte sequence? As a Rust user: this is what your function returns if the user inputs non-UTF-8 bytes into something that expects UTF-8 bytes and the programmer explicitly choses not to handle that error. I don’t see anything wrong with that. Sometimes you might be interested in receiving, processing, storing valid UTF-8 strings rather than arbitrary byte sequences, that may or may not be able to be translated back into something valid that you can display. I always hated to do encoding related work with a passion before I started using Rust. Rust forced me to do it the right way and actually understand why I am doing it a certain way. When it comes to encoding I feel safer in Rust then e.g. in Python, despite having used it for four times as long.