5 ms·

How do you do the rendering? Who (process-wise) is responsible for converting bytes to pixels? How do users on social media put in their name? How is it stored?

by kortex 7y ago

How do you do the rendering? Who (process-wise) is responsible for converting bytes to pixels? How do users on social media put in their name? How is it stored? How do users get urls with specific usernames? Now take all that and multiply by the complexity of world languages, many which don't even map to one glyph == one morpheme. The ol' apple message crash bug was due to the property of some Arabic not being monotonic in rendering space vs string length. I think we could have skipped utf8 and just gone to 4byte runes. But even then, that would not have avoided the above bug. Utf16 is a hot mess though, worst of all worlds.

BoppreH 7y ago
The problem is now deeply entrenched, so I don't have perfect answers. But here are my guesses: > Who (process-wise) is responsible for converting bytes to pixels? The operating system, with minor exceptions (word processors, for example). Rendering logic is too complicated to be embedded into every application. And you get better accessibility and consistency. > How do users on social media put in their name? How is it stored? Keyboards, or their preferred input methods, and stored in UTF-8. I'm not saying to get rid of all strings, just don't use it for infrastructure. > How do users get urls with specific usernames? You don't, because that's how you get little Bobby FRACTION-SLASH. Also, if you have a valuable namespace like URLs, people will hack each other to get valuable names, but that's only tangentially related. > Now take all that and multiply by the complexity of world languages, many which don't even map to one glyph == one morpheme. The ol' apple message crash bug was due to the property of some Arabic not being monotonic in rendering space vs string length. That's exactly my point! You get this multiplied complexity when people try to peek into string contents instead of treating them like black boxes. Stop with the dangerous string operations and you now support usernames with zalgo-ed hieroglyphs if that's what users want. > I think we could have skipped utf8 and just gone to 4byte runes. But even then, that would not have avoided the above bug. > Utf16 is a hot mess though, worst of all worlds. Agreed with UTF-16. I like UTF-8, and I honestly think it solved our encoding problems for non-legacy applications. Everyone should be using it, as long as the contents are for human consumption only.
- camgunz 7y ago
  UTF-16 is way better for many Asian languages. You know, languages billions of people use.
  - BoppreH 7y ago
    Space-wise, yes, but the size of our user-entered strings is rarely a concern. On the other hand, UTF-16 is often mixed with UCS-2, which is not really Unicode, and lulls developers into a false sense of security by almost never needing two code units for a given code point. It's a partial fix that makes bugs harder to catch. There's also the BOM issue, and not being ASCII compatible...
    - camgunz 7y ago
      Space is always a concern, and encodings should largely be transparent to your application unless you're writing something like a rendering engine. Consider things like databases (relational/document/key-value/etc.), JSON payloads, compression algorithms, battery life, etc. etc. There is a problem with lots of developers assuming UTF-16 is fixed-width (just look at this thread), but that's a mistake for any UTF encoding, and it's almost always a mistake in any encoding--you shouldn't be indexing into strings. People say, "just use UTF-8, you never have to worry about it and it's ASCII compatible", but you shouldn't be using the ASCII compatibility (arguably this is an anti-feature) and you shouldn't have to worry about the details like BOM because you're using your language's built-in string support or a library right?
  - jhanschoo 7y ago
    What's your reasoning? Where there is significant markup, UTF-16 encoding is rarely more space-efficient than UTF-8. When compression is involved, there is rarely a difference.
    - thaumasiotes 7y ago
      UTF-16 may not be great, but UTF-8 is an obviously bad choice for Asian languages. UTF-8 is a variable-width encoding on the following system: ASCII: 1 byte European characters: 2 bytes Asian characters: 3 bytes There is no reason you'd want this in Asia. China uses GB2312, which flips it around: ASCII: 1 byte Asian characters: 2 bytes European characters: 3 bytes
swiley 7y ago
That harfbuzz guy, he’s the only one allowed to do it.

jhanschoo 7y ago
This doesn't address my points. That where there is markup, the additional space required by the ASCII markup on UTF16 may very well offset the space savings of Asian text in UTF8. And also, that with compression, these size arguments become negligible for any sufficiently long document written in one language.
- thaumasiotes 7y ago
  I conceded that with "UTF-16 may not be great". I'm responding to the idea that UTF-8 is such a good solution to the problem that everyone should just use UTF-8. It isn't.
  - hsivonen 7y ago
    For writing globally appropriate software, UTF-8 is the best option that exists in practice. Note that “Asian” is quite a gamut in terms of UTF-8 size: Chinese is at the very compact end of the spectrum and the Burmese script at the other end: https://hsivonen.fi/string-length/#counts https://hsivonen.fi/string-length/#counts
    - camgunz 7y ago
      Wow what a resource, I love this. I'm a little skeptical about that table though, because as you point out, there's not a great way of figuring "meaning per character". For some more (real flawed) comparison, the English version of The Tale of Genji is ~60k words and 224 pages, and the Chinese version is ~75k words and 300 pages. [1] [2] So yeah, I'm skeptical about the claim that some languages have more meaning per character, and as a result you'll end up storing less text overall. I think it'd be cool to look at more data about this though, like (for example) stats from Treasure Data [3]. But regardless, it's pretty indisputable that UTF-16 is a lot better space-wise for languages that tend to be multibyte in UTF-8. Mostly what I'm trying (Quixotically) to say is "UTF-8 the world" ignores a lot of the world. [1]: https://www.readinglength.com/book/isbn-4805314648 https://www.readinglength.com/book/isbn-4805314648 [2]: https://www.readinglength.com/book/isbn-7544717275 https://www.readinglength.com/book/isbn-7544717275 [3]: https://www.treasuredata.com/ https://www.treasuredata.com/
      - thaumasiotes 7y ago
        > the Chinese version is ~75k words and 300 pages. [1] [2] Note that link [2] says "guess [of # of words] based on page count". 75k words, 300 pages is one statistic, not two separate statistics. (Estimating words based on page count is likely to be highly reliable, but still.) > I'm skeptical about the claim that some languages have more meaning per character, and as a result you'll end up storing less text overall. Your skepticism is unwarranted. From a pure information-theoretic perspective, the claim that some languages have more meaning per character is a slam dunk, and less than a second of examination proves it conclusively. For example, an entire English novel is unlikely to use more than 256 unique characters. A Chinese novel couldn't use anywhere near that few without sounding incredibly artificial. Here are some single-character words in modern Mandarin: 高 - high/tall 低 - low 大 - big 小 - small 最 - most (superlative marker) 更 - more (comparative marker) 到 - arrive 走 - leave (go away) 玩 - play (e.g. a game) 看 - look 听 - listen 贵 - expensive 爱 - love (verb) 恨 - hate (verb) 龍 - dragon For something more representative, here are some song lyrics -- I'll enclose every word of multiple characters. Unenclosed stretches of text are words of one character each: 都怪我不[小心]地 [偷看了]你的[眼睛] [迷失了][自己] [不知][为什么][欢喜] [期待]听你的[声音] 是[那么][甜蜜] 我[拿着][电话]在你家门外想对你[说出]我[内心]中对你的[期待] [忽然][感觉][浑身][热血][澎湃] [今天]我[想要][[鼓起][勇气]][大声][表白] How many distinct one-character words do you think English could practically support? How many twos? In this verse-and-a-half, there's one word that's always[1] three characters and two that have reached three characters by picking up a verb suffix, for a total of three words that are longer than two characters. (鼓起勇气 is kind of a special case, in that it's a well-known fixed expression, but its meaning is transparent as an ordinary combination of the two words 鼓起 and 勇气, which also see use outside the expression.) [1] Actually, 什么 ["what"] has the vernacular contraction 啥, and this also applies to 为什么 ["why"], so you could argue that the word is sometimes just two characters.
        camgunz 7y ago
        Ugh, I was trying to get a good comparison of translations, but too bad I guess. Yeah I mean, there's no question ideographic languages have more meaning per character than alphabetic languages. But other comparisons and considerations aren't as obvious: - how do ideographic languages compare with each other? - do people using ideographic languages write more? - are ideographic languages as effective at compression as general (or special) compression algorithms? Moving up the conceptual ladder from "average bytes per codepoint" to "average size of encoded tweet" (for example) is a big leap is all I'm saying.
    - thaumasiotes 7y ago
      > For writing globally appropriate software, UTF-8 is the best option that exists in practice. OK, I lied. China doesn't use GB2312 anymore. They use its update, GB18030, which carries arbitrary Unicode data just as UTF-8 does... except that, like GB2312, it puts the Asian characters in two bytes and the European characters in three. It's certainly not obvious to me that UTF-8 is better for globally appropriate software. It looks worse. Software for Europeans, sure.
      - hsivonen 7y ago
        GB18030 has no 3-byte sequences. Only 1, 2, and 4. There are more important qualities than optimizing byte length. Processing GB18030 as Unicode scalar values involves lookup tables. A single byte error cascades potentially further than in UTF-8. If optimizing Chinese byte length is really important for you, UTF-16 is easier to work with than GB18030.
SAI_Peregrinus 7y ago
Yet the higher information density of Chinese & Japanese means the total file size tends to be about the same. Korean gets a bit short-changed, and some of the Indian subcontinent languages have low-density (like Latin) but high size (in with the Asian characters) and lose out.

lopsidedBrain 7y ago

I think I'll disagree, and I'm someone with a native Asian language. A character doesn't mean anything in Asian languages, and any attempt to use a fixed-length encoding is pointless. The concepts you want instead are either a code point or a glyph. The concept of a code point is useful as a part (and not the whole) of Unicode-validating, encoding, and decoding. A glyph is useful mostly in rendering engines (i.e. webkit-internals and UI rendering frameworks). Okay, maybe they are also kind of useful for sorting and collating. But fixed-width character encodings are almost never useful, and invite programmer to make assumptions about how strings can be sliced. Most server-side applications should never have to know what these concepts even are. Or any library that is not user-facing. They get bytes from the UI layer, and they can keep them as opaque bytes. For user-facing apps, you can ask your renderer library for a pixel-width or similar for a string, and let them handle how to parse it. Very little code ever needs to know about unicode. Any kind of input-sanitization is vastly simplified in utf8, and that makes it worth it for me. For me the really troubling trends are conventions like Rust Utf8Error, where they can cause what I'd consider a UI-related exception in code that had no business even interpreting what those bytes are. Unfortunately, every API uses strings, so they are kind of hard to avoid. It introduces what I'd consider a software layering problem. Maybe others here with more experience with internationalization can chime in and tell me I'm wrong.

Freak_NL 7y ago
Also, UTF-16 is not fixed width in modern usage. As soon as someone uses an emoji, boom!, surrogate pairs. So unless UTF-32 is used at a glorious four bytes per character, you won't get fixed width. > For me the really troubling trends are conventions like Rust Utf8Error,[…] Interesting. Isn't that only returned when the input bytes contain a non-UTF-8 byte sequence? How is Rust's approach different from other languages?
- atoav 7y ago
  > Interesting. Isn't that only returned when the input bytes contain a non-UTF-8 byte sequence? As a Rust user: this is what your function returns if the user inputs non-UTF-8 bytes into something that expects UTF-8 bytes and the programmer explicitly choses not to handle that error. I don’t see anything wrong with that. Sometimes you might be interested in receiving, processing, storing valid UTF-8 strings rather than arbitrary byte sequences, that may or may not be able to be translated back into something valid that you can display. I always hated to do encoding related work with a passion before I started using Rust. Rust forced me to do it the right way and actually understand why I am doing it a certain way. When it comes to encoding I feel safer in Rust then e.g. in Python, despite having used it for four times as long.
- camgunz 7y ago
  Using any UTF encoding as fixed-width is almost always a mistake. There's no concept of "characters" in Unicode or the UTF encodings, they use codepoints, and those are only very rarely useful--you cannot treat them as "characters". This is a common misconception with UTF encodings.
camgunz 7y ago
I think we actually agree. > A character doesn't mean anything in Asian languages, and any attempt to use a fixed-length encoding is pointless. Totally agree, indexing into a string is bad practice, no matter the encoding (because you probably can't ever guarantee what the encoding will be, or how it was converted, etc.) This is true for UTF-8 and UTF-16, and really any encoding because again, you can't be sure what you're dealing with. Code points, "characters", glyphs, etc. are all concepts that work at different parts of the stack (well, characters doesn't), which again is true for all encodings. The advantage UTF-16 has for languages that tend to be multibyte is it's representation takes up less space. Other than that, it has all the same disadvantages any other encoding has. > Any kind of input-sanitization is vastly simplified in utf8, and that makes it worth it for me. For me the really troubling trends are conventions like Rust Utf8Error, where they can cause what I'd consider a UI-related exception in code that had no business even interpreting what those bytes are. Unfortunately, every API uses strings, so they are kind of hard to avoid. It introduces what I'd consider a software layering problem. This is the problem right here. The "UTF-8" the world initiative ignores that UTF-16 is a lot more practical for most people, and as a result you get platforms expecting UTF-8 that really have no business doing so.