3 ms·

Not to be that guy, but utf-32 is fixed length code point, not fixed length character. Characters can consist of multiple code points. I agree that utf-8 is p

Not to be that guy, but utf-32 is fixed length code point, not fixed length character. Characters can consist of multiple code points. I agree that utf-8 is preferable to utf-16 though in most cases where the root script is English/Latin.

usrusr 3y ago
Makes me wonder how bad utf-8 in off-latin environments really is: what's the fraction of strings that is "language content", and how much is colons, quotes, decimal numbers and the like? What fraction of coding cultures embraces home locale script for identifiers, and how many continue on the trajectory set in motion by pre-unicode programming languages? My guess would be that utf-8 is still quite bad, but I wouldn't be entirely surprised if actual numbers turned out considerably less bad than one might expect. And how much of the "utf-8 tax" remains if the actual format of most data at rest and in transit isn't json but gzipped json?
- vlovich123 3y ago
  Experiments I recall from a while back from browsers and JDK indicated the vast majority of memory is consumed by latin1 strings (even for apps in Asia). Of course, those experiments I recall are something like 20 years old and likely biased (websites were smaller, content was still latin1 dominated). It would be interesting to know the results from Chinese apps and websites today. For websites, the markup used the most space and that stuff remains latin1. For apps I don’t know. But also, text size in memory is typically a joke because the expensive bit is all the multimedia that accompanies text these days. Remember, 100mib is 25million characters even in utf32. That’s 1 hour or so of high quality audio or a few seconds of video. In terms of compression, the smaller initial file will typically compress better so not sure what you mean by “utf-8” tax. UTF8 is not slower to process than utf32 as far as I’m aware except for one small corner case of random codepoint access (but then in UTF text processing, you really shouldn’t be doing that I think but I’m not a utf expert ).
3PS 3y ago
> Characters can consist of multiple code points. I enjoy being "that guy", and so, to be unfathomably pedantic, I will point out that the Unicode standard actually uses the word "character" to refer to _encoded characters_, i.e. the mapping from a code point to an abstract character [0]. So 1 code point = 1 character for Unicode. Of course, in real life, character is a silly made-up word that means whatever you think it means and using it to refer to an extended grapheme cluster or glyph is probably closer to how people really think of them anyway. [0]: See section 3.4 of https://www.unicode.org/versions/Unicode15.0.0/UnicodeStandard-15.0.pdf https://www.unicode.org/versions/Unicode15.0.0/UnicodeStanda...
- torstenvl 3y ago
  Your position is obviously wrong both with regard to the Unicode standard and with regard to the meaning of the word "character." Unicode does not define "character" because that term is far too imprecise for a deterministic standard. And the Unicode standard is extremely clear that an abstract character may be represented by multiple code points. "A single abstract character may also be represented by a sequence of code points—for example, latin capital letter g with acute may be represented by the sequence <U+0047 latin capital letter g, U+0301 combining acute accent>, rather than being mapped to a single code point." > Of course, in real life, character is a silly made-up word that means whatever you think it means and using it to refer to an extended grapheme cluster or glyph is probably closer to how people really think of them anyway. Human language and writing are the defining inventions of the known universe. Unicode just happens to be one way of representing them. Referring to them as "silly and made up" in comparison to Unicode is nothing short of extreme mental illness.
  - 3PS 3y ago
    Unicode Standard section 3.4: > Unless specified otherwise for clarity, in the text of the Unicode Standard the term character alone designates an encoded character. Reading is free. I even gave you the section number. > A single abstract character may also be represented by a sequence of code points An "abstract character" is not the same thing as a character. The whole point of my comment was that the general word "character" is a general class of definitions but means nothing by itself. Much like the word "number" - you will notice that mathematicians define many kinds of numbers, but never the word itself. > Human language and writing are the defining inventions of the known universe. Unicode just happens to be one way of representing them. Referring to them as "silly and made up" in comparison to Unicode is nothing short of extreme mental illness. All words are made up. I'm sorry you had to find out this way.

usrusr 3y ago

Makes me wonder how bad utf-8 in off-latin environments really is: what's the fraction of strings that is "language content", and how much is colons, quotes, decimal numbers and the like? What fraction of coding cultures embraces home locale script for identifiers, and how many continue on the trajectory set in motion by pre-unicode programming languages? My guess would be that utf-8 is still quite bad, but I wouldn't be entirely surprised if actual numbers turned out considerably less bad than one might expect. And how much of the "utf-8 tax" remains if the actual format of most data at rest and in transit isn't json but gzipped json?

vlovich123 3y ago
Experiments I recall from a while back from browsers and JDK indicated the vast majority of memory is consumed by latin1 strings (even for apps in Asia). Of course, those experiments I recall are something like 20 years old and likely biased (websites were smaller, content was still latin1 dominated). It would be interesting to know the results from Chinese apps and websites today. For websites, the markup used the most space and that stuff remains latin1. For apps I don’t know. But also, text size in memory is typically a joke because the expensive bit is all the multimedia that accompanies text these days. Remember, 100mib is 25million characters even in utf32. That’s 1 hour or so of high quality audio or a few seconds of video. In terms of compression, the smaller initial file will typically compress better so not sure what you mean by “utf-8” tax. UTF8 is not slower to process than utf32 as far as I’m aware except for one small corner case of random codepoint access (but then in UTF text processing, you really shouldn’t be doing that I think but I’m not a utf expert ).

3PS 3y ago

> Characters can consist of multiple code points. I enjoy being "that guy", and so, to be unfathomably pedantic, I will point out that the Unicode standard actually uses the word "character" to refer to _encoded characters_, i.e. the mapping from a code point to an abstract character [0]. So 1 code point = 1 character for Unicode. Of course, in real life, character is a silly made-up word that means whatever you think it means and using it to refer to an extended grapheme cluster or glyph is probably closer to how people really think of them anyway. [0]: See section 3.4 of https://www.unicode.org/versions/Unicode15.0.0/UnicodeStandard-15.0.pdf https://www.unicode.org/versions/Unicode15.0.0/UnicodeStanda...

torstenvl 3y ago
Your position is obviously wrong both with regard to the Unicode standard and with regard to the meaning of the word "character." Unicode does not define "character" because that term is far too imprecise for a deterministic standard. And the Unicode standard is extremely clear that an abstract character may be represented by multiple code points. "A single abstract character may also be represented by a sequence of code points—for example, latin capital letter g with acute may be represented by the sequence <U+0047 latin capital letter g, U+0301 combining acute accent>, rather than being mapped to a single code point." > Of course, in real life, character is a silly made-up word that means whatever you think it means and using it to refer to an extended grapheme cluster or glyph is probably closer to how people really think of them anyway. Human language and writing are the defining inventions of the known universe. Unicode just happens to be one way of representing them. Referring to them as "silly and made up" in comparison to Unicode is nothing short of extreme mental illness.
- 3PS 3y ago
  Unicode Standard section 3.4: > Unless specified otherwise for clarity, in the text of the Unicode Standard the term character alone designates an encoded character. Reading is free. I even gave you the section number. > A single abstract character may also be represented by a sequence of code points An "abstract character" is not the same thing as a character. The whole point of my comment was that the general word "character" is a general class of definitions but means nothing by itself. Much like the word "number" - you will notice that mathematicians define many kinds of numbers, but never the word itself. > Human language and writing are the defining inventions of the known universe. Unicode just happens to be one way of representing them. Referring to them as "silly and made up" in comparison to Unicode is nothing short of extreme mental illness. All words are made up. I'm sorry you had to find out this way.