5 ms·

I consider changing a glyph to some other glyph(s) as corruption. Take an emoji flag character as an example. Split it between the code points and you end up

• kazinator 10y ago
  UTF-8 slicing destroys characters and graphemes. Slicing a code-point-character string destroys only graphemes. Clear win. A code point string has other niceties, like being indexed by simple integers. If end is the index of the last code point of a grapheme, then the next grapheme starts at end + 1. If end is the index of the last UTF-8 encoding of a code point, then the next grapheme does not start at end + 1. We can have it so that it does by making end point to the last byte of the UTF-8 encoding of the code point; but then it doesn't point at the start of the character, recovering which is awkward. The code uglification can be addressed by piling on abstractions: integer-like iteration gizmos that can be incremented and decremented thanks to function or operator overloading. I feel that that level of abstraction has no place in character-level data processing, if anywhere, whose basic operations should be expressible tersely in a few machine instructions. Also, we mustn't lose sight of what the T means in UTF-8: transfer. It's not called UPF-8 (the Unicode processing format in 8 bits). Working with UTF-8 instead of with the objects that UTF-8 denotes is like working with a textual representation of Lisp s-expressions that still contain the parentheses and whitespace delimitation, and quotes around strings and so on, refusing to parse them to obtain the object which they represent. People who do this should immediately turn in their CS degrees. All those other issues you refer to are addressed by more parsing. If you want the glyphs, the correct thing is to parse the code-point string and make a list or vector of glyph representations. With that representation you can still break the text "carpet" into "car" "pet" which destroys semantics; that is dealt with by parsing into words. Chopping lists of words destroys phrases; so parse phrases, and transform at the phrase level. And so on.
  • Avernar 10y ago
    > Clear win. I'd call it a slight improvement. And after the major step back of using 2 to 4 times more memory for strings I'd call it a net loss. > A code point string has other niceties, like being indexed by simple integers. Again, no benefit of this. The only argument I've heard here is to prevent bad slicing and I've shown a way to prevent that. > Also, we mustn't lose sight of what the T means in UTF-8: transfer. It's not called UPF-8 (the Unicode processing format in 8 bits). By this argument we can't use UTF-16 or UTF-32 for internal processing of strings either. Back to code pages then.
    • paulddraper 10y ago
      His point was that the transfer format should be conceptually independent of the processing. Obviously it has to be encoded in RAM in some way, but the programmer doesn't need to worry about the memory layout.
      • Avernar 10y ago
        Why? Why should it be conceptually different if it's easy to work with the encoded form? Many unicode languages work with UTF-8 or UTF-16 internally. So working with the "transfer format" is common practice. While it may not be necessary to know who the languages your program in work under the hood, expert programmers do want/need to know. That way they can write better code, or switch to another language or get the language devs to improve their internal handling.
        • paulddraper 10y ago
          Think of JSON. The programmer shouldn't have to know that a newline character is written as \n in a JSON string. The JSON string "a\nb" take 6 characters to write, but it's length should be given as 3. 99% people want to manipulate a JSON model, not the JSON (or BSON) serialization itself. The 1% can still use a byte array and do whatever hacks they like.
          • 10y ago