3 ms·

Not impossible. If you slice in the middle of a grapheme then you get a corrupted string as well. You'll get an alternate glyph instead of a square box but it

by Avernar 10y ago

Not impossible. If you slice in the middle of a grapheme then you get a corrupted string as well. You'll get an alternate glyph instead of a square box but it's still corrupted.

paulddraper 10y ago
It's not a corrupted string. You may have mangled it in a way that doesn't preserve all semantics, but no one will crash with an encoding issue. A "string" means "a sequence of characters". Wide characters (or the equivalent interface) preserve this property. Graphemes operate at a higher level than characters. You could construct a grapheme-strings, I suppose, but that has tons of edge cases, and if you don't like character-strings, I doubt you will like grapheme-strings.
- kazinator 10y ago
  Or, if by "string" we mean "sequence of code points" (rather than graphemes) then it doesn't get corrupted by any chopping or rearrangement which only permutes the code points. If we chop UTF-8, we can end up with bad characters, or possibly invalid overlong forms.
  - Avernar 10y ago
    I consider changing a glyph to some other glyph(s) as corruption. Take an emoji flag character as an example. Split it between the code points and you end up with two boxed letters. If you chop in the middel of a code unit then you end up with U-FFFDs. In both cases the visual representation has been altered. As I wrote elsewhere it is easy for the slice routines of a language to check to see if the programmer tried to slice in the middle of a code unit and either return an error or just advance to the start of a code unit.
    - kazinator 10y ago
      UTF-8 slicing destroys characters and graphemes. Slicing a code-point-character string destroys only graphemes. Clear win. A code point string has other niceties, like being indexed by simple integers. If end is the index of the last code point of a grapheme, then the next grapheme starts at end + 1. If end is the index of the last UTF-8 encoding of a code point, then the next grapheme does not start at end + 1. We can have it so that it does by making end point to the last byte of the UTF-8 encoding of the code point; but then it doesn't point at the start of the character, recovering which is awkward. The code uglification can be addressed by piling on abstractions: integer-like iteration gizmos that can be incremented and decremented thanks to function or operator overloading. I feel that that level of abstraction has no place in character-level data processing, if anywhere, whose basic operations should be expressible tersely in a few machine instructions. Also, we mustn't lose sight of what the T means in UTF-8: transfer. It's not called UPF-8 (the Unicode processing format in 8 bits). Working with UTF-8 instead of with the objects that UTF-8 denotes is like working with a textual representation of Lisp s-expressions that still contain the parentheses and whitespace delimitation, and quotes around strings and so on, refusing to parse them to obtain the object which they represent. People who do this should immediately turn in their CS degrees. All those other issues you refer to are addressed by more parsing. If you want the glyphs, the correct thing is to parse the code-point string and make a list or vector of glyph representations. With that representation you can still break the text "carpet" into "car" "pet" which destroys semantics; that is dealt with by parsing into words. Chopping lists of words destroys phrases; so parse phrases, and transform at the phrase level. And so on.
      - Avernar 10y ago
        > Clear win. I'd call it a slight improvement. And after the major step back of using 2 to 4 times more memory for strings I'd call it a net loss. > A code point string has other niceties, like being indexed by simple integers. Again, no benefit of this. The only argument I've heard here is to prevent bad slicing and I've shown a way to prevent that. > Also, we mustn't lose sight of what the T means in UTF-8: transfer. It's not called UPF-8 (the Unicode processing format in 8 bits). By this argument we can't use UTF-16 or UTF-32 for internal processing of strings either. Back to code pages then.
- Avernar 10y ago
  Why should it crash? The proper procedure when validating a UTF-8 string is to replace errors with U+FFFD. The term character has many meanings. Graphemes are characters and that's what most users expect, something that's displayed as a single graphical unit.
  - paulddraper 10y ago
    I use "character" in the same way that the Unicode Consortium uses the word. Though "code point" would be more precise.
    - Avernar 10y ago
      That's what they were hoping for. Didn't tuen out thst way. From icu-project.org: "As with glyphs, there is no one-to-one relationship between characters and code points. What an end-user thinks of as a single character (grapheme) may in fact be represented by multiple code points; conversely, a single code point may correspond to multiple characters."

by Avernar 10y ago

Not impossible. If you slice in the middle of a grapheme then you get a corrupted string as well. You'll get an alternate glyph instead of a square box but it's still corrupted.

paulddraper 10y ago

It's not a corrupted string. You may have mangled it in a way that doesn't preserve all semantics, but no one will crash with an encoding issue. A "string" means "a sequence of characters". Wide characters (or the equivalent interface) preserve this property. Graphemes operate at a higher level than characters. You could construct a grapheme-strings, I suppose, but that has tons of edge cases, and if you don't like character-strings, I doubt you will like grapheme-strings.

kazinator 10y ago
Or, if by "string" we mean "sequence of code points" (rather than graphemes) then it doesn't get corrupted by any chopping or rearrangement which only permutes the code points. If we chop UTF-8, we can end up with bad characters, or possibly invalid overlong forms.
- Avernar 10y ago
  I consider changing a glyph to some other glyph(s) as corruption. Take an emoji flag character as an example. Split it between the code points and you end up with two boxed letters. If you chop in the middel of a code unit then you end up with U-FFFDs. In both cases the visual representation has been altered. As I wrote elsewhere it is easy for the slice routines of a language to check to see if the programmer tried to slice in the middle of a code unit and either return an error or just advance to the start of a code unit.
  - kazinator 10y ago
    UTF-8 slicing destroys characters and graphemes. Slicing a code-point-character string destroys only graphemes. Clear win. A code point string has other niceties, like being indexed by simple integers. If end is the index of the last code point of a grapheme, then the next grapheme starts at end + 1. If end is the index of the last UTF-8 encoding of a code point, then the next grapheme does not start at end + 1. We can have it so that it does by making end point to the last byte of the UTF-8 encoding of the code point; but then it doesn't point at the start of the character, recovering which is awkward. The code uglification can be addressed by piling on abstractions: integer-like iteration gizmos that can be incremented and decremented thanks to function or operator overloading. I feel that that level of abstraction has no place in character-level data processing, if anywhere, whose basic operations should be expressible tersely in a few machine instructions. Also, we mustn't lose sight of what the T means in UTF-8: transfer. It's not called UPF-8 (the Unicode processing format in 8 bits). Working with UTF-8 instead of with the objects that UTF-8 denotes is like working with a textual representation of Lisp s-expressions that still contain the parentheses and whitespace delimitation, and quotes around strings and so on, refusing to parse them to obtain the object which they represent. People who do this should immediately turn in their CS degrees. All those other issues you refer to are addressed by more parsing. If you want the glyphs, the correct thing is to parse the code-point string and make a list or vector of glyph representations. With that representation you can still break the text "carpet" into "car" "pet" which destroys semantics; that is dealt with by parsing into words. Chopping lists of words destroys phrases; so parse phrases, and transform at the phrase level. And so on.
    - Avernar 10y ago
      > Clear win. I'd call it a slight improvement. And after the major step back of using 2 to 4 times more memory for strings I'd call it a net loss. > A code point string has other niceties, like being indexed by simple integers. Again, no benefit of this. The only argument I've heard here is to prevent bad slicing and I've shown a way to prevent that. > Also, we mustn't lose sight of what the T means in UTF-8: transfer. It's not called UPF-8 (the Unicode processing format in 8 bits). By this argument we can't use UTF-16 or UTF-32 for internal processing of strings either. Back to code pages then.
Avernar 10y ago
Why should it crash? The proper procedure when validating a UTF-8 string is to replace errors with U+FFFD. The term character has many meanings. Graphemes are characters and that's what most users expect, something that's displayed as a single graphical unit.
- paulddraper 10y ago
  I use "character" in the same way that the Unicode Consortium uses the word. Though "code point" would be more precise.
  - Avernar 10y ago
    That's what they were hoping for. Didn't tuen out thst way. From icu-project.org: "As with glyphs, there is no one-to-one relationship between characters and code points. What an end-user thinks of as a single character (grapheme) may in fact be represented by multiple code points; conversely, a single code point may correspond to multiple characters."