4 ms·

It's very easy to not get corrupted strings when byte indexing into UTF-8. In a while loop, if the index is not at the end of the string and the top two bits o

by Avernar 10y ago

It's very easy to not get corrupted strings when byte indexing into UTF-8. In a while loop, if the index is not at the end of the string and the top two bits of the character at the index are both one, advance the index by one. Or throw an invalid index exception if the top to bits are one if that makes more sense for the language you're using.

imron 10y ago
In a while loop, if the index is not at the end of the string and the top two bits of the character at the index are both one, advance the index by one. So what you're saying, is that it's very easy to get corrupted strings by anyone who doesn't have an understanding of utf-8 at the bit-level - which in my experience seems to be the majority of programmers.
- Avernar 10y ago
  Not at all. The language/library impementor should handle the details. My example was the argument checking of the slice function. And indexing by code points doesn't solve the problem either. The majority of programmers don't know what a grapheme is or how to collate or sort unicode strings.
  - imron 10y ago
    Except what happens in the real world is that people who are used to indexing and slicing ASCII strings however they please don't think "I should use a library for this", instead they just keep indexing and slicing as per usual and don't think anything of it until their Chinese customers start complaining of random program crashes, or missing text - which the developer then has difficulty trying to reproduce because hey, it works for them. My only gripe with your argument is that I don't think it's easy to avoid corrupted text in modern text processing - which is precisely why there are libraries for it because it's actually really easy to get it wrong - even if you know what you're doing.
    - Avernar 10y ago
      Which is why we have languages like Go where we can put those types of developers. Incidentally Go use UTF-8. Higher level languages like Go, Python, etc were designed so newbie and/or ignorant programers could do less damage. When I was working on a project before Unicode we would switch our dev PCs to the other languages we supported. What a pain that was. Only issues we had was when a translated string was much longer than the screen space allocated to it. I belive Swedish was the main culprit. No problems with simplified and traditional Chinese as those were more compact. I have no sympathy for dev shops that can't get internationalization right. As with everything else in the corporate dev world management doesn't seem to want to hire/retain the more experienced programmers. I think you have a gripe with my argument because you may be missing my point. If a high level language chooses to let a programmer index into a UTF-8 string at the byte level (for performance and other reasons) it's very easy for it to prevent the the programmer from slicing in the middle of a code unit. The reason being is that the language function to slice a unicode string would either throw an exception or just advance to the next valid index. There wouldn't be a way for the programmer to slice a unicode string in the middle of a code unit.
      - imron 10y ago
        I think you have a gripe with my argument because you may be missing my point I get your point, it just doesn't apply to many real world situations I've seen where you don't have the luxury of just using a higher level language or a library that takes care of all these things, or keeping programmers who don't understand what they are doing away from that sort of thing. The most egregious example that I've personally seen was a developer working on a legacy Cobol banking program that needed Chinese support retro-fitted to it. The app was originally only developed with ASCII in mind and so sliced through strings willy-nilly, which naturally caused problems with Chinese text. The developer working on the "fix" before me, was calling out to ICU through the C API of the version of Cobol that we used and was still messing things up - he'd actually modified ICU in some custom way to prevent the bug from crashing the program, but was still causing corrupted text. I basically undid all his changes, and wrapped all COBOL string splicing to call a function that always split a string at a valid position - truncating invalid bytes at the start/end as necessary. Much simpler and resulted in the removal of an unnecessary dependency on ICU. This bug had been outstanding for several months when I first joined that company, and it was the first one I was assigned to work on - and luckily for them they'd accidentally hired someone who had done lots of multilingual programming before. it's very easy for it to prevent the the programmer from slicing in the middle of a code unit. Okay, but even you made a mistake in your first example of what to do, and that's the sort of code that someone who knows what they are doing could write, and will seem to work in the conditions under which it was tested (working on my machine, ship it!), but that will cause seemingly random problems once it hits users.
kazinator 10y ago
It's impossible to get corrupted strings if you use wide characters. This is better than hard not to.
- Avernar 10y ago
  Not impossible. If you slice in the middle of a grapheme then you get a corrupted string as well. You'll get an alternate glyph instead of a square box but it's still corrupted.
  - paulddraper 10y ago
    It's not a corrupted string. You may have mangled it in a way that doesn't preserve all semantics, but no one will crash with an encoding issue. A "string" means "a sequence of characters". Wide characters (or the equivalent interface) preserve this property. Graphemes operate at a higher level than characters. You could construct a grapheme-strings, I suppose, but that has tons of edge cases, and if you don't like character-strings, I doubt you will like grapheme-strings.
    - kazinator 10y ago
      Or, if by "string" we mean "sequence of code points" (rather than graphemes) then it doesn't get corrupted by any chopping or rearrangement which only permutes the code points. If we chop UTF-8, we can end up with bad characters, or possibly invalid overlong forms.
      - Avernar 10y ago
        I consider changing a glyph to some other glyph(s) as corruption. Take an emoji flag character as an example. Split it between the code points and you end up with two boxed letters. If you chop in the middel of a code unit then you end up with U-FFFDs. In both cases the visual representation has been altered. As I wrote elsewhere it is easy for the slice routines of a language to check to see if the programmer tried to slice in the middle of a code unit and either return an error or just advance to the start of a code unit.
Avernar 10y ago
Oops, made a mistake. The bit pattern is 10 in the upper two bits if you're not at the first byte of a code point and not 11.
- imron 10y ago
  Good thing it's very easy not to make mistakes :->

by Avernar 10y ago

imron 10y ago

In a while loop, if the index is not at the end of the string and the top two bits of the character at the index are both one, advance the index by one. So what you're saying, is that it's very easy to get corrupted strings by anyone who doesn't have an understanding of utf-8 at the bit-level - which in my experience seems to be the majority of programmers.

Avernar 10y ago
Not at all. The language/library impementor should handle the details. My example was the argument checking of the slice function. And indexing by code points doesn't solve the problem either. The majority of programmers don't know what a grapheme is or how to collate or sort unicode strings.
- imron 10y ago
  Except what happens in the real world is that people who are used to indexing and slicing ASCII strings however they please don't think "I should use a library for this", instead they just keep indexing and slicing as per usual and don't think anything of it until their Chinese customers start complaining of random program crashes, or missing text - which the developer then has difficulty trying to reproduce because hey, it works for them. My only gripe with your argument is that I don't think it's easy to avoid corrupted text in modern text processing - which is precisely why there are libraries for it because it's actually really easy to get it wrong - even if you know what you're doing.
  - Avernar 10y ago
    Which is why we have languages like Go where we can put those types of developers. Incidentally Go use UTF-8. Higher level languages like Go, Python, etc were designed so newbie and/or ignorant programers could do less damage. When I was working on a project before Unicode we would switch our dev PCs to the other languages we supported. What a pain that was. Only issues we had was when a translated string was much longer than the screen space allocated to it. I belive Swedish was the main culprit. No problems with simplified and traditional Chinese as those were more compact. I have no sympathy for dev shops that can't get internationalization right. As with everything else in the corporate dev world management doesn't seem to want to hire/retain the more experienced programmers. I think you have a gripe with my argument because you may be missing my point. If a high level language chooses to let a programmer index into a UTF-8 string at the byte level (for performance and other reasons) it's very easy for it to prevent the the programmer from slicing in the middle of a code unit. The reason being is that the language function to slice a unicode string would either throw an exception or just advance to the next valid index. There wouldn't be a way for the programmer to slice a unicode string in the middle of a code unit.
    - imron 10y ago
      I think you have a gripe with my argument because you may be missing my point I get your point, it just doesn't apply to many real world situations I've seen where you don't have the luxury of just using a higher level language or a library that takes care of all these things, or keeping programmers who don't understand what they are doing away from that sort of thing. The most egregious example that I've personally seen was a developer working on a legacy Cobol banking program that needed Chinese support retro-fitted to it. The app was originally only developed with ASCII in mind and so sliced through strings willy-nilly, which naturally caused problems with Chinese text. The developer working on the "fix" before me, was calling out to ICU through the C API of the version of Cobol that we used and was still messing things up - he'd actually modified ICU in some custom way to prevent the bug from crashing the program, but was still causing corrupted text. I basically undid all his changes, and wrapped all COBOL string splicing to call a function that always split a string at a valid position - truncating invalid bytes at the start/end as necessary. Much simpler and resulted in the removal of an unnecessary dependency on ICU. This bug had been outstanding for several months when I first joined that company, and it was the first one I was assigned to work on - and luckily for them they'd accidentally hired someone who had done lots of multilingual programming before. it's very easy for it to prevent the the programmer from slicing in the middle of a code unit. Okay, but even you made a mistake in your first example of what to do, and that's the sort of code that someone who knows what they are doing could write, and will seem to work in the conditions under which it was tested (working on my machine, ship it!), but that will cause seemingly random problems once it hits users.

kazinator 10y ago

It's impossible to get corrupted strings if you use wide characters. This is better than hard not to.

Avernar 10y ago
Not impossible. If you slice in the middle of a grapheme then you get a corrupted string as well. You'll get an alternate glyph instead of a square box but it's still corrupted.
- paulddraper 10y ago
  It's not a corrupted string. You may have mangled it in a way that doesn't preserve all semantics, but no one will crash with an encoding issue. A "string" means "a sequence of characters". Wide characters (or the equivalent interface) preserve this property. Graphemes operate at a higher level than characters. You could construct a grapheme-strings, I suppose, but that has tons of edge cases, and if you don't like character-strings, I doubt you will like grapheme-strings.
  - kazinator 10y ago
    Or, if by "string" we mean "sequence of code points" (rather than graphemes) then it doesn't get corrupted by any chopping or rearrangement which only permutes the code points. If we chop UTF-8, we can end up with bad characters, or possibly invalid overlong forms.
    - Avernar 10y ago
      I consider changing a glyph to some other glyph(s) as corruption. Take an emoji flag character as an example. Split it between the code points and you end up with two boxed letters. If you chop in the middel of a code unit then you end up with U-FFFDs. In both cases the visual representation has been altered. As I wrote elsewhere it is easy for the slice routines of a language to check to see if the programmer tried to slice in the middle of a code unit and either return an error or just advance to the start of a code unit.

Avernar 10y ago

Oops, made a mistake. The bit pattern is 10 in the upper two bits if you're not at the first byte of a code point and not 11.

imron 10y ago
Good thing it's very easy not to make mistakes :->