4 ms·

I've read so many stories over the years of Unicode edge cases being broken in major applications and frameworks. I don't know the current state of all of these

I've read so many stories over the years of Unicode edge cases being broken in major applications and frameworks. I don't know the current state of all of these but Chrome had all kinds of Unicode bugs for years, standard Windows text boxes didn't correctly support combining characters, many consoles and edit boxes don't support mixed right to left and left to right text properly, many widely used languages seem to lack good standard library support for correctly manipulating strings with combining characters, although its hard to find clear explanations of what the right thing to do even is.

eridius 8y ago
Most of what you describe sounds like applications that don't bother to use unicode-aware string manipulation routines, period, rather than applications that use buggy unicode handling.
- mattnewport 8y ago
  A lot of the problems I'm describing revolve around text input / editing / display rather than straight string manipulation. Let's say I want to type 'My name is Matt.' in Farsi which is right to left. Transliterated that is 'esme (name) man (my) Matt ast (is)' and you might think I'd type that by typing the words in that order. اسم من Matt است The above is what I get if I type the words in that order while switching between Persian / English keyboards but it's not really what I'd expect as a user. In terms of right to left word order the above says 'ast Matt esme man' which is not the order I typed and is not correct. Now I'm not an expert in Farsi, Unicode or multi-lingual text entry and I don't know if the problem here is user error, browser implementation, Windows, or what. I know there's something about directional override characters in Unicode and a complex algorithm for dealing with text with mixed directions. I discovered while trying to write this post that it's been exploited in file names as a malware vector. I just know that this stuff is more complicated than I have time to deal with as a programmer on most projects I've worked on, so if Unicode is supported I don't even know if it's working correctly or how to test it in many cases. Try to select, copy and paste the above text into a text editor and then move the cursor around with the cursor keys in it. Is the behavior what you'd expect? Is it correct? I don't know enough to be able to tell honestly.

eridius 8y ago

Most of what you describe sounds like applications that don't bother to use unicode-aware string manipulation routines, period, rather than applications that use buggy unicode handling.

mattnewport 8y ago
A lot of the problems I'm describing revolve around text input / editing / display rather than straight string manipulation. Let's say I want to type 'My name is Matt.' in Farsi which is right to left. Transliterated that is 'esme (name) man (my) Matt ast (is)' and you might think I'd type that by typing the words in that order. اسم من Matt است The above is what I get if I type the words in that order while switching between Persian / English keyboards but it's not really what I'd expect as a user. In terms of right to left word order the above says 'ast Matt esme man' which is not the order I typed and is not correct. Now I'm not an expert in Farsi, Unicode or multi-lingual text entry and I don't know if the problem here is user error, browser implementation, Windows, or what. I know there's something about directional override characters in Unicode and a complex algorithm for dealing with text with mixed directions. I discovered while trying to write this post that it's been exploited in file names as a malware vector. I just know that this stuff is more complicated than I have time to deal with as a programmer on most projects I've worked on, so if Unicode is supported I don't even know if it's working correctly or how to test it in many cases. Try to select, copy and paste the above text into a text editor and then move the cursor around with the cursor keys in it. Is the behavior what you'd expect? Is it correct? I don't know enough to be able to tell honestly.