3 ms·

The gist of unicode is that code like const char const end = str + strlen(str); for (const char it = str; it < end; ++i) { .. } is almost always wrong

The gist of unicode is that code like const char *const end = str + strlen(str); for (const char *it = str; it < end; ++i) { .. } is almost always wrong unless your locale is C and you explicitly say you only work with ASCII. It's wrong with UTF-32 and char32_t too, because as the post suggests Unicode codepoints are not glyphs - you can have glyphs spanning over multiple codepoints, with multiple normalization forms. This is particularly tricky, because characters printed the same can be represented in multiple ways and it's hard to discern them, i.e. è can be a single codepoint or a combination of a '◌́ ' modifier and 'e'. In general the trivial C assumption "strings are just char arrays" most people are accustomed to is dangerous and broken in Unicode. str[x] in Unicode is wrong 99% of the time, thus UTF16 and 32 are nonsense¹ and either you treat strings as bytes you just pass around, or you need a full Unicode library such as ICU to operate on Unicode text safely. ¹: in my experience they are NEVER used mostly, UTF-16 is actually two incompatible encodings and use it as a way to "UTF-wash" the broken UCS-2 they so eagerly adopted in the '90s (including Java, Windows NT, Cocoa, Qt, Unreal, JavaScript, [partially] Python, ...) .

electroly 3y ago
UTF-16 is the internal encoding of ICU to this day. If you're using ICU, you're using UTF-16. The library treats UTF-8 as a conversion target rather than a native representation. If you ever see a new project pick UTF-16 and you don't know why, it's because of ICU; any other choice forces a round trip conversion on every ICU call. If you pick UTF-16 you can just use icu::UnicodeString as your string representation and life is easy.
johnnyanmac 3y ago
So, as someone who has had to briefly dabble in Unicode muck some years ago: how would I properly iterate through Unicode? How do I know how many bytes to iterate before I get a proper glyph instead of part of a glyph, or worse: a glyph and some extra codepoints? Is the concept so esoteric that it's best to just find a blackbox library and not worry about the finer details?
- HelloNurse 3y ago
  If you want a "glyph" you are trolling yourself because selecting and assembling glyphs in fonts is an additional layer of punishing complication on top of Unicode itself. Correct text rendering requires passing as much text as possible (even multiple lines) to a "black box" library that figures out layout, ligatures, glyph replacement etc. If instead you want to find, as is more likely, whole grapheme clusters that can be treated as editing units (e.g. select/copy/delete "the character after the cursor", move the cursor N "characters", count "letters" in a string) you can go forward through your string, codepoint by codepoint, and decide depending on how those codepoints are classified where a "character" ends. Your ideas and requirements about the criteria might or might not match those of your libraries and of the Unicode standard itself.
- jeroenhd 3y ago
  To do it correctly, you decode the unicode. Then you apply whatever algorithms you need. Decide things like "are the diacritics on combined characters extra glyphs or not" and "how many characters is '﷽'" and "ௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌ is technically a single glyph built around several character combinations but it's as long as a sentence". Actually, you don't want glyphs, you're probably interested in grapheme clusters. You can try to be smart and measure the width of the rendered string, but then you find out that Mongolian is written top-to-bottom and that Zalgo allows for freaky combinations that break the line boundaries. The real answer is the answer to the question "how do I deal with timezones in my code": you use a library by people who already put the effort in. Take something like ICU and let that deal with the text. If you need to ask the question, you're probably not a programmer with a minor in linguistics so letting the people who do know do the work makes everyone's lives a lot easier. Just make sure to read the documentation right.
  - ElectricalUnion 3y ago
    grapheme clusters: what your users and business people actually mean when they say "characters". Normal text inputs usually allow placement of the cursor and editing only at grapheme cluster level.
- qalmakka 3y ago
f33d5173 3y ago
If str is utf-8, as it likely is nowadays outside certain OS APIs, then that code is fine. You can't grab an arbitrary character from the str and move it around and have it retain its meaning. But you couldn't necessarily do that before either, as you would be potentially breaking up a word, eg. To do unicode while pretending it's ascii in C, you look for ascii characters you recognize, like punctuation and the like, which you split the string on. You then treat every other substring as a black box of characters that can only be moved around as a unit.

The gist of unicode is that code like const char *const end = str + strlen(str); for (const char *it = str; it < end; ++i) { .. } is almost always wrong

electroly 3y ago

UTF-16 is the internal encoding of ICU to this day. If you're using ICU, you're using UTF-16. The library treats UTF-8 as a conversion target rather than a native representation. If you ever see a new project pick UTF-16 and you don't know why, it's because of ICU; any other choice forces a round trip conversion on every ICU call. If you pick UTF-16 you can just use icu::UnicodeString as your string representation and life is easy.

johnnyanmac 3y ago

So, as someone who has had to briefly dabble in Unicode muck some years ago: how would I properly iterate through Unicode? How do I know how many bytes to iterate before I get a proper glyph instead of part of a glyph, or worse: a glyph and some extra codepoints? Is the concept so esoteric that it's best to just find a blackbox library and not worry about the finer details?

HelloNurse 3y ago
If you want a "glyph" you are trolling yourself because selecting and assembling glyphs in fonts is an additional layer of punishing complication on top of Unicode itself. Correct text rendering requires passing as much text as possible (even multiple lines) to a "black box" library that figures out layout, ligatures, glyph replacement etc. If instead you want to find, as is more likely, whole grapheme clusters that can be treated as editing units (e.g. select/copy/delete "the character after the cursor", move the cursor N "characters", count "letters" in a string) you can go forward through your string, codepoint by codepoint, and decide depending on how those codepoints are classified where a "character" ends. Your ideas and requirements about the criteria might or might not match those of your libraries and of the Unicode standard itself.
jeroenhd 3y ago
To do it correctly, you decode the unicode. Then you apply whatever algorithms you need. Decide things like "are the diacritics on combined characters extra glyphs or not" and "how many characters is '﷽'" and "ௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌ is technically a single glyph built around several character combinations but it's as long as a sentence". Actually, you don't want glyphs, you're probably interested in grapheme clusters. You can try to be smart and measure the width of the rendered string, but then you find out that Mongolian is written top-to-bottom and that Zalgo allows for freaky combinations that break the line boundaries. The real answer is the answer to the question "how do I deal with timezones in my code": you use a library by people who already put the effort in. Take something like ICU and let that deal with the text. If you need to ask the question, you're probably not a programmer with a minor in linguistics so letting the people who do know do the work makes everyone's lives a lot easier. Just make sure to read the documentation right.
- ElectricalUnion 3y ago
  grapheme clusters: what your users and business people actually mean when they say "characters". Normal text inputs usually allow placement of the cursor and editing only at grapheme cluster level.
qalmakka 3y ago

f33d5173 3y ago

If str is utf-8, as it likely is nowadays outside certain OS APIs, then that code is fine. You can't grab an arbitrary character from the str and move it around and have it retain its meaning. But you couldn't necessarily do that before either, as you would be potentially breaking up a word, eg. To do unicode while pretending it's ascii in C, you look for ascii characters you recognize, like punctuation and the like, which you split the string on. You then treat every other substring as a black box of characters that can only be moved around as a unit.