3 ms·
So, as someone who has had to briefly dabble in Unicode muck some years ago: how would I properly iterate through Unicode? How do I know how many bytes to itera
by johnnyanmac 3y ago
So, as someone who has had to briefly dabble in Unicode muck some years ago: how would I properly iterate through Unicode? How do I know how many bytes to iterate before I get a proper glyph instead of part of a glyph, or worse: a glyph and some extra codepoints?
Is the concept so esoteric that it's best to just find a blackbox library and not worry about the finer details?
- HelloNurse 3y agoIf you want a "glyph" you are trolling yourself because selecting and assembling glyphs in fonts is an additional layer of punishing complication on top of Unicode itself. Correct text rendering requires passing as much text as possible (even multiple lines) to a "black box" library that figures out layout, ligatures, glyph replacement etc. If instead you want to find, as is more likely, whole grapheme clusters that can be treated as editing units (e.g. select/copy/delete "the character after the cursor", move the cursor N "characters", count "letters" in a string) you can go forward through your string, codepoint by codepoint, and decide depending on how those codepoints are classified where a "character" ends. Your ideas and requirements about the criteria might or might not match those of your libraries and of the Unicode standard itself.
- jeroenhd 3y agoTo do it correctly, you decode the unicode. Then you apply whatever algorithms you need. Decide things like "are the diacritics on combined characters extra glyphs or not" and "how many characters is '﷽'" and "ௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌௌ is technically a single glyph built around several character combinations but it's as long as a sentence". Actually, you don't want glyphs, you're probably interested in grapheme clusters. You can try to be smart and measure the width of the rendered string, but then you find out that Mongolian is written top-to-bottom and that Zalgo allows for freaky combinations that break the line boundaries. The real answer is the answer to the question "how do I deal with timezones in my code": you use a library by people who already put the effort in. Take something like ICU and let that deal with the text. If you need to ask the question, you're probably not a programmer with a minor in linguistics so letting the people who do know do the work makes everyone's lives a lot easier. Just make sure to read the documentation right.
- ElectricalUnion 3y agographeme clusters: what your users and business people actually mean when they say "characters". Normal text inputs usually allow placement of the cursor and editing only at grapheme cluster level.
- qalmakka 3y ago> how would I properly iterate through Unicode? Either you use a (big, complex) library like ICU or you just don't. People have been thought for too long that strings are arrays of characters - they are not, they are array of bytes, and bytes may mean lots of things. If you need to iterate or randomly access a string you should ask yourself why are you doing it. Any code that attempts to operate on codepoints will inevitably only ever work properly for ASCII. If that's fine for you (for instance, you want to match all text between a pair of <>) then you can probably work on raw bytes and UTF-8. If that's not the case, you need a library to handle the million different cases related to supporting RTL, invisible characters, pictograms, combinations, ... Never use wchar_t, and even things like char16_t and char32_t have very few correct usecases to be honest (like implementing computing sizes, etc). Even counting characters is broken, `wcslen(L"Salò")` may still be 4 or 5 depending on the current normalization form. The correct solution is to take strings as black boxes, array of bytes you only access through APIs (like binary data). It is also high time we get rid of obsolete concepts like case sensivity - it just doesn't work at all outside of regional code pages (try doing `"ciao".toUpperCase().equals("CIAO")` in Java with locale tr_TR.UTF-8 and look what happens).
- electroly 3y agoYou use "icu::BreakIterator::createCharacterInstance" from the ICU library to get a character-walker. In .NET, you have "StringInfo.GetTextElementEnumerator". You must use a Unicode library to walk "grapheme clusters" (what users recognize as individual characters). There's no other way to do it correctly.