5 ms·
Iterating code points is OK, as long as you know that know that iterating code points is not the same as iterating grapheme clusters aka user perceived characte
by nabla9 7y ago
Iterating code points is OK, as long as you know that know that iterating code points is not the same as iterating grapheme clusters aka user perceived characters. You get away with it most of the time, but you should know you are not dealing with full Unicode and have a plan to deal with exceptions. Unicode normalization does not solve it all.
Unfortunately almost all "Absolute minimum you must know about Unicode" articles don't cover the absolute minimum you have to know about Unicode.
Arbitrary well formed UTF-8 combined with advanced string algorithms and data structures where the unit is 'char' requires more than code points.
- mont 7y agoDo you know any "absolute minimum you must know about Unicode" articles that do go into enough depth?
- nabla9 7y agoUnfortunately I don't. I started to learn Unicode, then realized how complicated it is to do right and stopped because I realized that nobody really cares if it works almost all the time. As Joel below demonstrates, you can get away with 29 languages by treating code points as characters and without knowing about grapheme clusters and other stuff. https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim... >When CityDesk publishes the web page, it converts it to UTF-8 encoding, which has been well supported by web browsers for many years. That’s the way all 29 language versions of Joel on Software are encoded and I have not yet heard a single person who has had any trouble viewing them.
- masklinn 7y ago> As Joel below demonstrates, you can get away with 29 languages by treating code points as characters and without knowing about grapheme clusters and other stuff. If you treat text as completely opaque it does work fine. Issues crop up when you want or need to manipulate said text, either to extract information or to modify it.
- jfk13 7y agoNot really relevant. That just demonstrates that displaying those languages works adequately; it doesn't show anything about other processing that your software might care about (e.g. sorting, searching, case conversion, keyboard input, selection and editing, etc.)
- planteen 7y agoOne key to know is that encoding (UTF-8, UTF-16, UTF-32) is a completely separate problem from rendering text. I have had a couple people say to me recently something along the lines of, "We don't need text shaping since UTF-8 takes care of it." That isn't remotely true. An encoding gets you a series of Unicode code points. To render this, these code points must get the bidirectional algorithm applies (bidi) and then these "runs" from the bidi algoritm are then shaped. The text shaper uses OpenType tables within the font to convert these code points into a series of glyph indices with x/y offsets. The renderer then works entirely on glyphs, which might not even map back to a code point in the font. The HarfBuzz manual touches on some of this: https://harfbuzz.github.io/why-do-i-need-a-shaping-engine.html https://harfbuzz.github.io/why-do-i-need-a-shaping-engine.ht...
- lifthrasiir 7y agoI found that even a suggestion to use "grapheme clusters" is misleading. People like to think that there is one kind of grapheme clusters, namely one specified in the UAX #29 [1], but that's just the default and the UAX is pretty much clear about it! Consider an `ij` digraph in Dutch [2] that should count as a single character as a motivating example. "Code points", or rather "Unicode scalar values" are formally defined and not changing; "grapheme clusters" are generally locale-dependent. I think that we should ask what people want to do with "characters" instead: - If you want an array of very short strings, use an array. Do not abuse strings. Recent languages even don't like it. - If you want a string that is not too long when displayed, first and foremost try to resolve that on the actual display (say, with CSS). If that's really impossible, pick a font and actually measure. Make sure that you are using the actual size being printed---the bounding box is not linearly scaled when the font size changes. - If that display is a terminal, you may also consider using the East Asian Width [3]. But keep in mind that the default width of "Ambiguous" characters greatly varies across locales (Asians tend to prefer dual-width ambiguous characters, for example). - If you want a string that is not too long when stored, use the encoded byte size. Oh, do you want to make sure that certain languages are not disadvantaged? Do your research to determine the appropriate limit per language then, but I bet you would be much better with generous limits (cough cough Twitter cough). - (While this is not counting characters, for the completeness,) if you want to navigate a string with a cursor or a keyboard, then default grapheme clusters may actually fit a bill, as that might be the best possible! Someone will obviously complain though. - If you have to, uh, really, really count characters, the chance is that you can assume a particular script and/or language. Reject others and use the local convention. You may want to read about the Unicode Script Property [4]. [1] https://unicode.org/reports/tr29/ https://unicode.org/reports/tr29/ [2] https://en.wikipedia.org/wiki/IJ_(digraph) https://en.wikipedia.org/wiki/IJ_(digraph) [3] https://www.unicode.org/reports/tr11/ https://www.unicode.org/reports/tr11/ [4] https://www.unicode.org/reports/tr24/ https://www.unicode.org/reports/tr24/
- nabla9 7y agoYou are correct. The lesson is that code points are low level representation and the semantics does not transfer across languages in Unicode. If you don't know the context you can get characters that don't render correctly or you may split string into parts in the wrong UTF-8 strings can be treated as ASCII or Latin-1 replacements. If you want to deal with full Unicode with all cases you need locale. Exercise for the reader: Try to write radix tree data and rope data structure that works with every language in all cases with Unicode. http://cldr.unicode.org/ http://cldr.unicode.org/ is your friend.
- eridius 7y agoIterating grapheme clusters is OK, as long as you know that iterating grapheme clusters is not the same as iterating unicode scalars (code points) aka the fundamental unit of textual parsing grammars. This is something that really bugs me about how Swift changed its mind and made the String type a Collection of Characters (i.e. grapheme clusters). Originally they recognized this issue and required you to write `str.characters` to work with the grapheme clusters as a collection (and String itself wasn't a collection at all), but then in Swift 3 (I think) they changed course and said String is a collection after all. And the problem is now people work with Characters without even thinking about it when they really should be working with unicode scalars. In my personal experience, I only ever actually want to work with grapheme clusters when I'm doing something relating to user text editing (for example, if the user hits delete with an empty selection, I want to delete the last grapheme cluster). Most of my string manipulation wants to operate on scalars instead.
- weberc2 7y agoIn my naive opinion, it seems like a good choice then for languages (the ones with first-class utf-8 support, anyway) to operate on scalars and leave graphemes / user-text-editing-use-cases to libraries. (This is meant as an extention to your comment, not in contradiction to it).
- eridius 7y agoI'm glad that Swift has first-class support for grapheme clusters. It's just very irritating that they made it the default way to interact with strings.
- comex 7y agoWhy not just use bytes? Most text parsing operations of the sort I think you’re describing can be done on UTF-8 bytes just as well as on codepoints, faster and without sacrificing correctness.
- eridius 7y agoUp until Swift 5, Swift's String type was backed by UTF-16 (except for all-ASCII native strings, which just stored ASCII). Even with Swift 5, it's sometimes backed by UTF-16 (namely, when it contains an NSString bridged from Obj-C code that contains non-ASCII characters, which can happen even in pure-Swift code due to all of the String APIs that are really just wrappers around Obj-C Foundation APIs) and sometimes backed by UTF-8. In truly performance-sensitive code with Swift 5 I will go ahead and use the UTF-8 view with the assumption that input strings are backed by UTF-8, and even force it to native UTF-8 if I'm doing enough processing that the potential string copy is outweighed by the savings during processing, but that's something that's only worth dealing with if there's a clear benefit to doing so. In most cases it's simpler just to use the unicode scalar view, as that doesn't have the potential for having to map UTF-8 sub-scalar offsets into a UTF-16 backing store (whereas unicode scalar offsets always lie on both UTF-8 and UTF-16 code unit boundaries). All that said, I would have been much happier if Swift could have been 100% UTF-8 from the get-go, which would drastically simplify a lot of this stuff. But the requirement for bridging to/from NSString makes that untenable as it would otherwise involve a lot of string copying every time you cross the Swift/Obj-C boundary.