4 ms·

> Code point length is the most useful for people who are actually writing string algorithms based upon Unicode. What algorithms would you be writing against c

> Code point length is the most useful for people who are actually writing string algorithms based upon Unicode. What algorithms would you be writing against code points?

insanitybit 3y ago
I suspect primarily substring, where if you index by bytes you'll mangle the string, but if you index by codepoints everything works out.
- deleted 3y ago
  [deleted]
- jlokier 3y ago
  You'll mangle the string if you index and search by code points too, when the string contains the emoji this article is about, or for that matter an "e" following by combining acute accent. The string will be fine if you only move to, split and concatenate at indices outside those grapheme clusters. But that is also true when indexing by bytes or UTF-16 code units. So in some senses, indexing by bytes is just as good as indexing by code points, but faster. Either way to avoid mangling strings you need to restrict the indices of whatever type to meaningful character boundaries. If you have decided to avoid string indices inside grapheme clusters, there comes the awkard question of what should you do when editing text in an environment rendered with font ligatures like "->" rendered as → (rightward arrow). From one perspective, that's just a font. From another, the user sees a single character yet there are valid positions (such as from cursor movement and character search) that land mid-way through the character, and editing at those positions changes the character. Neither is clearly best for all situations.
Izkata 3y ago
Isn't that the level emoji sequences work at?
zarzavat 3y ago
Everything that you can do to a Unicode string, except concatenation, is defined in terms of code points. Normalization, case transformations, collation, regexes, layout and rendering and encoding. For example, let’s say you want to define a “natural sort” order that sorts e.g. “A2” < “A10”. To do that you divide the string at boundaries between code points in ranges of each numeral type that you are supporting (e.g. western numerals, Arabic numerals, Chinese numerals).
- wtetzner 3y ago
  Wouldn't layout/rendering etc. be done in terms of grapheme clusters?
josephg 3y ago
Codepoints is best for collaborative text editing / CRDTs (diamond types, automerge, etc). We generally model the document as a big list of unicode codepoints. We could use grapheme clusters, but the grapheme cluster boundary points change as unicode evolves, and not all systems update at the same time. Separating strings based on grapheme cluster boundaries also requires a big lookup table to be embedded in every app. Unicode codepoints are obvious, stable, and easy to work with. And they're encoding-agnostic, so there's no weird UCS2-to-UTF8-bytes conversion needed in javascript, C#, etc.
- chrismorgan 3y ago
  Using code points (or scalar values, I hope) just means that it’s inefficient for everyone, because now everyone has to convert indexes (well, except Python, but it has other problems), instead of only half the people. Going UTF-8 is fairly clearly superior: it will be the wire format, even if it’s not the language’s string format, so now environments that use UTF-8 strings never need any conversions (apart from decoding escape sequences, most likely). Much as I hate UTF-16, I would even be inclined to argue that UTF-16 was a better choice than code points, as it will reduce the amount of extra work UTF-16 environments have to do, without changing how much UTF-8 environments have to do at all; but it also has the disadvantage that validation is vanishingly rare in UTF-16, so you’re sure to end up with lone surrogate trouble at some point, whereas UTF-8 tooling has a much stronger culture of validation, so you’re much less likely to encounter it directly and can much more comfortably just declare “valid Unicode only”. Yes, code points is a purer concept to use. I don’t care: it’s less efficient than choosing UTF-8, which adds negative-to-negligible complexity. Please, just abandon code point indexing and embrace the UTF-8.
  - josephg 3y ago
    No, I don’t agree. The problem with utf8 byte offsets is that it creates a data validation problem. In diamond types I’m using document positions / offsets in my wire format. With utf8 byte offsets, you can receive changes from remote peers which name invalid insertion positions. (Ie an insert inside a character, or deleting half of a codepoint). Validating remote changes received like this is a nightmare, because you need to reconstruct the whole document state to be able to tell if the edit is valid. Using Unicode codepoints makes invalid state unrepresentable. So the validation problem goes away. (You might still need to check that an insert isn’t past the end of the document, but that’s a much easier check). Almost all application programming languages use utf16 anyway, (javascript, c#, swift, Java) so you still need to convert positions anyway. Even in rust it’s common to see line/col positions from text editors. Using utf8 byte offsets just doesn’t really give you any benefits in exchange for making validation much harder.

> Code point length is the most useful for people who are actually writing string algorithms based upon Unicode. What algorithms would you be writing against code points?

insanitybit 3y ago

I suspect primarily substring, where if you index by bytes you'll mangle the string, but if you index by codepoints everything works out.

deleted 3y ago
[deleted]
jlokier 3y ago
You'll mangle the string if you index and search by code points too, when the string contains the emoji this article is about, or for that matter an "e" following by combining acute accent. The string will be fine if you only move to, split and concatenate at indices outside those grapheme clusters. But that is also true when indexing by bytes or UTF-16 code units. So in some senses, indexing by bytes is just as good as indexing by code points, but faster. Either way to avoid mangling strings you need to restrict the indices of whatever type to meaningful character boundaries. If you have decided to avoid string indices inside grapheme clusters, there comes the awkard question of what should you do when editing text in an environment rendered with font ligatures like "->" rendered as → (rightward arrow). From one perspective, that's just a font. From another, the user sees a single character yet there are valid positions (such as from cursor movement and character search) that land mid-way through the character, and editing at those positions changes the character. Neither is clearly best for all situations.

Izkata 3y ago

Isn't that the level emoji sequences work at?

zarzavat 3y ago

Everything that you can do to a Unicode string, except concatenation, is defined in terms of code points. Normalization, case transformations, collation, regexes, layout and rendering and encoding. For example, let’s say you want to define a “natural sort” order that sorts e.g. “A2” < “A10”. To do that you divide the string at boundaries between code points in ranges of each numeral type that you are supporting (e.g. western numerals, Arabic numerals, Chinese numerals).

wtetzner 3y ago
Wouldn't layout/rendering etc. be done in terms of grapheme clusters?

josephg 3y ago

Codepoints is best for collaborative text editing / CRDTs (diamond types, automerge, etc). We generally model the document as a big list of unicode codepoints. We could use grapheme clusters, but the grapheme cluster boundary points change as unicode evolves, and not all systems update at the same time. Separating strings based on grapheme cluster boundaries also requires a big lookup table to be embedded in every app. Unicode codepoints are obvious, stable, and easy to work with. And they're encoding-agnostic, so there's no weird UCS2-to-UTF8-bytes conversion needed in javascript, C#, etc.

chrismorgan 3y ago
Using code points (or scalar values, I hope) just means that it’s inefficient for everyone, because now everyone has to convert indexes (well, except Python, but it has other problems), instead of only half the people. Going UTF-8 is fairly clearly superior: it will be the wire format, even if it’s not the language’s string format, so now environments that use UTF-8 strings never need any conversions (apart from decoding escape sequences, most likely). Much as I hate UTF-16, I would even be inclined to argue that UTF-16 was a better choice than code points, as it will reduce the amount of extra work UTF-16 environments have to do, without changing how much UTF-8 environments have to do at all; but it also has the disadvantage that validation is vanishingly rare in UTF-16, so you’re sure to end up with lone surrogate trouble at some point, whereas UTF-8 tooling has a much stronger culture of validation, so you’re much less likely to encounter it directly and can much more comfortably just declare “valid Unicode only”. Yes, code points is a purer concept to use. I don’t care: it’s less efficient than choosing UTF-8, which adds negative-to-negligible complexity. Please, just abandon code point indexing and embrace the UTF-8.
- josephg 3y ago
  No, I don’t agree. The problem with utf8 byte offsets is that it creates a data validation problem. In diamond types I’m using document positions / offsets in my wire format. With utf8 byte offsets, you can receive changes from remote peers which name invalid insertion positions. (Ie an insert inside a character, or deleting half of a codepoint). Validating remote changes received like this is a nightmare, because you need to reconstruct the whole document state to be able to tell if the edit is valid. Using Unicode codepoints makes invalid state unrepresentable. So the validation problem goes away. (You might still need to check that an insert isn’t past the end of the document, but that’s a much easier check). Almost all application programming languages use utf16 anyway, (javascript, c#, swift, Java) so you still need to convert positions anyway. Even in rust it’s common to see line/col positions from text editors. Using utf8 byte offsets just doesn’t really give you any benefits in exchange for making validation much harder.