3 ms·

Using code points (or scalar values, I hope) just means that it’s inefficient for everyone, because now everyone has to convert indexes (well, except Python, bu

Using code points (or scalar values, I hope) just means that it’s inefficient for everyone, because now everyone has to convert indexes (well, except Python, but it has other problems), instead of only half the people. Going UTF-8 is fairly clearly superior: it will be the wire format, even if it’s not the language’s string format, so now environments that use UTF-8 strings never need any conversions (apart from decoding escape sequences, most likely). Much as I hate UTF-16, I would even be inclined to argue that UTF-16 was a better choice than code points, as it will reduce the amount of extra work UTF-16 environments have to do, without changing how much UTF-8 environments have to do at all; but it also has the disadvantage that validation is vanishingly rare in UTF-16, so you’re sure to end up with lone surrogate trouble at some point, whereas UTF-8 tooling has a much stronger culture of validation, so you’re much less likely to encounter it directly and can much more comfortably just declare “valid Unicode only”. Yes, code points is a purer concept to use. I don’t care: it’s less efficient than choosing UTF-8, which adds negative-to-negligible complexity. Please, just abandon code point indexing and embrace the UTF-8.

josephg 3y ago
No, I don’t agree. The problem with utf8 byte offsets is that it creates a data validation problem. In diamond types I’m using document positions / offsets in my wire format. With utf8 byte offsets, you can receive changes from remote peers which name invalid insertion positions. (Ie an insert inside a character, or deleting half of a codepoint). Validating remote changes received like this is a nightmare, because you need to reconstruct the whole document state to be able to tell if the edit is valid. Using Unicode codepoints makes invalid state unrepresentable. So the validation problem goes away. (You might still need to check that an insert isn’t past the end of the document, but that’s a much easier check). Almost all application programming languages use utf16 anyway, (javascript, c#, swift, Java) so you still need to convert positions anyway. Even in rust it’s common to see line/col positions from text editors. Using utf8 byte offsets just doesn’t really give you any benefits in exchange for making validation much harder.
- chrismorgan 3y ago
  The data validation concern seems fair enough. > Almost all application programming languages use utf16 anyway, (javascript, c#, swift, Java) Swift 5 switched to UTF-8: https://www.swift.org/blog/utf8-string/ https://www.swift.org/blog/utf8-string/. I’m hopeful that other UTF-16 environments might eventually manage to switch to UTF-8 internally despite retaining some UTF-16 code unit semantics for compatibility; two projects have already demonstrated you can very practically do this sort of thing: Servo from fairly early on (WTF-8 despite the web’s UTF-16 code unit semantics), and PyPy since 7.1 (UTF-8 despite code point semantics, not sure what they do about surrogate code points). I know the web has largely backed away from UTF-16 and uses code point semantics (well, scalar values plus loose surrogates) on almost all new stuff, with good UTF-8 support too.
- wtetzner 3y ago
  > Using Unicode codepoints makes invalid state unrepresentable. Is that true? Maybe it's not "invalid", but you might very well slice through the middle of a grapheme cluster.
  - chrismorgan 3y ago
    I contemplated querying that myself, but decided that for CRDT editing purposes, it’s probably never practical to think about grapheme clusters. Given text in a language with syllabic grapheme clusters (e.g. Indic languages), if you start with “ba” (one syllable/grapheme cluster) and edit it to “boca” (two syllables/grapheme clusters), you could say “replaced ‘ba’ with ‘boca’”, but I’d be surprised if any CRDT did it that way if it could instead handle it as “inserted ‘oc’”, even though “oc” mightn’t make sense by itself, linguistically. But Unicode doesn’t do much in the way of defining what makes sense or not, and I don’t think there’s any coherent “bad grapheme clustering” detection algorithm. (Aside: on reflection, my Indic languages example is messier still, since -a probably represents the inherent vowel, so in “ba” → “boca” those “a”s are probably actually represented by the absence of a vowel sign code point—and if you wanted to suppress the inherent vowel, you’d need to add a virama sign. Fun stuff.) But then again, I know that some CRDTs struggle with interleaving, and maybe grapheme-awareness could help things out in some way or other. I dunno.

josephg 3y ago

No, I don’t agree. The problem with utf8 byte offsets is that it creates a data validation problem. In diamond types I’m using document positions / offsets in my wire format. With utf8 byte offsets, you can receive changes from remote peers which name invalid insertion positions. (Ie an insert inside a character, or deleting half of a codepoint). Validating remote changes received like this is a nightmare, because you need to reconstruct the whole document state to be able to tell if the edit is valid. Using Unicode codepoints makes invalid state unrepresentable. So the validation problem goes away. (You might still need to check that an insert isn’t past the end of the document, but that’s a much easier check). Almost all application programming languages use utf16 anyway, (javascript, c#, swift, Java) so you still need to convert positions anyway. Even in rust it’s common to see line/col positions from text editors. Using utf8 byte offsets just doesn’t really give you any benefits in exchange for making validation much harder.

chrismorgan 3y ago
The data validation concern seems fair enough. > Almost all application programming languages use utf16 anyway, (javascript, c#, swift, Java) Swift 5 switched to UTF-8: https://www.swift.org/blog/utf8-string/ https://www.swift.org/blog/utf8-string/. I’m hopeful that other UTF-16 environments might eventually manage to switch to UTF-8 internally despite retaining some UTF-16 code unit semantics for compatibility; two projects have already demonstrated you can very practically do this sort of thing: Servo from fairly early on (WTF-8 despite the web’s UTF-16 code unit semantics), and PyPy since 7.1 (UTF-8 despite code point semantics, not sure what they do about surrogate code points). I know the web has largely backed away from UTF-16 and uses code point semantics (well, scalar values plus loose surrogates) on almost all new stuff, with good UTF-8 support too.
wtetzner 3y ago
> Using Unicode codepoints makes invalid state unrepresentable. Is that true? Maybe it's not "invalid", but you might very well slice through the middle of a grapheme cluster.
- chrismorgan 3y ago
  I contemplated querying that myself, but decided that for CRDT editing purposes, it’s probably never practical to think about grapheme clusters. Given text in a language with syllabic grapheme clusters (e.g. Indic languages), if you start with “ba” (one syllable/grapheme cluster) and edit it to “boca” (two syllables/grapheme clusters), you could say “replaced ‘ba’ with ‘boca’”, but I’d be surprised if any CRDT did it that way if it could instead handle it as “inserted ‘oc’”, even though “oc” mightn’t make sense by itself, linguistically. But Unicode doesn’t do much in the way of defining what makes sense or not, and I don’t think there’s any coherent “bad grapheme clustering” detection algorithm. (Aside: on reflection, my Indic languages example is messier still, since -a probably represents the inherent vowel, so in “ba” → “boca” those “a”s are probably actually represented by the absence of a vowel sign code point—and if you wanted to suppress the inherent vowel, you’d need to add a virama sign. Fun stuff.) But then again, I know that some CRDTs struggle with interleaving, and maybe grapheme-awareness could help things out in some way or other. I dunno.