6 ms·

Is there any real case where code point indexing is useful? It seems like all these attempts to restrict strings in such a way to accommodate code points is jus

Is there any real case where code point indexing is useful? It seems like all these attempts to restrict strings in such a way to accommodate code points is just introducing complexity with no gain. UTF-8 was designed to be an encoding (of code points) on top of the "bytes" abstraction just as Unicode is designed to be an encoding (of human text) on top of the "code points" abstraction. I think it should be uncontroversial that there are very good reasons to at least handle arbitrary sequences of code points (eg, you want to be able to handle input from future versions of Unicode, and you don't know about the grapheme clustering of those code points), but I don't see a good reason not to handle arbitrary sequences of bytes. The only reason I can see is in ensuring that text is losslessly convertible to other UTFs, particularly UTF-16 (which exists for historical reasons), but this just seems like a matter of when the information is lost (is it during conversion from string to UTF-16, or from bytes to string), not if it is lost. As far as I can tell with the Python story, for example, people decided to add special "Unicode" strings into Python 2, then presumably some code used the "Unicode" strings and some code used the "byte" strings, so this situation is obviously underisable .. then in Python 3 they tried fixing it by replacing which sort of string was the default. Why would it not have been better to just improve Unicode support for the existing strings instead of splitting the type into two and forcing everyone to decide whether their strings are for "bytes" or for "Unicode"?

dtech 7y ago
Programmers are rarely interested in individual bytes, except sometimes if a string is abused as a byte array. In all other cases iterating or indexing over characters is the intention, and code points are the proper abstraction for characters, not bytes. Also I might be wrong, but you can just look at the bytes to know how much bytes a UTF-8 character is, since the first byte with value 0 to 127 represents the final byte while 128 to 255 represents that the next byte is part of the character.
- maxdamantus 7y ago
  But code points are not an abstraction for characters. The character "á" can be written as two code points (U+61, U+301). If a character can be two code points, why can't it be three bytes?
- danieldk 7y ago
  Code points are not the proper abstraction, since a character can be composed from a variable number of code points. Also, bytes from 0-127 in UTF-8 (MSB is 0) are ASCII characters. In multibyte code points, the MSBs of the first byte are 11 and of continuation bytes 10.
- mort96 7y ago
  Except that code points aren't the proper abstraction for "characters". Most people would think of <Family: Woman, Woman, Girl, Boy>[1] as one character, but it's really five code points; woman, zero width joiner, woman, zero width joiner, girl, zero width joiner, boy. If you tried doing an operation like reversing a string or removing the last character, and you treated a unicode code point as a "character", you would end up with the wrong result. If you just removed the last code point to implement a backspace, you would end up with a string which ends in a zero width joiner, which makes little sense; and when the user wants to insert, say, a girl emoji, that emoji will end up as a part of the family due to that trailing joiner, when the user expected it to be a separate emoji. This applies to more than just emojis by the way; there are languages whose unicode representation is much more complicated than english or other languages with latin characters. [1]: https://emojipedia.org/family-woman-woman-girl-boy/ https://emojipedia.org/family-woman-woman-girl-boy/ EDIT: This comment originally used the actual emojis as examples, but hacker news just replaced every code point in the emoji with a space.
  - Someone 7y ago
    Swift handles both cases well: var test = "test\u{1F469}" var tset = String(test.reversed()) var tes = test.dropLast() (The second line needs the extra ‘String’ to turn a sequence into a String; and yes, the names of the variable do not match their content)
    - raiph 7y ago
      Doing it with just one emoji sort of misses the point... One way in P6, combining the full family into one character: my \test = "test\c[Woman,ZWJ,Woman,ZWJ,Girl,ZWJ,Boy]"; say test.chars; # 5 say test; # test say flip test; # tset say test.chop; # test HN displays the 7 codepoint family as three spaces. To see that P6 treats the family as one: https://tio.run/##K0gtyjH7/18BCHIrFWJKUotLFGwV1EG0ukKdQmleZkFiUXGqgk14fm5ingKEdM8sylFwyq@008vKz8yzUlDXiQr30lG35gIZU5xYqQDSrpecAdRpraCsYIoibq0AAcpg3of5E1c@auhFUMsh1DK4nrSczAKYRmUFnOpLilNL0O3PL7CG2fP/PwA https://tio.run/##K0gtyjH7/18BCHIrFWJKUotLFGwV1EG0ukKdQmleZk...
- gpderetta 7y ago
  a code unit with value 0-127 always maps to a single code point, i.e. it is never part of a multibyte character. It also maps 1:1 to ASCII values. So, if you are looking for specific characters in the ASCII code set, it is perfectly fine to iterate one byte at a time on an utf-8 string.
jhanschoo 7y ago
> The only reason I can see is in ensuring that text is losslessly convertible to other UTFs As other top-level comments and the article have mentioned, different systems still use different internal representations, but all modern ones commit to being able to represent Unicode code points, with no automatic normalization. To that end I suppose that in terms of reasoning about the consistency of data in different systems, it is better to use code points than the actual size, which is left to the implementation. The possibly better alternative would be to use lengths in UTF-8, but that might seem arbitrary to some. Perhaps counting code points is useful in that it gives a lower bound on the length in any reasonable encoding.
- maxdamantus 7y ago
  > Perhaps counting code points is useful in that it gives a lower bound on the length in any reasonable encoding. But "á" is two code points (U+61, U+301). If you're looking for some lower bound (whatever that means), shouldn't it be 1? I imagine if you're looking for something like information density, the count of UTF-8 code units would at least be somewhat more informative than the count of code points. I guess the crux of this whole point is that a sequence of code points is arbitrary in the same way as a sequence of bytes; neither "code point" nor "byte" necessarily corresponds to something that a user would see as a unit in human text. So why are we not using the simpler abstraction?
Rusky 7y ago
> The only reason I can see is in ensuring that text is losslessly convertible to other UTFs, particularly UTF-16 (which exists for historical reasons), but this just seems like a matter of when the information is lost (is it during conversion from string to UTF-16, or from bytes to string), not if it is lost. And you can even avoid losing that information at all with encodings like WTF-8: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/
masklinn 7y ago
> The only reason I can see is in ensuring that text is losslessly convertible to other UTFs, particularly UTF-16 Text is by definition losslessly convertible between UTFs. You only "lose" information if your source is not a correct UTF stream e.g. if you have lone surrogates you don't actually have UTF-16, you have a garbage pile of UTF-16 code units. Now it may be useful to properly round-trip this garbage pile (e.g. you're dealing with filenames on Windows), but this should not be confused with data conversion between UTFs: your source simply is not UTF-16.
- maxdamantus 7y ago
  I'm not sure there's a clear definition of what valid "text" is, but surely classifying it as something like "a sequence of Unicode scalar values" (equivalent to "something that can be encoded in a UTF") is a bit arbitrary. Is a sequence of unassigned Unicode scalar values really "text"? Maybe "text" should not start with a combining character. Maybe it should be grammatically valid in some human language. Again, unless the point of all of this is to cater for obsolete encodings for the rest of eternity, "sequence of code points" (or here "sequence of Unicode scalar values" [0]) seems just as arbitrary as "sequence of bytes". [0] Probably worth mentioning that as far as I'm aware, these systems tend to use code points, not Unicode scalar values, so the strings are not guaranteed representable by UTFs anyway (Python allows "\udc80", and even started making use of such strings as a latent workaround to handle non-UTF-8 input some time after Python 3 was released [1]) [1] https://www.python.org/dev/peps/pep-0383/ https://www.python.org/dev/peps/pep-0383/
  - masklinn 7y ago
    I've no idea what you're trying to say. All I'm saying is > ensuring that text is losslessly convertible to other UTFs is a truism, it can't not be true. You can always convert back and forth between UTF8 and UTF16 with no loss or alteration of data.
    - maxdamantus 7y ago
      It's not a truism if "text" includes bytes that are not UTF-8. A program would probably naturally model a filename as being text, but if that means "sequence of Unicode scalar values", it's arguably incorrect on some obscure systems, such as Linux and git.

dtech 7y ago

Programmers are rarely interested in individual bytes, except sometimes if a string is abused as a byte array. In all other cases iterating or indexing over characters is the intention, and code points are the proper abstraction for characters, not bytes. Also I might be wrong, but you can just look at the bytes to know how much bytes a UTF-8 character is, since the first byte with value 0 to 127 represents the final byte while 128 to 255 represents that the next byte is part of the character.

maxdamantus 7y ago
But code points are not an abstraction for characters. The character "á" can be written as two code points (U+61, U+301). If a character can be two code points, why can't it be three bytes?
danieldk 7y ago
Code points are not the proper abstraction, since a character can be composed from a variable number of code points. Also, bytes from 0-127 in UTF-8 (MSB is 0) are ASCII characters. In multibyte code points, the MSBs of the first byte are 11 and of continuation bytes 10.
mort96 7y ago
Except that code points aren't the proper abstraction for "characters". Most people would think of <Family: Woman, Woman, Girl, Boy>[1] as one character, but it's really five code points; woman, zero width joiner, woman, zero width joiner, girl, zero width joiner, boy. If you tried doing an operation like reversing a string or removing the last character, and you treated a unicode code point as a "character", you would end up with the wrong result. If you just removed the last code point to implement a backspace, you would end up with a string which ends in a zero width joiner, which makes little sense; and when the user wants to insert, say, a girl emoji, that emoji will end up as a part of the family due to that trailing joiner, when the user expected it to be a separate emoji. This applies to more than just emojis by the way; there are languages whose unicode representation is much more complicated than english or other languages with latin characters. [1]: https://emojipedia.org/family-woman-woman-girl-boy/ https://emojipedia.org/family-woman-woman-girl-boy/ EDIT: This comment originally used the actual emojis as examples, but hacker news just replaced every code point in the emoji with a space.
- Someone 7y ago
  Swift handles both cases well: var test = "test\u{1F469}" var tset = String(test.reversed()) var tes = test.dropLast() (The second line needs the extra ‘String’ to turn a sequence into a String; and yes, the names of the variable do not match their content)
  - raiph 7y ago
    Doing it with just one emoji sort of misses the point... One way in P6, combining the full family into one character: my \test = "test\c[Woman,ZWJ,Woman,ZWJ,Girl,ZWJ,Boy]"; say test.chars; # 5 say test; # test say flip test; # tset say test.chop; # test HN displays the 7 codepoint family as three spaces. To see that P6 treats the family as one: https://tio.run/##K0gtyjH7/18BCHIrFWJKUotLFGwV1EG0ukKdQmleZkFiUXGqgk14fm5ingKEdM8sylFwyq@008vKz8yzUlDXiQr30lG35gIZU5xYqQDSrpecAdRpraCsYIoibq0AAcpg3of5E1c@auhFUMsh1DK4nrSczAKYRmUFnOpLilNL0O3PL7CG2fP/PwA https://tio.run/##K0gtyjH7/18BCHIrFWJKUotLFGwV1EG0ukKdQmleZk...
gpderetta 7y ago
a code unit with value 0-127 always maps to a single code point, i.e. it is never part of a multibyte character. It also maps 1:1 to ASCII values. So, if you are looking for specific characters in the ASCII code set, it is perfectly fine to iterate one byte at a time on an utf-8 string.

jhanschoo 7y ago

> The only reason I can see is in ensuring that text is losslessly convertible to other UTFs As other top-level comments and the article have mentioned, different systems still use different internal representations, but all modern ones commit to being able to represent Unicode code points, with no automatic normalization. To that end I suppose that in terms of reasoning about the consistency of data in different systems, it is better to use code points than the actual size, which is left to the implementation. The possibly better alternative would be to use lengths in UTF-8, but that might seem arbitrary to some. Perhaps counting code points is useful in that it gives a lower bound on the length in any reasonable encoding.

maxdamantus 7y ago
> Perhaps counting code points is useful in that it gives a lower bound on the length in any reasonable encoding. But "á" is two code points (U+61, U+301). If you're looking for some lower bound (whatever that means), shouldn't it be 1? I imagine if you're looking for something like information density, the count of UTF-8 code units would at least be somewhat more informative than the count of code points. I guess the crux of this whole point is that a sequence of code points is arbitrary in the same way as a sequence of bytes; neither "code point" nor "byte" necessarily corresponds to something that a user would see as a unit in human text. So why are we not using the simpler abstraction?

Rusky 7y ago

> The only reason I can see is in ensuring that text is losslessly convertible to other UTFs, particularly UTF-16 (which exists for historical reasons), but this just seems like a matter of when the information is lost (is it during conversion from string to UTF-16, or from bytes to string), not if it is lost. And you can even avoid losing that information at all with encodings like WTF-8: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/

masklinn 7y ago

> The only reason I can see is in ensuring that text is losslessly convertible to other UTFs, particularly UTF-16 Text is by definition losslessly convertible between UTFs. You only "lose" information if your source is not a correct UTF stream e.g. if you have lone surrogates you don't actually have UTF-16, you have a garbage pile of UTF-16 code units. Now it may be useful to properly round-trip this garbage pile (e.g. you're dealing with filenames on Windows), but this should not be confused with data conversion between UTFs: your source simply is not UTF-16.

maxdamantus 7y ago
I'm not sure there's a clear definition of what valid "text" is, but surely classifying it as something like "a sequence of Unicode scalar values" (equivalent to "something that can be encoded in a UTF") is a bit arbitrary. Is a sequence of unassigned Unicode scalar values really "text"? Maybe "text" should not start with a combining character. Maybe it should be grammatically valid in some human language. Again, unless the point of all of this is to cater for obsolete encodings for the rest of eternity, "sequence of code points" (or here "sequence of Unicode scalar values" [0]) seems just as arbitrary as "sequence of bytes". [0] Probably worth mentioning that as far as I'm aware, these systems tend to use code points, not Unicode scalar values, so the strings are not guaranteed representable by UTFs anyway (Python allows "\udc80", and even started making use of such strings as a latent workaround to handle non-UTF-8 input some time after Python 3 was released [1]) [1] https://www.python.org/dev/peps/pep-0383/ https://www.python.org/dev/peps/pep-0383/
- masklinn 7y ago
  I've no idea what you're trying to say. All I'm saying is > ensuring that text is losslessly convertible to other UTFs is a truism, it can't not be true. You can always convert back and forth between UTF8 and UTF16 with no loss or alteration of data.
  - maxdamantus 7y ago
    It's not a truism if "text" includes bytes that are not UTF-8. A program would probably naturally model a filename as being text, but if that means "sequence of Unicode scalar values", it's arguably incorrect on some obscure systems, such as Linux and git.