6 ms·
Programmers are rarely interested in individual bytes, except sometimes if a string is abused as a byte array. In all other cases iterating or indexing over cha
by dtech 7y ago
Programmers are rarely interested in individual bytes, except sometimes if a string is abused as a byte array. In all other cases iterating or indexing over characters is the intention, and code points are the proper abstraction for characters, not bytes.
Also I might be wrong, but you can just look at the bytes to know how much bytes a UTF-8 character is, since the first byte with value 0 to 127 represents the final byte while 128 to 255 represents that the next byte is part of the character.
- maxdamantus 7y agoBut code points are not an abstraction for characters. The character "á" can be written as two code points (U+61, U+301). If a character can be two code points, why can't it be three bytes?
- danieldk 7y agoCode points are not the proper abstraction, since a character can be composed from a variable number of code points. Also, bytes from 0-127 in UTF-8 (MSB is 0) are ASCII characters. In multibyte code points, the MSBs of the first byte are 11 and of continuation bytes 10.
- mort96 7y agoExcept that code points aren't the proper abstraction for "characters". Most people would think of <Family: Woman, Woman, Girl, Boy>[1] as one character, but it's really five code points; woman, zero width joiner, woman, zero width joiner, girl, zero width joiner, boy. If you tried doing an operation like reversing a string or removing the last character, and you treated a unicode code point as a "character", you would end up with the wrong result. If you just removed the last code point to implement a backspace, you would end up with a string which ends in a zero width joiner, which makes little sense; and when the user wants to insert, say, a girl emoji, that emoji will end up as a part of the family due to that trailing joiner, when the user expected it to be a separate emoji. This applies to more than just emojis by the way; there are languages whose unicode representation is much more complicated than english or other languages with latin characters. [1]: https://emojipedia.org/family-woman-woman-girl-boy/ https://emojipedia.org/family-woman-woman-girl-boy/ EDIT: This comment originally used the actual emojis as examples, but hacker news just replaced every code point in the emoji with a space.
- Someone 7y agoSwift handles both cases well: var test = "test\u{1F469}" var tset = String(test.reversed()) var tes = test.dropLast() (The second line needs the extra ‘String’ to turn a sequence into a String; and yes, the names of the variable do not match their content)
- raiph 7y agoDoing it with just one emoji sort of misses the point... One way in P6, combining the full family into one character: my \test = "test\c[Woman,ZWJ,Woman,ZWJ,Girl,ZWJ,Boy]"; say test.chars; # 5 say test; # test say flip test; # tset say test.chop; # test HN displays the 7 codepoint family as three spaces. To see that P6 treats the family as one: https://tio.run/##K0gtyjH7/18BCHIrFWJKUotLFGwV1EG0ukKdQmleZkFiUXGqgk14fm5ingKEdM8sylFwyq@008vKz8yzUlDXiQr30lG35gIZU5xYqQDSrpecAdRpraCsYIoibq0AAcpg3of5E1c@auhFUMsh1DK4nrSczAKYRmUFnOpLilNL0O3PL7CG2fP/PwA https://tio.run/##K0gtyjH7/18BCHIrFWJKUotLFGwV1EG0ukKdQmleZk...
- Someone 7y agoSorry. Tested with the family, but didn’t notice that I only took the single code point when making _something_ show upon HN. It also works for “test\u{1F469}\u{200d}\u{1f469}\u{200d}\u{1f467}\u{200d}\u{1f466}"
- a1369209993 7y agoYou don't even need emoji; eg U+63,U+300 (c̀) is one characters but two code points, and U+1C4 (DŽ) is two characters, but one code point. There's also U+1F3,U+308 (dz̈), which is two characters in two code points, but segments incorrectly if you split on code points instead of characters. It's ambigous how to encode latin-small-a-with-ring-above (U+E4 vs U+61,U+30A). Decoding is also ambigous (most infamously Han grapheme clusters), but I'm not fluent enough in any of the affected languages to have a ready example. Also, that's seven code points, not five.
- planteen 7y ago
- gpderetta 7y agoa code unit with value 0-127 always maps to a single code point, i.e. it is never part of a multibyte character. It also maps 1:1 to ASCII values. So, if you are looking for specific characters in the ASCII code set, it is perfectly fine to iterate one byte at a time on an utf-8 string.