3 ms·

A character in UTF-8 can even be more than 4 bytes long. One example are flags or skin-colored emojis.

by foepys 5y ago

barrkel 5y ago
UTF-8 encodes code points, as do UTF-16 and UTF-32. Once you go from a sequence of bytes to a sequence of code points, you've moved beyond the specifics of the encoding. Code points might be combined to form graphemes and grapheme clusters. Some of the latest emojis are extended grapheme clusters, for e.g. handling the combinatorics of mixed families. This is a higher level composition than UTF-x, it's logically a separate layer. IMO talking about characters in the context of Unicode is often unhelpful because it's vague.
- tialaramex 5y ago
  Right. "Character" is almost never what you meant, unless your goal was to be as vague as possible. In human languages I like the word "squiggle" to mean this thing you have fuzzy intuitive beliefs about, rather than "character". In Unicode the Code Unit, and Code Point are maybe things to know about, but neither of them is a "character". In programming languages or APIs where precision matters, your goal should be to avoid this notion of characters as much as practical. In a high level language with types, just do not offer a built-in "char" data type. Sub-strings are all anybody in a high level language actually needs to get their job done, "A" is a perfectly good sub-string of "CAT" there's no need to pretend you can slice strings up into "characters" like 'A' that have any distinct properties worth inventing a whole datatype. If you're writing device drivers, once again, what do you care about "characters"? You want a byte data type, most likely, some address types, that sort of thing, but who wants a "character" ? However, somewhere down in the guts a low-level language will need to think about Unicode encoding, and so eventually they do need a datatype for that when a 32-bit integer doesn't really cut it. I think Rust's "char" is a little bit too prominent for example, it needn't be more in your face than say, std::num::NonZeroUsize. Most people won't need it, most of the time and that's as it should be.

A character in UTF-8 can even be more than 4 bytes long. One example are flags or skin-colored emojis.

by foepys 5y ago

A character in UTF-8 can even be more than 4 bytes long. One example are flags or skin-colored emojis.

UTF-8 encodes code points, as do UTF-16 and UTF-32. Once you go from a sequence of bytes to a sequence of code points, you've moved beyond the specifics of the encoding. Code points might be combined to form graphemes and grapheme clusters. Some of the latest emojis are extended grapheme clusters, for e.g. handling the combinatorics of mixed families. This is a higher level composition than UTF-x, it's logically a separate layer. IMO talking about characters in the context of Unicode is often unhelpful because it's vague.

tialaramex 5y ago
Right. "Character" is almost never what you meant, unless your goal was to be as vague as possible. In human languages I like the word "squiggle" to mean this thing you have fuzzy intuitive beliefs about, rather than "character". In Unicode the Code Unit, and Code Point are maybe things to know about, but neither of them is a "character". In programming languages or APIs where precision matters, your goal should be to avoid this notion of characters as much as practical. In a high level language with types, just do not offer a built-in "char" data type. Sub-strings are all anybody in a high level language actually needs to get their job done, "A" is a perfectly good sub-string of "CAT" there's no need to pretend you can slice strings up into "characters" like 'A' that have any distinct properties worth inventing a whole datatype. If you're writing device drivers, once again, what do you care about "characters"? You want a byte data type, most likely, some address types, that sort of thing, but who wants a "character" ? However, somewhere down in the guts a low-level language will need to think about Unicode encoding, and so eventually they do need a datatype for that when a 32-bit integer doesn't really cut it. I think Rust's "char" is a little bit too prominent for example, it needn't be more in your face than say, std::num::NonZeroUsize. Most people won't need it, most of the time and that's as it should be.