4 ms·
There are multiple factors at play here and it ultimately comes down to what you need: If you to access code points, then UTF-32 gives you a nice fixed-width (
by ygra 13y ago
There are multiple factors at play here and it ultimately comes down to what you need:
If you to access code points, then UTF-32 gives you a nice fixed-width (albeit quite large) encoding to work with. Some applications or libraries may use UTF-32 internally to speed things up.
If you deal with things on screen or what humans perceive as character (graphemes) then there is nothing really that can help you at the encoding level because graphemes can be arbitrarily long with regard to code points: e̬̱ͤ̈́̿̂̅́͟x̰̞̻̼̻̼͉̰ͫͧ̑͂͊͜ͅa̶̳͖͔̞̰͈̯͓͆̓̎̉̏̎̚m̛̮̩͙ͦ͗͆̄̋́̄p͎̥̠͈̮͚̾ͅl̛̗̈̄e̴̞̫͖̝̘̋̑̄̎ͥͬ̾ (this probably doesn't even render properly).
Sadly, most APIs use either UTF-8 or UTF-16 and consider code units the most important entity, not code points. And that's inexcusable and frankly a mess, because I can count the number of times I needed to access a code unit instead of a code point on zero hands.
- Flimm 13y agoPlease define what you mean by code point and code unit.
- ygra 13y agoI mean the same thing that Unicode uses these terms for. Code Unit. The minimal bit combination that can represent a unit of encoded text for processing or interchange. The Unicode Standard uses 8-bit code units in the UTF-8 encoding form, 16-bit code units in the UTF-16 encoding form, and 32-bit code units in the UTF-32 encoding form. Code Point. (1) Any value in the Unicode codespace; that is, the range of integers from 0 to 10FFFF16. (See definition D10 in Section 3.4, Characters and Encoding.) Not all code points are assigned to encoded characters. See code point type. (2) A value, or position, for a character, in any coded character set. From http://www.unicode.org/glossary/ http://www.unicode.org/glossary/
- iv_08 13y agoFor index based operations code units are more useful than code points. a = s.indexOf("e̬̱ͤ̈́̿̂̅́͟") b = s.indexOf("m̛̮̩͙ͦ͗͆̄̋́̄") s[a:b] This operation is fast with code unit indices and slow with code point indices.
- deleted 13y ago[deleted]
- tripzilch 13y ago> If you deal with things on screen or what humans perceive as character (graphemes) then there is nothing really that can help you at the encoding level because graphemes can be arbitrarily long with regard to code points: > e̬̱ͤ̈́̿̂̅́͟x̰̞̻̼̻̼͉̰ͫͧ̑͂͊͜ͅa̶̳͖͔̞̰͈̯͓͆̓̎̉̏̎̚m̛̮̩͙ͦ͗͆̄̋́̄p͎̥̠͈̮͚̾ͅl̛̗̈̄e̴̞̫͖̝̘̋̑̄̎ͥͬ̾ As how many characters am I supposed to perceive this, anyway? Some of those tiny squiggles are in fact letters (I mean they look like letters, I see an e,m,u,o,i and r in there). What if you'd make a crossword this way?
- thaumasiotes 13y agoI've experienced crosswords where e.g. the word going across uses a diacritic on a particular letter ("élan") and the word going down doesn't. It drives me crazy.
- riffraff 13y agoFYI, that is the way crosswords work in italian, accents are considered irrelevant.
- dbaupp 13y agoIt says "example" with a pile of combining characters (77 to be precise): http://i.imgur.com/acyHBkR.png http://i.imgur.com/acyHBkR.png
- Dylan16807 13y agoIf you use stream-safe normalization rules, then you have a worst-case limit of 30 combining code points in a row. You could even make an array of 128 byte single-character-strings if you wanted.