5 ms·
"16 bits should be enough for anybody." --Anonymous The funny thing is that 16 bits wasn't enough. We thought it would be, back then. Just use 16 bits for eac
by Stratoscope 13y ago
"16 bits should be enough for anybody." --Anonymous
The funny thing is that 16 bits wasn't enough.
We thought it would be, back then. Just use 16 bits for each character and we can return to the joy and simplicity of a fixed width encoding.
That didn't work out. UCS-2 was a fixed width 16-bit encoding, but it it had to be extended into the variable width UTF-16 to get more code points. You get the bloat of 16-bit data, incompatibility with ASCII, and variable width characters too.
You have to go all the way to UTF-32 to get back to a fixed width encoding.
So UTF-8 really was a smart move. Bite the bullet on the variable width, but make it so you can always find the beginning of the next character even if you start in the middle of one, and make it backward compatible with ASCII too. Genius.
https://en.wikipedia.org/wiki/UTF-8 https://en.wikipedia.org/wiki/UTF-8
https://en.wikipedia.org/wiki/UTF-16 https://en.wikipedia.org/wiki/UTF-16
https://en.wikipedia.org/wiki/UTF-32 https://en.wikipedia.org/wiki/UTF-32
- sopooneo 13y agoPlease correct me if I'm wrong, but you don't really get fixed width in anyway do you? Because unicode itself is not fixed width. There are situations where one character on the screen is represented by two consecutive unicode code points.
- plorkyeran 13y agoOr a lot more than two, or even multiple characters for a single code point. UTF-32 appears to simplify things a lot, but in practice O(1) indexing of code points and simple iteration over code points is not actually all that useful, because code points are not characters.
- runn1ng 13y agoUTF8 != unicode. Unicode is a standard, that maps the giant number of letters to unique numbers. UTF8 (and UTF16, UTF32) are various encodings of that. Unicode itself is not an encoding; unicode just says "letter 'č' is 269"; UTF-8 says "We will represent it as two bytes, first is C4, second is 8D". UTF8 is indeed variable width; UTF16 and UTF32 are, AFAIK, fixed width. Both use unicode standard.
- pjscott 13y agoWhat the comment you're replying to is saying that glyphs may be represented by more than one code point. So sure, UTF32 has a fixed width for each code point, but it's not quite so clear how useful that is.
- runn1ng 13y agoOh, in that case I misunderstood parent. Sorry about that.
- Stratoscope 13y agoEven if we're talking about code points and not characters, UTF-16 is variable width. UTF-8 is variable width using 8, 16, 24, or 32 bits per code point. UCS-2 is fixed width using 16 bits per code point. UTF-16 is variable width using 16 or 32 bits per code point. It's an extension of the fixed width UCS-2 format: every valid 16-bit UCS-2 code point is also a valid UTF-16 code point, but UTF-16 also includes 32-bit code points. UTF-32 is fixed width using 32 bits per code point.
- alister 13y agoUTF-8 is variable width using 8, 16, 24, or 32 bits per code point. I wondered why you didn't write that "UTF-8 is variable width using 8, 16, 24, 32, 40, or 48 bits per code point", because that's how Prosser/Thompson/Pike's UTF-8 idea was proposed, as shown in the "most elegant hack" article. And here's the answer from Wikipedia: In November 2003 UTF-8 was restricted by RFC 3629 to end at U+10FFFF, in order to match the constraints of the UTF-16 character encoding. This removed all 5- and 6-byte sequences, and about half of the 4-byte sequences. Even though they eliminated the 5th and 6th bytes, UTF-8 can still represent all of Unicode's 1,114,112 code points. I'm mentioning this because others probably are wondering the same thing.
- option_greek 13y agoHow sad. One defective format limits another great format.
- duaneb 13y agoCharacters are not fixed width, no, but codepoints are. The idea of "what exactly constitutes a character" gets pretty fuzzy in some cases anyway with utility codepoints.
- jameshart 13y agoEven Windows-format ASCII isn't fixed width - CR-LF is a two-byte 'codepoint'...
- nitrogen 13y agoYou can think of it as two separate instructions, CR to move horizontally to the beginning of the line, and LF to move down by one line.
- zorked 13y agoNot really because a single CR or LF does not do a real CR or LF.
- prewett 13y agoIt used to, back in The Day, when you had hard-copy consoles or even dot matrix printers. I think sending CR (but not LF) and then printing accents over top of the original line was a way of printing some European languages with just ASCII.
- barrkel 13y ago$ man terminfo | grep cursor_down | head -1 cursor_down cud1 do down one line $ tput cud1 | hexdump -C | cut -f 1-20 | head -1 00000000 0a |.| $ man terminfo | grep carriage_return | head -1 carriage_return cr cr carriage return (P*) $ tput cr | hexdump -C | cut -f 1-20 | head -1 00000000 0d |.| If you ever write a bash script that does more advanced things with the terminal, you'll get familiar with tput, termcap and terminfo pretty quickly.
- kps 13y agoIt does in ASCII, and (consequently) in any ANSI-conforming terminal [emulator].
- jameshart 13y ago
- ygra 13y agoThere are multiple factors at play here and it ultimately comes down to what you need: If you to access code points, then UTF-32 gives you a nice fixed-width (albeit quite large) encoding to work with. Some applications or libraries may use UTF-32 internally to speed things up. If you deal with things on screen or what humans perceive as character (graphemes) then there is nothing really that can help you at the encoding level because graphemes can be arbitrarily long with regard to code points: e̬̱ͤ̈́̿̂̅́͟x̰̞̻̼̻̼͉̰ͫͧ̑͂͊͜ͅa̶̳͖͔̞̰͈̯͓͆̓̎̉̏̎̚m̛̮̩͙ͦ͗͆̄̋́̄p͎̥̠͈̮͚̾ͅl̛̗̈̄e̴̞̫͖̝̘̋̑̄̎ͥͬ̾ (this probably doesn't even render properly). Sadly, most APIs use either UTF-8 or UTF-16 and consider code units the most important entity, not code points. And that's inexcusable and frankly a mess, because I can count the number of times I needed to access a code unit instead of a code point on zero hands.
- Flimm 13y agoPlease define what you mean by code point and code unit.
- ygra 13y agoI mean the same thing that Unicode uses these terms for. Code Unit. The minimal bit combination that can represent a unit of encoded text for processing or interchange. The Unicode Standard uses 8-bit code units in the UTF-8 encoding form, 16-bit code units in the UTF-16 encoding form, and 32-bit code units in the UTF-32 encoding form. Code Point. (1) Any value in the Unicode codespace; that is, the range of integers from 0 to 10FFFF16. (See definition D10 in Section 3.4, Characters and Encoding.) Not all code points are assigned to encoded characters. See code point type. (2) A value, or position, for a character, in any coded character set. From http://www.unicode.org/glossary/ http://www.unicode.org/glossary/
- iv_08 13y agoFor index based operations code units are more useful than code points. a = s.indexOf("e̬̱ͤ̈́̿̂̅́͟") b = s.indexOf("m̛̮̩͙ͦ͗͆̄̋́̄") s[a:b] This operation is fast with code unit indices and slow with code point indices.
- evincarofautumn 13y ago> You have to go all the way to UTF-32 to get back to a fixed width encoding. UTF-21, actually, as there are no code points above 0x10FFFF. Powers of two have other nice properties, though.
- ygra 13y agoThere is no UTF-21. The number after UTF signifies the size of the code unit, which in UTF-32 is 32 bits wide. That code points are only defined up to U+10FFFF is irrelevant for that; the code unit is still 32 bits.
- evincarofautumn 13y agoYeah, I was joking. UTF-32 is a waste of 11 perfectly good bits[1], so you could just as well write each code point as 21 bits (0o0000000–0o7777777), and call it UTF-21. 24 bits (3 × 0x00–0x7F) would be fine too, but 21 is the smallest possible fixed-width Unicode encoding. [1]: https://github.com/evincarofautumn/protodata https://github.com/evincarofautumn/protodata