10 ms·
We should be grateful to UTF-8 for saving us from fixed multibyte Internet (so persistently pushed by MS, IBM and others), which would have made little sense in
by mojuba 13y ago
We should be grateful to UTF-8 for saving us from fixed multibyte Internet (so persistently pushed by MS, IBM and others), which would have made little sense in this predominantly 8-bit world. I remember someone saying at the time: the future is 16-bit modems and floppy disks anyway, so why not switch to Unicode now? Somehow that sounded absurd to me.
Anyway, 20 years later, hardware is still mostly 8-bit, and basically nobody cares about Unicode apart from font designers and the Unicode Consortium.
(On a side note, UTF-8 as a hack is a distant relative to Huffman encoding, itself a beautiful thing.)
- Stratoscope 13y ago"16 bits should be enough for anybody." --Anonymous The funny thing is that 16 bits wasn't enough. We thought it would be, back then. Just use 16 bits for each character and we can return to the joy and simplicity of a fixed width encoding. That didn't work out. UCS-2 was a fixed width 16-bit encoding, but it it had to be extended into the variable width UTF-16 to get more code points. You get the bloat of 16-bit data, incompatibility with ASCII, and variable width characters too. You have to go all the way to UTF-32 to get back to a fixed width encoding. So UTF-8 really was a smart move. Bite the bullet on the variable width, but make it so you can always find the beginning of the next character even if you start in the middle of one, and make it backward compatible with ASCII too. Genius. https://en.wikipedia.org/wiki/UTF-8 https://en.wikipedia.org/wiki/UTF-8 https://en.wikipedia.org/wiki/UTF-16 https://en.wikipedia.org/wiki/UTF-16 https://en.wikipedia.org/wiki/UTF-32 https://en.wikipedia.org/wiki/UTF-32
- sopooneo 13y agoPlease correct me if I'm wrong, but you don't really get fixed width in anyway do you? Because unicode itself is not fixed width. There are situations where one character on the screen is represented by two consecutive unicode code points.
- plorkyeran 13y agoOr a lot more than two, or even multiple characters for a single code point. UTF-32 appears to simplify things a lot, but in practice O(1) indexing of code points and simple iteration over code points is not actually all that useful, because code points are not characters.
- runn1ng 13y agoUTF8 != unicode. Unicode is a standard, that maps the giant number of letters to unique numbers. UTF8 (and UTF16, UTF32) are various encodings of that. Unicode itself is not an encoding; unicode just says "letter 'č' is 269"; UTF-8 says "We will represent it as two bytes, first is C4, second is 8D". UTF8 is indeed variable width; UTF16 and UTF32 are, AFAIK, fixed width. Both use unicode standard.
- pjscott 13y agoWhat the comment you're replying to is saying that glyphs may be represented by more than one code point. So sure, UTF32 has a fixed width for each code point, but it's not quite so clear how useful that is.
- runn1ng 13y agoOh, in that case I misunderstood parent. Sorry about that.
- Stratoscope 13y agoEven if we're talking about code points and not characters, UTF-16 is variable width. UTF-8 is variable width using 8, 16, 24, or 32 bits per code point. UCS-2 is fixed width using 16 bits per code point. UTF-16 is variable width using 16 or 32 bits per code point. It's an extension of the fixed width UCS-2 format: every valid 16-bit UCS-2 code point is also a valid UTF-16 code point, but UTF-16 also includes 32-bit code points. UTF-32 is fixed width using 32 bits per code point.
- alister 13y agoUTF-8 is variable width using 8, 16, 24, or 32 bits per code point. I wondered why you didn't write that "UTF-8 is variable width using 8, 16, 24, 32, 40, or 48 bits per code point", because that's how Prosser/Thompson/Pike's UTF-8 idea was proposed, as shown in the "most elegant hack" article. And here's the answer from Wikipedia: In November 2003 UTF-8 was restricted by RFC 3629 to end at U+10FFFF, in order to match the constraints of the UTF-16 character encoding. This removed all 5- and 6-byte sequences, and about half of the 4-byte sequences. Even though they eliminated the 5th and 6th bytes, UTF-8 can still represent all of Unicode's 1,114,112 code points. I'm mentioning this because others probably are wondering the same thing.
- duaneb 13y agoCharacters are not fixed width, no, but codepoints are. The idea of "what exactly constitutes a character" gets pretty fuzzy in some cases anyway with utility codepoints.
- jameshart 13y agoEven Windows-format ASCII isn't fixed width - CR-LF is a two-byte 'codepoint'...
- nitrogen 13y agoYou can think of it as two separate instructions, CR to move horizontally to the beginning of the line, and LF to move down by one line.
- zorked 13y agoNot really because a single CR or LF does not do a real CR or LF.
- prewett 13y agoIt used to, back in The Day, when you had hard-copy consoles or even dot matrix printers. I think sending CR (but not LF) and then printing accents over top of the original line was a way of printing some European languages with just ASCII.
- barrkel 13y ago$ man terminfo | grep cursor_down | head -1 cursor_down cud1 do down one line $ tput cud1 | hexdump -C | cut -f 1-20 | head -1 00000000 0a |.| $ man terminfo | grep carriage_return | head -1 carriage_return cr cr carriage return (P*) $ tput cr | hexdump -C | cut -f 1-20 | head -1 00000000 0d |.| If you ever write a bash script that does more advanced things with the terminal, you'll get familiar with tput, termcap and terminfo pretty quickly.
- kps 13y agoIt does in ASCII, and (consequently) in any ANSI-conforming terminal [emulator].
- jameshart 13y ago
- ygra 13y agoThere are multiple factors at play here and it ultimately comes down to what you need: If you to access code points, then UTF-32 gives you a nice fixed-width (albeit quite large) encoding to work with. Some applications or libraries may use UTF-32 internally to speed things up. If you deal with things on screen or what humans perceive as character (graphemes) then there is nothing really that can help you at the encoding level because graphemes can be arbitrarily long with regard to code points: e̬̱ͤ̈́̿̂̅́͟x̰̞̻̼̻̼͉̰ͫͧ̑͂͊͜ͅa̶̳͖͔̞̰͈̯͓͆̓̎̉̏̎̚m̛̮̩͙ͦ͗͆̄̋́̄p͎̥̠͈̮͚̾ͅl̛̗̈̄e̴̞̫͖̝̘̋̑̄̎ͥͬ̾ (this probably doesn't even render properly). Sadly, most APIs use either UTF-8 or UTF-16 and consider code units the most important entity, not code points. And that's inexcusable and frankly a mess, because I can count the number of times I needed to access a code unit instead of a code point on zero hands.
- Flimm 13y agoPlease define what you mean by code point and code unit.
- ygra 13y agoI mean the same thing that Unicode uses these terms for. Code Unit. The minimal bit combination that can represent a unit of encoded text for processing or interchange. The Unicode Standard uses 8-bit code units in the UTF-8 encoding form, 16-bit code units in the UTF-16 encoding form, and 32-bit code units in the UTF-32 encoding form. Code Point. (1) Any value in the Unicode codespace; that is, the range of integers from 0 to 10FFFF16. (See definition D10 in Section 3.4, Characters and Encoding.) Not all code points are assigned to encoded characters. See code point type. (2) A value, or position, for a character, in any coded character set. From http://www.unicode.org/glossary/ http://www.unicode.org/glossary/
- iv_08 13y agoFor index based operations code units are more useful than code points. a = s.indexOf("e̬̱ͤ̈́̿̂̅́͟") b = s.indexOf("m̛̮̩͙ͦ͗͆̄̋́̄") s[a:b] This operation is fast with code unit indices and slow with code point indices.
- evincarofautumn 13y ago> You have to go all the way to UTF-32 to get back to a fixed width encoding. UTF-21, actually, as there are no code points above 0x10FFFF. Powers of two have other nice properties, though.
- ygra 13y agoThere is no UTF-21. The number after UTF signifies the size of the code unit, which in UTF-32 is 32 bits wide. That code points are only defined up to U+10FFFF is irrelevant for that; the code unit is still 32 bits.
- evincarofautumn 13y agoYeah, I was joking. UTF-32 is a waste of 11 perfectly good bits[1], so you could just as well write each code point as 21 bits (0o0000000–0o7777777), and call it UTF-21. 24 bits (3 × 0x00–0x7F) would be fine too, but 21 is the smallest possible fixed-width Unicode encoding. [1]: https://github.com/evincarofautumn/protodata https://github.com/evincarofautumn/protodata
- asveikau 13y ago> so persistently pushed by MS People still flame how MS uses 16-bit characters, which IMO is a bit of an unfair criticism. (Full disclosure: I used to work for them. But I was never a koolaid drinker.) I don't think they have any religious aversion to UTF-8, it's just history. By adopting Unicode as early as they did, they made this decision before UTF-8 existed and they stuck with it. Probably the bigger crime is not switching to UTF-8 as the default "ANSI"/"multi-byte" codepage (to use the Windows terms). This means C programmers who are not Windows experts often end up writing non-Unicode-safe software because they expect every string to be char*. (Also as has been mentioned UTF-16 is not fixed. And even in UTF-32 there are instances where a single glyph takes multiple code points - the decomposed accent marks are the one example I know, possibly there are others?)
- erichurkman 13y agoI still remember the day that the "View > Encoding" menu of browsers was absolutely critical in browsing the web if you browsed enough sites that were outside of your normal locale (your default codepage in Windows). The menu is still there, at least in Firefox & Chrome, but I haven't used it in a few years, as most things Just Work™ for modern websites, largely thanks to UTF-8.
- gsnedders 13y agoThe eventual browser default is still (sadly) browser locale dependent, so the menu exists for sites designed for a different locale. I and some others still hope one day we can get rid of that (likely replaced with per domain, primarily TLDs, defaults).
- deathanatos 13y agoIn IE, you mean? A standards-compliant browser shouldn't need to guess at the encoding. It can be found in the <meta> tag or (if that wasn't supplied or we're not dealing with HTML) in the Content-Type header. If it isn't found there, I believe HTML says, "it's ISO-8859-1 (latin1)". That said, certain browsers have been known to guess.
- contingencies 13y agoAnyway, 20 years later, hardware is still mostly 8-bit, and basically nobody cares about Unicode apart from font designers and the Unicode Consortium. Err, failure to parse sarcasm? Most people on most networks use it daily. On mobile networks, it's used for tens of thousands of messages per hour as SMS messages in almost every country with non-Romanized scripts uses UCS-2 encoding. I've had to write a raw SMS PDU generator for a whole bunch of human languages, including right to left ones like Farsi and Arabic, or Chinese, Japanese, Korean and Thai. Furthermore, Unicode is now used by everyone in China, having finally mostly migrated from GB encoding. That means "most of the internet uses it daily". I guess your point may have been sarcastic, because the above is obvious to me. Either that, or you have a really bad case of a merry centrism.
- ufmace 13y agoI think he means "nobody cares about Unicode" in the sense that nobody cares about the technology itself. The guy in Japan probably loves that he can make a website in Japanese and have it look right on any system in the world. He probably loves even more that this just works without him having to know or care about encoding systems. Even most programmers using modern languages and frameworks can get away with not paying much attention to encodings and still probably have good support for international characters.
- contingencies 13y agoOK, thanks. So you've translated that "nobody cares" apparently means "everyone uses it and it works really well". And that's supposed to be a point? God, I must be getting old. (Edit: I am not getting frustrated with you, rather my incapacity to grok the original post.)
- jlgreco 13y agoI don't know why you are getting frustrated with him, he just (correctly, as far as I can tell) interpreted the above comment for you.
- mercurial 13y ago
- ww520 13y agoAnd thanks to Java popularizing UTF-8. Java was the major language to support Unicode natively. The lack of Unicode support in C++ was disappointing. (Yes, UTF-8 hack is nothing compared to Huffman coding. The Deflate compression algorithm using Huffman coding to encode the LZ77 output and the Huffman encoding table itself is another notch up on the hacking coolness scale.)
- sanderjd 13y agoIs this right? Doesn't Java use UTF-16 internally?
- rspeer 13y agoAnd Java's first attempt to implement UTF-8, the one they called "UTF8", is not UTF-8. It's almost but not quite CESU-8. It will probably fuck up your astral characters (like emoji). The real Java UTF-8 is, intuitively, named "AL32UTF8".
- davidgerard 13y agoAnd then there's the unremitting horror that MySQL calls UTF-8, properly known as WTF-8.
- rspeer 13y agoI'd want to save the name "WTF-8" for that case where someone takes some text encoded as UTF-8, decodes it as if it's in a single-byte encoding, and then encodes the resulting mojibake as UTF-8 again. Double UTF-8. WTF-8.
- shadowmatter 13y agoFrom http://docs.oracle.com/javase/7/docs/api/java/nio/charset/Charset.html http://docs.oracle.com/javase/7/docs/api/java/nio/charset/Ch...: "The native character encoding of the Java programming language is UTF-16." That said, at least they adopted a Unicode encoding, and helped it establish mindshare.
- statictype 13y agoand basically nobody cares about Unicode apart from font designers and the Unicode Consortium. If by nobody you mean, pretty much the entire world east of New York.
- stephencanon 13y agoJust the other day I was having a good laugh with the carpenters building my house here in New Hampshire at the expense of the rubes in Maryland who still only use ASCII.
- Patrick_Devine 13y agoI love that UTF-8 kind of followed in the tradition already set by the guys who came up with ASCII. ASCII itself is brilliant in that they aligned characters to be easy to test. Take upper and lower case 'A' which are 0x41 and 0x61 respectively: 0x41 0100 0001 0x61 0110 0001 So if you want to test whether a character is uppercase or not, you can just & 0x20. Another one is how the number digits got lined up, which start at 0x30 for 0 to 0x39 for 9, which look like: 0x30 0011 0000 0x31 0011 0001 0x32 0011 0010 etc. You can just read off the first nibble to convert from ASCII to an int which is pretty cool.
- jrochkind1 13y agoNobody cares about unicode? I couldn't write web software that deals with non-ascii, and especially non-Latin, alphabets without paying attention to Unicode. When I start getting into weird edge and corner issues -- I am continually amazed that Unicode works as well as it does. It turns out modeling alphabets is way more complicated than you might think. And Unicode gets an amazing amount right, making it practical to do things robustly correct. Including not just the character set and encodings, but things like Unicode collation algorithms. A bunch of really smart people have worked through a bunch of different non-trivial problems of dealing with diverse alphabets, and provided solutions you can use that pretty much just work, bug-free.
- mistercow 13y agoThese days, it wouldn't make much difference in terms of bandwidth, since everyone gzips everything on the server side.