7 ms·
A Programmer’s Introduction to Unicode
- stirner 10y agoU+0041 LATIN CAPITAL LETTER A is "A", not "a" (in the Unicode Codespace section towards the beginning).
- anonymousiam 10y agoYes, and for those who don't have the ASCII collating memorized, the lower case A ("a") is U+0061.
- andrewl 10y agoThere's also Joel Spolsky's The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!): https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
- thomk 10y agoHaha, I was going to post the same thing! If you like Joel Spolsky you might like this video where he angrily abuses everyone about how bad they are at Excel and shows you some very cool tricks: https://www.youtube.com/watch?v=0nbkaYsR94c&t=1206s https://www.youtube.com/watch?v=0nbkaYsR94c&t=1206s
- drewmate 10y agoI considered myself reasonably good at Excel prior to watching this. Thanks for posting this; I learned a lot!
- deleted 10y ago[deleted]
- slededit 10y agoFor most of that I was getting so annoyed "JUST USE A TABLE!!!", but he got there in the end. For those that don't know Joel Spolsky used to be a PM on Excel back in the early 90s.
- rspeer 10y agoCan an article be honorably retired? Joel's article was important 14 years ago, but that was a time when it was not clear that UTF-8 was going to become the default encoding of the Internet, and a time before emoji. In 2003, codepoints outside the Basic Multilingual Plane could be safely ignored unless you were a historical linguist; now, people use them every day. This article is much better at telling you relevant things (with nice data-driven visualizations!) about how Unicode is used now.
- ryandrake 10y agoThe content might be a little long in the tooth, but the exasperation about how programmers don't seem to give a shit about character encoding is, sadly, still relevant today. Unicode has been around since, what, the 90's? And UTF-8 has pretty much won the encoding war since 2008 or so. Yet, I still encounter software that fails to handle anything but ASCII, and it's 2017! Pick a random open source library, or look at a random company's software, and you're likely to find software that doesn't handle character encodings well. I think web software tends to be much better than desktop/mobile/embedded for some reason. Maybe an artifact of web guys tending to work with more recent/modern libraries and frameworks. Who's seen this before: You're getting up to speed on your new company's legacy code base. Q: "Hey, senior developer, This API takes a string as an argument and we need to display it, fine. What encoding are we using?" A: ¯\_(ツ)_/¯
- stevekemp 10y agoI don't disagree, but when you're an old-school developer it's hard to get this stuff right! I wrote a console-based mail client, and I struggled with UTF for months before it was all done, and even then there were niggles caused by libraries - for example passing a C++ string to a Lua script, and then using the `string:len()` lua function would return the number of bytes, rather than the length of the rendered string. Something I had to work around: https://github.com/lumail/lumail2/issues/277 https://github.com/lumail/lumail2/issues/277
- chrisseaton 10y ago
- nabla9 10y agoJoel's article confuses characters and code points and can be very misleading. >In Unicode, a letter maps to something called a code point which is still just a theoretical concept. This is incorrect. Letter or character (user perceived character) can be multiple code points. Code points don't have intrinsic meaning across languages. If you want to count number of letters in a text or edit text, you must work at grapheme cluster level. Absolute minimum about Unicode must at least mention grapheme clusters and user perceived characters.
- Manishearth 10y agoI wrote about this a bit in http://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning-to-unicode-code-points/ http://manishearth.github.io/blog/2017/01/14/stop-ascribing-...
- blipmusic 10y agoYes! Vietnamese and International Phonetic Alphabet says hello (among others).
- srean 10y agoIs there any reasonably popular encoding that is "hole free / complete" in the sense any sequence of bytes has a valid decoding ? I use B85 or B64 when needed but was wondering if there is a Unicode encoding that will do the job.
- deleted 10y ago[deleted]
- masklinn 10y ago> Is there any reasonably popular encoding that is "hole free / complete" in the sense any sequence of bytes has a valid decoding ? That's a property of the ISO-8859 encodings, it turns out to be pretty terrible as you have no way of distinguishing utter garbage/nonsense from actual text. > I use B85 or B64 when needed but was wondering if there is a Unicode encoding that will do the job. Base64 and 85 are pretty much the opposite of Unicode encodings. Encodings turn text (abstract unicode codepoints) into actual bytes you can store, Base64 and Base85 turn bytes into ASCII or ASCII-compatible text.
- srean 10y ago> Base64 and 85 are pretty much the opposite I know and that's the purpose I use it for: to smuggle in binary data as text, but no reason why an encoding scheme should not be dual purpose. Thanks for the tip about ISO-8859
- edblarney 10y agoIn b64, you're going to want to pick a short set of characters that can be encoded clearly and simply, i.e. ascii chars. And the shorter the better, i.e. even in 6 bits. You take the first six bits of your binary data and convert to some ascii char mapping. And then the the next 6 bits and so on. You can't do that with unicode, and it wouldn't make sense for any other encoding standard. They are completely different things.
- Moru 10y ago
- keithgabryelski 10y agoHere is my take on this subject: A Practical Guide to Character Sets and Encodings or: What’s all this about ASCII, Unicode and UTF-8? https://medium.com/@keithgabryelski/a-practical-guide-to-character-sets-and-encodings-b5362447456f#.61il3o5k2 https://medium.com/@keithgabryelski/a-practical-guide-to-cha...
- nabla9 10y ago>Character Sets: a collection of characters associated with numeric values. These pairings are called “code points”. This is very ambiguous definition and it can be very confusing. I sure there are many people who have read Joel Spolsky's Unicode intro and left confused. Using ASCII as an example is confusing because ASCII character maps into several different Unicode concepts: 1. byte 2. code point 3. encoded character 4. grapheme 5. grapheme cluster 6. abstract character 7. user perceived character Mapping from user perceived character to abstract characters is not total,injective, or surjective. Some abstract characters need more than one code point to express them. You can't split sequence of Unicode code points arbitrarily in code point boundaries, you must use grapheme clusters instead.
- pwdisswordfish 10y ago> 1. byte Actually 'code unit', which may have any number of bits, depending on the encoding. Otherwise spot on.
- keithgabryelski 10y agoit's a practical guide, not comprehensive -- for most people this is a great start, especially if they are familiar with ASCII you'll notice I didn't cover collation -- why? because explaining that would dilute process of understand UTF-x and UNICODE
- nabla9 10y agoIt's pedagogically wrong and extremely misleading. When people start with introduction like this, they end up thinking they have learned more than they actually have. I point this out because I was one of those mislead by several previous articles explaining Unicode encoding the same wrong way as you did. When I ran into trouble and asked help, everybody around me was misguided the same way. I didn't know I had to dig into manuals because everybody explained that this is how it is. Then I had to teach everyone else that we had all learned it wrong. Many people deal only with ascii or other easy western alphabets and they can work years with Unicode before they hit into trouble.
- majestic8 10y agoCan someone help me understand why prefixes used in UTF-8 jump from "0" to "110", "1110", "11110" and so on? Why is "10" missing?
- pkaye 10y ago"10" is used as a prefix for the bytes after the first. This gives it the self-synchronization property if it somehow ends up in the middle of a sequence. See the first table in this Wikipedia link: https://en.wikipedia.org/wiki/UTF-8 https://en.wikipedia.org/wiki/UTF-8
- jcranmer 10y agoAdditionally, 110xxxxx tells you that the character is two bytes, 1110xxxx three bytes, and 11110xxx four bytes, i.e., number of bytes in number = leading 1 count.
- deleted 10y ago[deleted]
- Someone 10y agohttps://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt: "Below are the guidelines that were used in defining the UCS transformation format: [...] 6) It should be possible to find the start of a character efficiently starting from an arbitrary location in a byte stream." If they used "10" as a marker for "this is the start of a two-byte sequence", it could not have been used for "this is a byte in a multi-byte sequence, but not the first one"
- nicwolff 10y agoThe goal is that by reading any byte you can tell if you are at the start of a character sequence, so we have to start each byte with some prefix – otherwise continuation bytes might sometimes look like start bytes. If we did as you suggest, we'd have to prefix continuation bytes with "111110", leaving only two bits of data in each!
- nemoniac 10y ago
- scrollaway 10y agoJust recently had to deal with Unicode line-breaking (aka word wrapping) in JS (Canvas). I love that Unicode is very nicely and strictly defining it, but god damn is it a complicated beast. http://www.unicode.org/reports/tr14/ http://www.unicode.org/reports/tr14/
- raphlinus 10y agoI think "nicely and strictly" is an overstatement here. You still have to deal with southeast Asian scripts (which require a dictionary to find line break opportunities), and then there's the whole complex regex for numeric expressions (Example 7 in the Examples section), which ICU implements. I didn't bother with that in xi-unicode (it's not clear it improves matters much), but I do want to get Thai breaking nicely. On top of that, the Unicode rules do a very poor job with things like URLs and email addresses. The Android text stack has its own layer (WordBreaker in the minikin lib) on top of that which recognizes those and implements its own rules. But TR14 is a good start, for sure.
- kps 10y agoAnyone familiar with Unicode development know why combining characters follow the base character? Prefix combining characters would have had two nice properties: [1] Keyboard dead keys could simply generate the combining character, rather than maintain internal state, and [2] it would be possible to find the end of a combining sequence without lookahead.
- Manishearth 10y agoA bunch of characters are prepended combining marks (so lookahead doesn't quite work) 0600..0605 ; Prepend # Cf [6] ARABIC NUMBER SIGN..ARABIC NUMBER MARK ABOVE 06DD ; Prepend # Cf ARABIC END OF AYAH 070F ; Prepend # Cf SYRIAC ABBREVIATION MARK 08E2 ; Prepend # Cf ARABIC DISPUTED END OF AYAH 0D4E ; Prepend # Lo MALAYALAM LETTER DOT REPH 110BD ; Prepend # Cf KAITHI NUMBER SIGN 111C2..111C3 ; Prepend # Lo [2] SHARADA SIGN JIHVAMULIYA..SHARADA SIGN UPADHMANIYA While Unicode-specced grapheme boundaries (UAX 29) do not deal with these, Indic scripts have an "infix" combining mark that takes two regular code points on either side to form a consonant cluster. (Also, Hangul is based on conjoining code points where none of the code points is really a "combiner", they're all equally "regular" characters) I suspect the actual answer to your question is "historical accident". But, the dead key argument doesn't quite hold -- I can equally ask why keyboard dead keys aren't typed after the character.
- kps 10y ago> I can equally ask why keyboard dead keys aren't typed after the character. That does have an answer: keyboards predate electronics. A mechanical typewriter key press slams a type bar into the ribbon, and on the way back, catches a mechanism that releases the carriage (which is pulled along by a spring) for one column. A dead key is very simple; it just omits the catch. A postfix dead key would be more complicated, since it would have to add an early backspace action, and harder to press, since backspacing pulls against the carriage spring.
- Manishearth 10y ago
- faragon 10y agoUTF-8 encoding could use up to 6 bytes per character, addressing up to 2^32 codes (UTF-8, 1993 version [1]). So even being currently limited to 4 bytes, it could be expanded to 6 anytime. [1] https://en.m.wikipedia.org/wiki/UTF-8 https://en.m.wikipedia.org/wiki/UTF-8
- lasthemy 10y agoIn 2003, RFC 3629 removed all 5 and 6 byte encodings, effectively limiting it to 4-bytes. Of course it could be expanded at any time, but that would be a significant change to established practice, and directly contradict the rationale in RFC 3629 (that because most people use 4 bytes in practice, allowing 5 and 6 constituted a security flaw). Source: the same Wikipedia article you linked.
- faragon 10y agoSure, that's why I pointed it could be expanded anytime, because the encoding already supports its expansion, by design :-)
- jcranmer 10y agoThe limiting factor on Unicode is UTF-16. There's only enough surrogates for 16 astral planes, which is why Unicode has 17 16-bit planes.
- faragon 10y agoUTF-16 has reserved codes as well, so it could be expanded for covering 2^32 codes, too.
- jcranmer 10y agoThe range U+D800-DFFF is reserved for UTF-16 surrogates, specifically in two pairs of low and high surrogates. That means every surrogate pair can encode 10 + 10 bits of information, which is where the 16 astral planes (4 bits of 16-bit planes) comes in. Otherwise, there are just 128 code points in unallocated blocks in the BMP. There is no space for expansion without reassigning private use areas or changing the encoding mechanism of surrogates--which is currently completely specified (each surrogate pair will produce a valid code point).
- GnarfGnarf 10y agoExcellent article! Best explanation of UTF-16 I've seen.
- alblue 10y agoYou might also like my presentation on the history of Unicode, explaining where it (and other codes like it) came from. https://www.infoq.com/presentations/unicode-history https://www.infoq.com/presentations/unicode-history
- jcranmer 10y agoA link to the Unicode Roadmap would be helpful to also understand some of how Unicode is allocated. The smallest allocatable block of Unicode is a 16 code point block of characters. In the BMP, there are just 8 such blocks remaining--and 1 of them is scheduled for use in Unicode 10 and 3 of them are tentatively reserved for a current proposal. Note however that there are ~10k unassigned code points within the BMP. After the BMP, the next most-full block is the SIP (plane 2), which is basically entirely reserved for "random rare characters in Chinese/Japanese" (although only about 85% or so of it is considered assigned as of Unicode 10). Plane 3, the TIP, is more or less reserved for SIP overflow and historical Chinese scripts, although only about 25% of it has tentative reservations. Around half of the SMP is already tentatively reserved for scripts, although I'm not sure if the Script Encoding Initiative's list of scripts to encode (http://linguistics.berkeley.edu/sei/scripts-not-encoded.html http://linguistics.berkeley.edu/sei/scripts-not-encoded.html) are all on the Unicode roadmap pages. There's about 200 scripts left to encode, although some of them may be consolidated in Unicode's script terminology (for example, Unifon is proposed for Latin Extensions D). I think the set of remaining historical scripts to encode is considered more or less complete, although several scripts definitely need a lot more research to actually encode (Mayan hieroglyphics is probably the hardest script left, since it requires rather complex layout constraints).
- chickenbane 10y agoReminds me a lot of Jesse Wilson's excellent talk, "Decoding the secrets of binary data". It's a fun video to watch on a lazy Saturday =D https://www.youtube.com/watch?v=T_p22jMZSrk https://www.youtube.com/watch?v=T_p22jMZSrk