6 ms·
How To Use UTF-8 Throughout Your Web Stack
- jrockway 15y agoHe's missing the most important part: ensuring that your application code is treating the text as text, not as an octet stream. This varies by language, but typically the code is something like "text = decode('utf8', binary)" when your application first sees data from the wire (or files, or a URI string, etc.), and "binary = encode('utf8', text)" when the data leaves your program, like to a log file or the terminal or a socket. I say "binary" and "text" because the Internet cannot transmit text, it can only transmit "binary" octet streams. (Similarly, UNIX files can only store octets, and UNIX file names can only store octets other than / and NUL.) But, your programming language supports both text manipulation and binary manipulation, so you have to tell it how you want to treat the data. Each language is different; Perl treats everything as Latin-1 text by default (which happens to work nicely for binary, as well, but not so nicely for UTF-8-encoded text). Often, libraries will handle this for you, since they have access to out-of-band information. If your locale is en_US.UTF-8, filenames can be assumed to be UTF-8-encoded. If the HTTP response's content-type says "charset=utf-8", your HTTP library will know to decode the octet stream into text for you. But it's important that you both test this and find the code that does it for you, because sometimes library authors forget or libraries have bugs, and one bug will ruin your whole operation. Handling Unicode text is hard because it's a rare case where you have to get everything right or the results of your program will be undefined. And, there are no "reasonable defaults", so you have to be explicit about everything. Finally, you can't guess about what encoding your data is; all binary data must come with an encoding out-of-band, or your program will break horribly. Proper text manipulation is the ultimate test of "can I write correct software", and it isn't easy.
- zokier 15y agoI agree on most of your points, but disagree on that guessing encoding should not be done. I think that it conflicts with basic robustness principle "be conservative in what you do, be liberal in what you accept from others".
- sid0 15y agoI personally think being liberal in what you accept from others is the second worst evil in computer science. The worst being null, of course.
- guard-of-terra 15y agoThen you will also personally produce programs which would be broken for 5/6ths of the world population who happen to use letters outside latin1. There's no way to avoid it unless you wrap it up and add some explicit checks and guesses.
- tlrobinson 15y agoWon't all modern browsers include the encoding in the Content-Type header? They should. If so there's no need to guess.
- guard-of-terra 15y agoIt's not just browsers. Browsers are pretty sane when it comes to charsets, because they had the time to make it right and the pressure to do so. (It wasn't like that in times of NN4/IE4, which would interpret your text as whatever they want and won't even let you override) Facing to something less agamant (like dreaded id3 tags), no such luck.
- hnbla 15y agoAgree, this is why html ended up being the mess it is today. Because every browser wanted it their way and then every other browser and website has to implement things both their own way and everybody elses to not break the Internet. Now you've got 10 versions to support instead of one!
- jrockway 15y agoHey now. If software started validating its input, what would virus writers do for a living?
- 15y ago
- justincormack 15y agoActually utf8 in MySql is 16 bit utf8 ie compatible with the obsolete UCS2 and you should be using utf8mb4 for real utf8. http://dev.mysql.com/doc/refman/5.6/en/charset-unicode.html http://dev.mysql.com/doc/refman/5.6/en/charset-unicode.html
- cpeterso 15y agoThanks for the warning! I know the MySQL developers have a huge challenge to maintain compatibility with existing data and clients, but it almost seems like they're trying to make Unicode difficult to use correctly when they: default to Latin1 encoding; call it something accurate yet surprising like "latin1_swedish_ci"; silently convert and corrupt data if the client connection specifies a different encoding; add a Unicode encoding called "utf8" but don't support everything that other UTF-8 does; and finally add real UTF-8 support but call it something different like "utf8mb4".
- blauwbilgorgel 15y agoI do use UTF-8 for my entire web stacks, but I encode characters in their special characters: Iñtërnâtiônàlizætiøn Becomes: Iñtërnâtiônàlizætiøn When outputted to the browser. Can anyone tell me if that is correct and sane? Translating characters like & and € to & and € saves me a lot of hassle with validation: http://validator.w3.org/check?verbose=1&uri=http%3A%2F%2Frentzsch.tumblr.com%2Fpost%2F9133498042%2Fhowto-use-utf-8-throughout-your-web-stack http://validator.w3.org/check?verbose=1&uri=http%3A%2F%2... Line 265, Column 190: non SGML character number 157 <code>“Iñtërnâtiônà lizætiønâ€</code> You have used an illegal character in your text. HTML uses the standard UNICODE Consortium character repertoire, and it leaves undefined (among others) 65 character codes (0 to 31 inclusive and 127 to 159 inclusive) that are sometimes used for typographical quote marks and similar in proprietary character sets. The validator has found one of these undefined characters in your document. The character may appear on your browser as a curly quote, or a trademark symbol, or some other fancy glyph; on a different computer, however, it will likely appear as a completely different character, or nothing at all. Your best bet is to replace the character with the nearest equivalent ASCII character, or to use an appropriate character entity. For more information on Character Encoding on the web, see Alan Flavell's excellent HTML Character Set Issues reference. This error can also be triggered by formatting characters embedded in documents by some word processors. If you use a word processor to edit your HTML documents, be sure to use the "Save as ASCII" or similar command to save the document without formatting information. Line 344, Column 79: cannot generate system identifier for general entity "src" An entity reference was found in the document, but there is no reference by that name defined. Often this is caused by misspelling the reference name, unencoded ampersands, or by leaving off the trailing semicolon (;). The most common cause of this error is unencoded ampersands in URLs as described by the WDG in "Ampersands in URLs".
- justincormack 15y agoSure that works for some things but 1. You know there are no entity names for most unicode characters? Eg Chinese. You may as well use the numeric entity codes 2. In XML the entity names other than lt gt amp and quot are not defined unless you have a dtd, so you should not use them across an xml api, eg for an Atom feed, or across an xml web service, unless it defines a dtd including them which is unlikely. 3. If you get those errors, it is because you have something set up wrong. Those things are fixable. Fixing them will help you understand whats going on better. As the article says, get out your hex...
- zaphoyd 15y agoIt is probably worth including a character outside the basic multilingual plane (e.g. anything above 0x10000, like http://unicodelookup.com/#0x22222/1 http://unicodelookup.com/#0x22222/1) when testing UTF-8 web support. I recently was working on a Japanese teaching web application that needed such characters and sadly learned that MySQL versions before 5.5 do not support UTF-8 characters outside of BMP (anything that needs more than 4 UTF-8 octets) and text to image drawing library support was also sketchy.
- wnoise 15y ago4 octets of UTF-8 suffice to cover all Unicode characters. Unicode is essentially 21-bit (U+0 to U+10FFFF), not 32-bit. The BMP is 16 bits, U+0 to U+FFFF. 3 octets suffice for it. It's useful to know that MySQL support outside the BMP doesn't work, but I would guess it's a generic problem affecting all Unicode support, not restricted to UTF-8. (Yes, UTF-8 was defined to go up to 6 octets and cover 31 bits. As used with Unicode, only up to 4 are supposed to be used...)
- zaphoyd 15y agoYes you are right on the 3 vs 4 octets for outside BMP, it is the 4 octet UTF8 that MySQL pre 5.5 doesnt work with. With MySQL 5.5 the full basic LAMP stack at least will now handle non-BMP characters.
- wnoise 15y ago(My guess is wrong. They really did have a hardcoded 3 octet pseudo-characterset. Ugh.)
- unconed 15y agoAside from the curly quotes, his "unicode canary" still only uses Latin-1 characters. Dumb. At least throw some latin-2 and/or Cyrillic/Greek in there.
- grok1fy 15y agoHere's a modified version with Cyrillic, Greek and Polish (ISO-8859-2) letters, along with a few more for good measure. ǏñҭëՐŋâťḯṏṋαłїżæŧíøṅ
- throw_away 15y agoa truly masochistic canary would include an old ASCII control character (x00-x32) some of which I don't believe encode properly into XML as either raw bytes or literal entities and also a character off the basic multilingual plane like say linear b
- deleted 15y ago[deleted]
- mcpherrinm 15y agoEven better: Include characters outside the Basic Multilingual Plane. That catches problems with UTF-16 that might exist along the way.
- pornel 15y agoEven trickier: And add combining characters. You can have more code points than characters, and letters-only string that contains codepoints that are not letters. Unicode is truly evil at the edges.
- pkaler 15y agoThis is probably a good time to re-link to "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets" by Joel. http://www.joelonsoftware.com/articles/Unicode.html http://www.joelonsoftware.com/articles/Unicode.html
- cpeterso 15y agoMySQL seems to be the cause of many of the web's character encoding problems; search for keywords like "mysql utf8 hell" and you will get many hits. Are the MySQL developer doing anything to improve the situation? Here are some possible improvements of the top of my head. I won't call them solutions. <:) 1. Use UTF-8 internally, converting based on the client's encoding. 2. Use UTF-8 internally and force client's to do their own conversions. 3. Tag all string data with its encoding type for run-time checks. 4. At the very least, default to UTF-8 rather than latin1-swedish-ci!
- Joakal 15y agoThere's many utf8 collations so you should specify something specific to default to. I recently read to use utf8_unicode_ci [0] is the best. [0] http://philsturgeon.co.uk/blog/2009/08/UTF-8-support-for-CodeIgniter http://philsturgeon.co.uk/blog/2009/08/UTF-8-support-for-Cod...
- troutwine 15y agoThis post got me wondering: what would a character set not requiring backward compatibility with ASCII look like? I tried to put the question to the StackOverflow folks, but it was apparently off topic: http://stackoverflow.com/questions/7128748/if-ascii-compatibility-were-not-an-issue-what-would-an-ideal-character-encoding Thoughts on a non-ASCII encumbered character set? What can be improved over UTF-8?
- thristian 15y agoThe obvious 'native' format for Unicode data is UTF-32, which forgoes ASCII-compatibility and gets you regular, consistent sizing. Unfortunately, Unicode itself is an ASCII-encumbered character set (or at least Latin-1-encumbered, and Latin-1 is ASCII-encumbered). Pretty much all the contents of Unicode are there because somebody needed them¹, so I imagine a non-ASCII-encumbered Universal Character Set would still contain all the ASCII characters... somewhere... but they'd likely be shuffled around a bit from the Unicode we know and love. ¹: I believe Unicode encodes some meaningless Kanji/Hanzi glyphs that were created by accidentally confusing two other, genuine glyphs; I'm pretty sure it only does so because it inherited them from legacy pre-Unicode encodings.
- guard-of-terra 15y agoIs it UTF-32LE or UTF-32BE? The good thing about UTF-8 is that you don't have to choose.
- dustingetz 15y agowhat would the web look like if we didn't care about backwards compat? ...who cares?
- troutwine 15y agoHow terribly unconstructive of you. 1. Sarcasm is not wit. 2. Dismissal by analogy is no dismissal at all. 3. The polite individual addresses well intentioned questions in good faith or not at all. Now, I've asked my question because, in a community of peers of various stripe, it's entirely possible that there will be members more versed in character encodings than myself and that there are, indeed, more optimal ways to encode characters which are not taken for need to maintain backwards compatibility with the ASCII character set. One poster mentioned UTF-32 as a non-backwards compatible example which, while space inefficient, is a good-faith answer to my question. As to your clumsy dismisal by analogy question, here are some things off the top of my head: * Substitute byte-code standards for browser javascript. * Specify parsing, rendering semantics from the start for all markup/presentation languages. * Automated browser compliance testing from the start for all standards. * Effective client-side user storage. * Include stateful communication channels from the word go. Innovation in a field of endeavor only occurs by examining the assumptions of the field and invalidating them as the context of their being changes. Those that don't care to do so are absolute hacks, stirring the sewer's mirk in the hopes that a shiny bobble will ocassional bubble up from the depths.
- dangrossman 15y agoRather than add --default-character-set every time you invoke the mysql client, add the setting to your my.cnf/my.ini config file.
- getsat 15y agoJust set the client and server defaults in my.cnf. MySQL has four places where encoding can be set: client connection, server, database, column. Setting the client and server in my.cnf covers everything.
- pilif 15y agoI don't want to gloat, but when I dump my database in postgres and then restore the dump, I don't have to concern myself with character sets and --default-encoding parameters because the dump will contain a "set encoding" statement that corresponds to the files content. Aside of that, IMHO, the database is the least problematic part of the chain as long as you tell the client library what character set the incoming data will be in. It then should transcode automatically if needed. One additional thing: I once witnessed MySQL silently truncating Latin data I accidentally tried to store in an utf-8 table, so you might to be a bit careful. Usually you should just get an error if you tell the database that your data is utf, but it isn't (http://pilif.github.com/2008/02/failing-silently-is-bad/ http://pilif.github.com/2008/02/failing-silently-is-bad/) Lastly, IMHO, the biggest issue is, as usual, the browser: to this day it's possible to have IE submit data in ISO-* (depending on the users locale) despite clearly stating to only accept utf. Be mindful of this and fix the encoding if you can (or have the database blow up - see above)