2 ms·
This is way too much to remember. The key to handling Unicode correctly is to think about the data that comes into your program and the data that leaves your p
by jrockway 15y ago
This is way too much to remember.
The key to handling Unicode correctly is to think about the data that comes into your program and the data that leaves your program. If it's text, it has some sort of encoding, and you need to decode those encoded characters before you can use them as text in your program. Similarly, you can't just leak Perl's internal representation of characters to the world: you have to explicitly encode the characters to octets.
The problem that people run into is not knowing that you have to do this (in every programming language), and then deciding which encoding to use. In some cases, it's easy: HTTP includes the encoding in the headers, so all you have to do is get the request, decode according to the headers, and you're set. Similar, when you send an HTTP response to someone, just set the headers, encode the characters, and everything ends up 100% correct.
The problem comes when poor design causes you to assume the character encoding. What encoding are file names on a random removable disk? What encoding is this text file in? What encoding are those database rows? If you don't know, you simply can't process that data correctly as text. But most people assume "something will magically decide for me and it will all work out". Nope, it won't. Don't rely on magic: be explicit.
The reason why things work most of the time is because you treat the data as binary: opaque, meaningless octets, that don't have semantics like "make the first letter capital". This will work if your world is entirely UTF-8-encoded: UTF-8 filenames, UTF-8 files, UTF-8 source code, UTF-8 database results, etc.
The reason why people run into trouble with Perl and not with other languages, is because Perl assumes that when you treat binary data as text, you really have Latin-1 text. It then says, "hey, this is text", upgrades it to Unicode, and then transforms it as such. When your text was Latin-1 (the backcompat case), it all works. When it's UTF-8, though, then you get double encoding: the dreaded "æ¥æ¬èª" instead of "日本語". But if you just tell perl, via Encode::decode_utf8, it will know that you actually have unicode text and not latin-1 text, and everything will work!
Anyway, the solution is to always specify the encoding via Encode::decode_utf8($str) or Encode::decode('my-encoding', $str) when reading data, and to Encode::encode_utf8 or Encode::encode when outputting data (even to the terminal!). The hardest part is making sure your libraries do this (DBD::SQLite does, Catalyst does, LWP does), and making sure that your libraries have enough information to do it right. HTTP is easy, there's a header. But that blob in your database is not easy, and you may have to handle it yourself... because the information about the encoding exists only in your brain, not anywhere the computer can find it.
(Oh, and this only gets you to the "I'm not trashing any information"
stage. If you want your Japanese text to sort あ い う え お instead
of in codepoint order, that's going to require a module. Even simpler
cases will require you to learn about collation and normalization.)
Edit: distilled the key points I raise here into an SO answer: http://stackoverflow.com/questions/6162484/why-does-modern-perl-avoid-utf-8-by-default/6192088#6192088 http://stackoverflow.com/questions/6162484/why-does-modern-p...