3 ms·
The main trick is to remember is, to get utf-8, you have to encode, not decode. When I didn't know how unicode works that well, I mistakenly assumed unicode is
by ludwigvan 16y ago
The main trick is to remember is, to get utf-8, you have to encode, not decode. When I didn't know how unicode works that well, I mistakenly assumed unicode is coded, so I have to decode it. Internally, unicode is of course coded (that is represented by some coding), but you don't have to know it. As the slides say, just encode to utf8 when you print text out, or display it in a web browser etc; and you will be fine.
One other unicode library I can recommend is icu, Python bindings to IBM's ICU. It solved some Turkish specific problems I have. (Turkish alphabet has I,İ, ı, i which makes upper-lower case conversions tricky. For example, mayıs becomes MAYIS when capitalized while ENGLISH becomes englısh when lowercased.)
Read http://en.wikipedia.org/wiki/Turkish_dotted_and_dotless_I http://en.wikipedia.org/wiki/Turkish_dotted_and_dotless_I , http://www.joelonsoftware.com/articles/Unicode.html http://www.joelonsoftware.com/articles/Unicode.html and http://www.codinghorror.com/blog/2008/03/whats-wrong-with-turkey.html http://www.codinghorror.com/blog/2008/03/whats-wrong-with-tu... for more info.
Another issue, if you are doing web dev is, how to represent non-ascii characters in urls. One choice is url encoding, the other is slugifying the url, that is, choosing an ascii equivalent for the non-ascii character. For example, Django has a slugify function that helps with this. It converts, for example, über to uber.
- zepolen 16y ago>It converts, for example, über to uber. What does it convert 'ぬびばざべ' into?
- admp 16y agoICU transliterator is fairly sophisticated (and configurable), actually. Take a look at the examples: キャンパス -> kyanpasu Αλφαβητικός Κατάλογος -> Alphabētikós Katálogos биологическом -> biologichyeskom From: http://userguide.icu-project.org/transforms/general http://userguide.icu-project.org/transforms/general Edit: reread, OP was talking about Django transliteration, which is much simpler.
- ludwigvan 16y agoYes, Django slugifier turns ぬびばざべ to an empty text. Maybe that's because my language is set to tr-tr, I am not sure. One possible downside with icu is that, afaik, you can't run it in Google App Engine.