4 ms·
The (Unfortunate) Truth About Unicode In Python
- jrockway 18y agoYeah, Unicode is really painful. One one hand, most languages have half-assed support (Python, Ruby, PHP), which makes it hard to "pick up" good unicode habits. On the other hand, programmers using the langauges that do have real support (Perl, Java) don't use it correctly. The result is that any correctly-displayed international characters in the data those programs process is merely a coincidence. That's scary. The problem is that reading UTF-8 from a web browser, then storing it in an 8-bit clean database (everything), and then displaying that binary data back to the web browser usually works OK. So people don't realize their app is completely broken until it starts behaving in a weird way, then they blame whoever is nearby. Anyway, I wish all the major languages would make misusing unicode a fatal error. That way things could never appear to work. (In Perl, you can do that with "use encoding::warnings 'fatal'", but nobody ever uses that... they just cross their fingers and hope for the best.) The really depressing part is that for every time I answer someone's unicode question on a mailing list, there are 10 other people who immediately ask the same question a week or two later. I'm not sure I understand why. It doesn't happen with other classes of questions. I guess people don't realize that other people could possibly have had the same problem, and don't Google it? Dunno. </rant>
- gizmo 18y agoThe main problem is that most programmers never experience Unicode problems themselves. 7 bits is enough for most of us. So we only start using Unicode when the pain caused by incessant nagging exceeds the pain of implementing Unicode. So we never fix the Unicode problems out of curiosity, we fix them because people make us. No wonder we never learn. The second problem is that we don't understand Unicode. Yes, we know that 7 bits isn't quite enough to represent all different characters, and that you have utf-8 and UTC, and so forth, but we really don't care enought to find out how it really works. We read some articles online, and we start using wchar_t / TCHAR instead of char, but that's basically it. Why bother with Unicode when you can simply write 'resume' instead of 'résumé'? Nobody cares. That's what it boils down to: Nobody really cares. So it doesn't get fixed. The only mystery is why people still expect Unicode support to "just happen".
- brlewis 18y agoWhat do people do wrong in Java? I wonder if I'm making the same mistake with my Java-based Scheme implementation. I never looked into Unicode deeply.
- fauigerzigerk 18y agoMost of the time the things that go wrong have little to do with individual languages. The issue with unicode is that you must be consistent across all layers of the application and there is no way to abstract it away in a general fashion. For a Java based scheme implementation the primary issue may be the internal representation of scheme strings. java.lang.String is immutable whereas scheme strings are mutable. It's probably still the cleanest route to stay with some UTF-16 representation internally. Maybe StringBuilder would work but what do you do about regular expressions? I'm glad I don't have to solve that one. You might also want to check out what the jruby guys say about this issue. I think I remember they made the switch from Java Strings to some other mutable representation, I believe UTF-8. They must be using some regex implementation that you might be able to reuse. Also look at SISC. Another issue is the source code character escaping stuff. I don't know what the scheme standard says about this but in any event you have to think about it. The last and easiest thing is to just make sure you always correctly translate back and forth when reading and writing files and network/database connections.
- gruseom 18y agoI haven't learned Unicode properly - but I will need to. What do you recommend? Is there a good book you know of, or some other way to go?
- jauco 18y agohttp://www.joelonsoftware.com/articles/Unicode.html http://www.joelonsoftware.com/articles/Unicode.html By the way, python 3000 will force every coder to at least make a conscious decision about the character encoding and will have no none-unicode strings anymore.' [update]Speaking of the devil: http://sayspy.blogspot.com/2008/07/trying-to-make-sense-of-unicode-terms.html http://sayspy.blogspot.com/2008/07/trying-to-make-sense-of-u...
- andrewf 18y agoThe problem is that reading UTF-8 from a web browser, then storing it in an 8-bit clean database (everything), and then displaying that binary data back to the web browser usually works OK. That sounds like something you could fix relatively easily. In my experience this does happen, but it's not with UTF-8, it's with Windows-1252.