5 ms·

I love to develop with perl, but unicode-related stuff was always frustrating for me. Probably because I deal with it so rarely and everytime I have to look doc

I love to develop with perl, but unicode-related stuff was always frustrating for me. Probably because I deal with it so rarely and everytime I have to look documentation to find out how to read unicode file, run regexps on unicode strings, etc. I am wondering, how other major languages deal with unicode? Is there language which work with unicode files/text transparently without /me involving into setting something differently from defaults?

draegtun 15y ago
Perl comes out best according to Tom Christiansen (author of this Perl Unicode Cookook post). See his Unicode Support Shootout - The Good, the Bad, & the (mostly) Ugly presentation at OSCON 2011 where he gave a comparison of Unicode handling in Javascript, PHP, Go, Python, Ruby, Java, and Perl. ref: http://training.perl.com/OSCON2011/index.html http://training.perl.com/OSCON2011/index.html
- joeyh 15y ago
  Perl is in the lower tier IMHO. Below Python and Haskell. Because while it has a good core unicode support, every single library makes different choices, so your code constantly needs to encode/decode. Javascript seems to avoid encoding issues entirely, at least in the browser, though Tom has some nice points about it not full supporting unicode. <http://kitenet.net/~joey/blog/entry/unicode_ate_my_homework/&#62 http://kitenet.net/~joey/blog/entry/unicode_ate_my_homework/...;
  - ajross 15y ago
    Can you be more specific? Perl's utf8 support is IMHO the cleanest and most transparent. Generally libraries don't have to care at all about the encoding of the stuff they deal with. I'd be curious what trouble you've had. (It's true that early versions of "use utf8" were horribly broken, munging strings silently at I/O time, but that was 10 years ago or more)
    - joeyh 15y ago
      Nearly every XS library takes input with utf-8 encoded to bytes, so you have to manually juggle the encoding when using it. There is no documentation stating what encoding any library expects. I just checked a source tree and found 41 calls to encode_utf8 and 42 calls to decode_utf8 to deal with this across dozens of libraries.
      - ajross 15y ago
        Um... of course they do. XS is a translation library and the domain you're translating to (C) has no (well, no sane) native encoding handling. So you pass byte buffers. That's not a ding on perl though, it's an inherent impedance you get when you cross interface domains. JNI libraries tend to have most of the same problems, either forcing the user to do the UTF16/UTF8 conversion or doing it in the library. I guess I'm surprised that Python and Haskell don't have equivalent messes when dealing with native libraries. Are you sure they don't?
        joeyh 15y ago
        Perl XS libraries often provide a full object oriented wrapper around the native C calls. They always have, at a minimum, a thin perl wrapper function. There's plenty of scope for the encoding to be done automatically, but the current state is that it's not even done manually by the libraries. Haskell's equivilant FFI interfaces do encoding conversions, so that if they return a String, it's a sequence of unicode code points, as the Haskell standard requires. Those that return raw data will use an appropriate ByteString type. Haskell didn't used to do encoding conversions for FilePaths in the FFI, but this has been fixed in version 7.4, and is even done in a way that doesn't require unix filenames be utf-8 encoded -- other encodings will pass through unchanged.
        chromatic 15y ago
        Encoding conversions at the FFI layer would be very nice. How do the Haskell interfaces know which encodings to expect?
  - gizzlon 15y ago
    "every time you call a library or do IO you have to manually deal with conversions.." (from the blog post) This is certainly not true for libraries; Only done that to workaround faults in other "systems"[1]. For IO, you don't have to "manually deal with conversion". But you do have to, if you want to do it correctly, tell it what encodings the files you're going to read are. But you do this once for all IO. There's a reason for this, and it has nothing to do with the language: You can't detect encodings. So, you either standardize on one encoding to rule them all or supply it when doing IO. To be honest, I seldom do it the "correct" way, and it works fine most of the time. [1] E.g., the DB table had one encoding, but the data was actually stored using another one. Edit: Just for the record, dealing with this can be a pain in Perl. Keeping things backwards-compatible means everything is not as streamlined as it could be. But at least it works (I'm looking at you PHP ;)
  - draegtun 15y ago
    As someone who munges Unicode data on (a far too!) regular basis I have never found Perl wanting and believe it to be the best tool for the job.
- jbjohns 15y ago
  >Perl comes out best according to Tom Christiansen It would be hard to find a more bias source.
  - chromatic 15y ago
    It would be exceedingly difficult to find someone more fastidious and willing to perform exhaustive research. If Tom has a bias, it's because he knows of that which he speaks.
gregheo 15y ago
Perl is definitely bipolar in this respect -- I've never had an easier time of dealing with Unicode issues than in Perl, but I've also never cursed Unicode so much while dealing with it in Perl.
- prakashk 15y ago
  Could you say in what other languages you dealt with Unicode?
- gauravk92 15y ago
  Python has native support for unicode strings, not Unicode support in native string types. Sigh, this is where it gets into bs world, because you have to figure out how to convert back and forth appropriately. Just thought I'd share my recent frustration.

draegtun 15y ago

Perl comes out best according to Tom Christiansen (author of this Perl Unicode Cookook post). See his Unicode Support Shootout - The Good, the Bad, & the (mostly) Ugly presentation at OSCON 2011 where he gave a comparison of Unicode handling in Javascript, PHP, Go, Python, Ruby, Java, and Perl. ref: http://training.perl.com/OSCON2011/index.html http://training.perl.com/OSCON2011/index.html

joeyh 15y ago
Perl is in the lower tier IMHO. Below Python and Haskell. Because while it has a good core unicode support, every single library makes different choices, so your code constantly needs to encode/decode. Javascript seems to avoid encoding issues entirely, at least in the browser, though Tom has some nice points about it not full supporting unicode. <http://kitenet.net/~joey/blog/entry/unicode_ate_my_homework/&#62 http://kitenet.net/~joey/blog/entry/unicode_ate_my_homework/...;
- ajross 15y ago
  Can you be more specific? Perl's utf8 support is IMHO the cleanest and most transparent. Generally libraries don't have to care at all about the encoding of the stuff they deal with. I'd be curious what trouble you've had. (It's true that early versions of "use utf8" were horribly broken, munging strings silently at I/O time, but that was 10 years ago or more)
  - joeyh 15y ago
    Nearly every XS library takes input with utf-8 encoded to bytes, so you have to manually juggle the encoding when using it. There is no documentation stating what encoding any library expects. I just checked a source tree and found 41 calls to encode_utf8 and 42 calls to decode_utf8 to deal with this across dozens of libraries.
    - ajross 15y ago
      Um... of course they do. XS is a translation library and the domain you're translating to (C) has no (well, no sane) native encoding handling. So you pass byte buffers. That's not a ding on perl though, it's an inherent impedance you get when you cross interface domains. JNI libraries tend to have most of the same problems, either forcing the user to do the UTF16/UTF8 conversion or doing it in the library. I guess I'm surprised that Python and Haskell don't have equivalent messes when dealing with native libraries. Are you sure they don't?
      - joeyh 15y ago
        Perl XS libraries often provide a full object oriented wrapper around the native C calls. They always have, at a minimum, a thin perl wrapper function. There's plenty of scope for the encoding to be done automatically, but the current state is that it's not even done manually by the libraries. Haskell's equivilant FFI interfaces do encoding conversions, so that if they return a String, it's a sequence of unicode code points, as the Haskell standard requires. Those that return raw data will use an appropriate ByteString type. Haskell didn't used to do encoding conversions for FilePaths in the FFI, but this has been fixed in version 7.4, and is even done in a way that doesn't require unix filenames be utf-8 encoded -- other encodings will pass through unchanged.
        chromatic 15y ago
        Encoding conversions at the FFI layer would be very nice. How do the Haskell interfaces know which encodings to expect?
- gizzlon 15y ago
  "every time you call a library or do IO you have to manually deal with conversions.." (from the blog post) This is certainly not true for libraries; Only done that to workaround faults in other "systems"[1]. For IO, you don't have to "manually deal with conversion". But you do have to, if you want to do it correctly, tell it what encodings the files you're going to read are. But you do this once for all IO. There's a reason for this, and it has nothing to do with the language: You can't detect encodings. So, you either standardize on one encoding to rule them all or supply it when doing IO. To be honest, I seldom do it the "correct" way, and it works fine most of the time. [1] E.g., the DB table had one encoding, but the data was actually stored using another one. Edit: Just for the record, dealing with this can be a pain in Perl. Keeping things backwards-compatible means everything is not as streamlined as it could be. But at least it works (I'm looking at you PHP ;)
- draegtun 15y ago
  As someone who munges Unicode data on (a far too!) regular basis I have never found Perl wanting and believe it to be the best tool for the job.
jbjohns 15y ago
>Perl comes out best according to Tom Christiansen It would be hard to find a more bias source.
- chromatic 15y ago
  It would be exceedingly difficult to find someone more fastidious and willing to perform exhaustive research. If Tom has a bias, it's because he knows of that which he speaks.

gregheo 15y ago

Perl is definitely bipolar in this respect -- I've never had an easier time of dealing with Unicode issues than in Perl, but I've also never cursed Unicode so much while dealing with it in Perl.

prakashk 15y ago
Could you say in what other languages you dealt with Unicode?
gauravk92 15y ago
Python has native support for unicode strings, not Unicode support in native string types. Sigh, this is where it gets into bs world, because you have to figure out how to convert back and forth appropriately. Just thought I'd share my recent frustration.