6 ms·

Perl is in the lower tier IMHO. Below Python and Haskell. Because while it has a good core unicode support, every single library makes different choices, so you

by joeyh 15y ago

Perl is in the lower tier IMHO. Below Python and Haskell. Because while it has a good core unicode support, every single library makes different choices, so your code constantly needs to encode/decode. Javascript seems to avoid encoding issues entirely, at least in the browser, though Tom has some nice points about it not full supporting unicode. <http://kitenet.net/~joey/blog/entry/unicode_ate_my_homework/&#62 http://kitenet.net/~joey/blog/entry/unicode_ate_my_homework/...;

ajross 15y ago
Can you be more specific? Perl's utf8 support is IMHO the cleanest and most transparent. Generally libraries don't have to care at all about the encoding of the stuff they deal with. I'd be curious what trouble you've had. (It's true that early versions of "use utf8" were horribly broken, munging strings silently at I/O time, but that was 10 years ago or more)
- joeyh 15y ago
  Nearly every XS library takes input with utf-8 encoded to bytes, so you have to manually juggle the encoding when using it. There is no documentation stating what encoding any library expects. I just checked a source tree and found 41 calls to encode_utf8 and 42 calls to decode_utf8 to deal with this across dozens of libraries.
  - ajross 15y ago
    Um... of course they do. XS is a translation library and the domain you're translating to (C) has no (well, no sane) native encoding handling. So you pass byte buffers. That's not a ding on perl though, it's an inherent impedance you get when you cross interface domains. JNI libraries tend to have most of the same problems, either forcing the user to do the UTF16/UTF8 conversion or doing it in the library. I guess I'm surprised that Python and Haskell don't have equivalent messes when dealing with native libraries. Are you sure they don't?
    - joeyh 15y ago
      Perl XS libraries often provide a full object oriented wrapper around the native C calls. They always have, at a minimum, a thin perl wrapper function. There's plenty of scope for the encoding to be done automatically, but the current state is that it's not even done manually by the libraries. Haskell's equivilant FFI interfaces do encoding conversions, so that if they return a String, it's a sequence of unicode code points, as the Haskell standard requires. Those that return raw data will use an appropriate ByteString type. Haskell didn't used to do encoding conversions for FilePaths in the FFI, but this has been fixed in version 7.4, and is even done in a way that doesn't require unix filenames be utf-8 encoded -- other encodings will pass through unchanged.
      - chromatic 15y ago
        Encoding conversions at the FFI layer would be very nice. How do the Haskell interfaces know which encodings to expect?
gizzlon 15y ago
"every time you call a library or do IO you have to manually deal with conversions.." (from the blog post) This is certainly not true for libraries; Only done that to workaround faults in other "systems"[1]. For IO, you don't have to "manually deal with conversion". But you do have to, if you want to do it correctly, tell it what encodings the files you're going to read are. But you do this once for all IO. There's a reason for this, and it has nothing to do with the language: You can't detect encodings. So, you either standardize on one encoding to rule them all or supply it when doing IO. To be honest, I seldom do it the "correct" way, and it works fine most of the time. [1] E.g., the DB table had one encoding, but the data was actually stored using another one. Edit: Just for the record, dealing with this can be a pain in Perl. Keeping things backwards-compatible means everything is not as streamlined as it could be. But at least it works (I'm looking at you PHP ;)
draegtun 15y ago
As someone who munges Unicode data on (a far too!) regular basis I have never found Perl wanting and believe it to be the best tool for the job.

by joeyh 15y ago

ajross 15y ago

Can you be more specific? Perl's utf8 support is IMHO the cleanest and most transparent. Generally libraries don't have to care at all about the encoding of the stuff they deal with. I'd be curious what trouble you've had. (It's true that early versions of "use utf8" were horribly broken, munging strings silently at I/O time, but that was 10 years ago or more)

joeyh 15y ago
Nearly every XS library takes input with utf-8 encoded to bytes, so you have to manually juggle the encoding when using it. There is no documentation stating what encoding any library expects. I just checked a source tree and found 41 calls to encode_utf8 and 42 calls to decode_utf8 to deal with this across dozens of libraries.
- ajross 15y ago
  Um... of course they do. XS is a translation library and the domain you're translating to (C) has no (well, no sane) native encoding handling. So you pass byte buffers. That's not a ding on perl though, it's an inherent impedance you get when you cross interface domains. JNI libraries tend to have most of the same problems, either forcing the user to do the UTF16/UTF8 conversion or doing it in the library. I guess I'm surprised that Python and Haskell don't have equivalent messes when dealing with native libraries. Are you sure they don't?
  - joeyh 15y ago
    Perl XS libraries often provide a full object oriented wrapper around the native C calls. They always have, at a minimum, a thin perl wrapper function. There's plenty of scope for the encoding to be done automatically, but the current state is that it's not even done manually by the libraries. Haskell's equivilant FFI interfaces do encoding conversions, so that if they return a String, it's a sequence of unicode code points, as the Haskell standard requires. Those that return raw data will use an appropriate ByteString type. Haskell didn't used to do encoding conversions for FilePaths in the FFI, but this has been fixed in version 7.4, and is even done in a way that doesn't require unix filenames be utf-8 encoded -- other encodings will pass through unchanged.
    - chromatic 15y ago
      Encoding conversions at the FFI layer would be very nice. How do the Haskell interfaces know which encodings to expect?

gizzlon 15y ago

"every time you call a library or do IO you have to manually deal with conversions.." (from the blog post) This is certainly not true for libraries; Only done that to workaround faults in other "systems"[1]. For IO, you don't have to "manually deal with conversion". But you do have to, if you want to do it correctly, tell it what encodings the files you're going to read are. But you do this once for all IO. There's a reason for this, and it has nothing to do with the language: You can't detect encodings. So, you either standardize on one encoding to rule them all or supply it when doing IO. To be honest, I seldom do it the "correct" way, and it works fine most of the time. [1] E.g., the DB table had one encoding, but the data was actually stored using another one. Edit: Just for the record, dealing with this can be a pain in Perl. Keeping things backwards-compatible means everything is not as streamlined as it could be. But at least it works (I'm looking at you PHP ;)

draegtun 15y ago

As someone who munges Unicode data on (a far too!) regular basis I have never found Perl wanting and believe it to be the best tool for the job.