3 ms·

Nearly every XS library takes input with utf-8 encoded to bytes, so you have to manually juggle the encoding when using it. There is no documentation stating wh

by joeyh 15y ago

Nearly every XS library takes input with utf-8 encoded to bytes, so you have to manually juggle the encoding when using it. There is no documentation stating what encoding any library expects. I just checked a source tree and found 41 calls to encode_utf8 and 42 calls to decode_utf8 to deal with this across dozens of libraries.

ajross 15y ago
Um... of course they do. XS is a translation library and the domain you're translating to (C) has no (well, no sane) native encoding handling. So you pass byte buffers. That's not a ding on perl though, it's an inherent impedance you get when you cross interface domains. JNI libraries tend to have most of the same problems, either forcing the user to do the UTF16/UTF8 conversion or doing it in the library. I guess I'm surprised that Python and Haskell don't have equivalent messes when dealing with native libraries. Are you sure they don't?
- joeyh 15y ago
  Perl XS libraries often provide a full object oriented wrapper around the native C calls. They always have, at a minimum, a thin perl wrapper function. There's plenty of scope for the encoding to be done automatically, but the current state is that it's not even done manually by the libraries. Haskell's equivilant FFI interfaces do encoding conversions, so that if they return a String, it's a sequence of unicode code points, as the Haskell standard requires. Those that return raw data will use an appropriate ByteString type. Haskell didn't used to do encoding conversions for FilePaths in the FFI, but this has been fixed in version 7.4, and is even done in a way that doesn't require unix filenames be utf-8 encoded -- other encodings will pass through unchanged.
  - chromatic 15y ago
    Encoding conversions at the FFI layer would be very nice. How do the Haskell interfaces know which encodings to expect?

3 ms·

Nearly every XS library takes input with utf-8 encoded to bytes, so you have to manually juggle the encoding when using it. There is no documentation stating wh

by joeyh 15y ago

ajross 15y ago
Um... of course they do. XS is a translation library and the domain you're translating to (C) has no (well, no sane) native encoding handling. So you pass byte buffers. That's not a ding on perl though, it's an inherent impedance you get when you cross interface domains. JNI libraries tend to have most of the same problems, either forcing the user to do the UTF16/UTF8 conversion or doing it in the library. I guess I'm surprised that Python and Haskell don't have equivalent messes when dealing with native libraries. Are you sure they don't?
- joeyh 15y ago
  Perl XS libraries often provide a full object oriented wrapper around the native C calls. They always have, at a minimum, a thin perl wrapper function. There's plenty of scope for the encoding to be done automatically, but the current state is that it's not even done manually by the libraries. Haskell's equivilant FFI interfaces do encoding conversions, so that if they return a String, it's a sequence of unicode code points, as the Haskell standard requires. Those that return raw data will use an appropriate ByteString type. Haskell didn't used to do encoding conversions for FilePaths in the FFI, but this has been fixed in version 7.4, and is even done in a way that doesn't require unix filenames be utf-8 encoded -- other encodings will pass through unchanged.
  - chromatic 15y ago
    Encoding conversions at the FFI layer would be very nice. How do the Haskell interfaces know which encodings to expect?