5 ms·

Charset conversion being painless where?

rwmj 3y ago
Not any that I'm aware of. You have your choice of programming languages where they got it wrong (Python 3, Ruby), languages which are incredibly nit-picky (Rust), languages which are full of footguns (C, C++), languages which pass on the issue (OCaml), languages which assume everything is UTF8 (Perl), languages which assume everything is UTF16 except when they forgot about planes and assume UCS-2 (Java, everything Windows), languages which are mad (APL), ...
- im3w1l 3y ago
  I'm curious what you think python got wrong.
  - rwmj 3y ago
    Your program will crash at runtime if you pass any string that isn't valid UTF-8 and you forgot to use bytes, and using .decode requires that your input has a valid, known encoding which is rarely true in the real world of messy data. It's a problem with Unix filenames where the encoding is just a convention. A Python program that doesn't take great care can crash on a parameter that takes a filename even if that filename is just passed to a function like 'open' so no sanitisation or conversion is necessary. This is a very real problem in hivex, our Windows registry library, where the Python bindings don't really work well. The Windows registry is a hodge podge of random encodings, essentially whatever the program that wrote the registry key thought it was using at the time. When parsed through Python as a string this means you'll get unicode decoding errors all over the place. Also more in this article: https://changelog.complete.org/archives/9938-the-python-unicode-mess https://changelog.complete.org/archives/9938-the-python-unic...
    - lyu07282 3y ago
      You can tell .decode what to do with errors so it won't throw an exception, the default is 'strict'. I think python 3.6+ did a pretty good job with it overall. Don't get me wrong it's still painful and annoying and bug prone, but the point is, it's encodings, it's always going to suck no matter what.
    - SAI_Peregrinus 3y ago
      > It's a problem with Unix filenames where the encoding is just a convention UNIX filenames are NOT necessarily printable text, they're byte strings. Don't treat them as printable text. They're sequences of bytes not containing 0x00 or 0x2F, but with no encoding. Same for Windows registry keys. Don't mistake byte strings for text. Text is a byte string with an encoding which describes which byte values are valid and which characters each byte (or sequence of bytes) corresponds to. If the encoding information is discarded, it stops being text and becomes just a byte string.
      - Karellen 3y ago
        > UNIX filenames are NOT necessarily printable text, they're byte strings. Don't treat them as printable text. But they nearly always also happen to be printable text. Treating them as printable text is soooo convenient 99.9% of the time. Having them be almost printable text, but not quite, is an API design wart that is begging to be used incorrectly. It's the inverse of an affordance.
        SAI_Peregrinus 3y ago
        Sure, I can agree with that. Just because the API is bad doesn't mean you can treat it as though it's the API you wish it were.
        Karellen 3y ago
        OTOH, if "stop whining about how hard the system is to use correctly, just pay attention, be careful, and don't write bugs" worked, we wouldn't have issues like the one this thread is about, and no-one would have needed to invent Rust.
        SAI_Peregrinus 3y ago
        Oh, 100%. It's critical to wrap the crappy APIs into good ones. But that means you have to write the wrapper, and do so very carefully. Then hope you can get your wrapper adopted widely enough to be standardized, and have the old crappy API deprecated & removed (unlikely). In practice, the crappy API will be kept for backwards compatibility, and will forever remain a footgun. You have to be aware of it, even if only in a "here be dragons" manner. The POSIX filename rules are a "here be dragons" sign. If you can avoid dealing with them, it's best to do so. If you can't avoid it, you'll need to parse your inputs into valid text, and fall back to a byte string handling path (or just fail with error) if they're not text. You can't safely just assume they're text & treat them as such, it doesn't end well.
    - hifromwork 3y ago
      >using .decode requires that your input has a valid, known encoding which is rarely true in the real world of messy data. How else would you decode a string without knowing its encoding? You can either guess (and risk invalid result/decode errors) or store this information somewhere. This is universal and true in every language. In most cases today people choose to guess utf-8.
  - rstuart4133 3y ago
    How about: >>> 'abc' + 'def'[1] abce >>> b'abc' + 'def'[1] TypeError: can't concat int to bytes It's been responsible for a lot of bugs in my code. They copied the "bytes is an array of integers" thing from Java. Big mistake. Python is not Java. Them treating filenames as strings is another bug factory on Linux. Almost no one unit tests filenames with invalid encodings, so the result is a whole pile of python3 programs will fall when given perfectly valid input, whereas those same programs in python2 were fine. It's a very odd outcome given python harks from Linux.
- tialaramex 3y ago
  Of these outcomes I like the choice to pick nits best. I felt the same way before I learned any Rust (ie I wrote C where I had to manually groom strings) so this went in my pile of things to like about Rust. Apparently Rust only chose relatively narrowly to have &str (the string slice type, which is guaranteed to be UTF-8 text) at all, rather than just &[u8] (a slice of bytes) everywhere and to my mind that's a pretty serious benefit. The choice to have this in a language with the safe/ unsafe distinction works very nicely because in so many languages you'd have this promised UTF-8 type and then in practice everybody and their dog uses the unsafe assumed conversion because it's easier, but in Rust you're pulled up short because that conversion needs an unsafe block, your local style may require (and good practice certainly does) that you address this with a safety comment explaining why it's OK and... it just isn't, in most cases. So you write the safe conversion instead unless you really need not to. This is a really nice nudge, you can do the Wrong Thing™, but it's just easier not to.
  - masfuerte 3y ago
    Rust still has its warts. When dealing with an archive, say, you can find yourself needing to deal with Windows strings on Unix, or vice versa, but Rust only provides the string type for the platform you are running on. It could do with UnixString and WindowsString in addition to OsString. MFC had a similar problem for years. It had a CString class which was ANSI or Unicode, depending on how your compiled your app, but moderately often you needed the other one, so it should have had a CStringA and CStringW too, with nice conversions between them.
    - tialaramex 3y ago
      OsString isn't "the native string type", it's a container for whatever was convenient for Rust's internals on that OS, there are convenience functions so it probably feels like it's "UnixString" or "WindowsString" but it's neither. In most of these file format cases what you've got is &[u8] or &[u16] and maybe it's the NonZero variant instead, so I think it's fine to be explicit that's what is going on and maybe in the process remind you to check - is this data UTF16LE? UTF16 with a BOM? UCS2 with a nod and a wink? Just arbitrary 16-bit integers and good luck? But like I said, I favoured "picking nits" long before I learned Rust, so mileage may vary.
- randrus 3y ago
  …languages which are mad.. Reference to “Celestial Emporium of Benevolent Knowledge”?
  - rwmj 3y ago
    It was a sly dig at APL for using non-ASCII characters as regular operators. Actually I have no idea how those were implemented in the language. Presumably not as Unicode since APL predates Unicode by quite a considerable number of years. Does anyone know? As a meta-joke I was also considering: APL (j/k)
    - mlochbaum 3y ago
      In modern APLs a character scalar is just a Unicode code point, which you might consider UTF-32. It's no trouble to work with. Although interfacing with things like filenames that can be invalid UTF-8 is a bit of a mess; Dyalog encodes these using the character range that is reserved for UTF-16 surrogates and therefore unused. If you know you want to work with a byte sequence instead, you can use Unicode characters 0-255, and an array of these will be optimized to use one byte per character in dialects that care about performance.
vbezhenar 3y ago
Java is pretty painless. Never had any issues with it.
- bobmcnamara 3y ago
  Java doesn't read uniboms sanely. Heaven help you parsing mixed input files.
  - neonsunset 3y ago
    It must be very difficult to use one of the dozens of packages that let you detect and pick correct encoding to wrap a file stream before reading into a string.
    - bobmcnamara 3y ago
      Empirically yes! Nearly everything we tried would do a passable job on European texts, and misconvert short texts in less common character sets. Because the problem is ambiguous, we had the best luck with a library that allowed us to disable possible encodings. We thought we could just remove encodings that were not widely in use in the countries we served, and that mostly fixed it, but it still came up a couple times a year. Eventually we had a large enough corpus of sample documents that we could test any changes to the character set bitmap for impact to at least some of each better supported character set. Just not my cup of tea though.
neonsunset 3y ago
Not in C/C++ which is what makes you ask this question :) Rust, C#, Java and Go are fairly straightforward in this regard. `Encoding.{Name}.GetString/GetBytes`

rwmj 3y ago

Not any that I'm aware of. You have your choice of programming languages where they got it wrong (Python 3, Ruby), languages which are incredibly nit-picky (Rust), languages which are full of footguns (C, C++), languages which pass on the issue (OCaml), languages which assume everything is UTF8 (Perl), languages which assume everything is UTF16 except when they forgot about planes and assume UCS-2 (Java, everything Windows), languages which are mad (APL), ...

im3w1l 3y ago
I'm curious what you think python got wrong.
- rwmj 3y ago
  Your program will crash at runtime if you pass any string that isn't valid UTF-8 and you forgot to use bytes, and using .decode requires that your input has a valid, known encoding which is rarely true in the real world of messy data. It's a problem with Unix filenames where the encoding is just a convention. A Python program that doesn't take great care can crash on a parameter that takes a filename even if that filename is just passed to a function like 'open' so no sanitisation or conversion is necessary. This is a very real problem in hivex, our Windows registry library, where the Python bindings don't really work well. The Windows registry is a hodge podge of random encodings, essentially whatever the program that wrote the registry key thought it was using at the time. When parsed through Python as a string this means you'll get unicode decoding errors all over the place. Also more in this article: https://changelog.complete.org/archives/9938-the-python-unicode-mess https://changelog.complete.org/archives/9938-the-python-unic...
  - lyu07282 3y ago
    You can tell .decode what to do with errors so it won't throw an exception, the default is 'strict'. I think python 3.6+ did a pretty good job with it overall. Don't get me wrong it's still painful and annoying and bug prone, but the point is, it's encodings, it's always going to suck no matter what.
  - SAI_Peregrinus 3y ago
    > It's a problem with Unix filenames where the encoding is just a convention UNIX filenames are NOT necessarily printable text, they're byte strings. Don't treat them as printable text. They're sequences of bytes not containing 0x00 or 0x2F, but with no encoding. Same for Windows registry keys. Don't mistake byte strings for text. Text is a byte string with an encoding which describes which byte values are valid and which characters each byte (or sequence of bytes) corresponds to. If the encoding information is discarded, it stops being text and becomes just a byte string.
    - Karellen 3y ago
      > UNIX filenames are NOT necessarily printable text, they're byte strings. Don't treat them as printable text. But they nearly always also happen to be printable text. Treating them as printable text is soooo convenient 99.9% of the time. Having them be almost printable text, but not quite, is an API design wart that is begging to be used incorrectly. It's the inverse of an affordance.
      - SAI_Peregrinus 3y ago
        Sure, I can agree with that. Just because the API is bad doesn't mean you can treat it as though it's the API you wish it were.
        Karellen 3y ago
        OTOH, if "stop whining about how hard the system is to use correctly, just pay attention, be careful, and don't write bugs" worked, we wouldn't have issues like the one this thread is about, and no-one would have needed to invent Rust.
        SAI_Peregrinus 3y ago
        Oh, 100%. It's critical to wrap the crappy APIs into good ones. But that means you have to write the wrapper, and do so very carefully. Then hope you can get your wrapper adopted widely enough to be standardized, and have the old crappy API deprecated & removed (unlikely). In practice, the crappy API will be kept for backwards compatibility, and will forever remain a footgun. You have to be aware of it, even if only in a "here be dragons" manner. The POSIX filename rules are a "here be dragons" sign. If you can avoid dealing with them, it's best to do so. If you can't avoid it, you'll need to parse your inputs into valid text, and fall back to a byte string handling path (or just fail with error) if they're not text. You can't safely just assume they're text & treat them as such, it doesn't end well.
  - hifromwork 3y ago
    >using .decode requires that your input has a valid, known encoding which is rarely true in the real world of messy data. How else would you decode a string without knowing its encoding? You can either guess (and risk invalid result/decode errors) or store this information somewhere. This is universal and true in every language. In most cases today people choose to guess utf-8.
- rstuart4133 3y ago
  How about: >>> 'abc' + 'def'[1] abce >>> b'abc' + 'def'[1] TypeError: can't concat int to bytes It's been responsible for a lot of bugs in my code. They copied the "bytes is an array of integers" thing from Java. Big mistake. Python is not Java. Them treating filenames as strings is another bug factory on Linux. Almost no one unit tests filenames with invalid encodings, so the result is a whole pile of python3 programs will fall when given perfectly valid input, whereas those same programs in python2 were fine. It's a very odd outcome given python harks from Linux.
tialaramex 3y ago
Of these outcomes I like the choice to pick nits best. I felt the same way before I learned any Rust (ie I wrote C where I had to manually groom strings) so this went in my pile of things to like about Rust. Apparently Rust only chose relatively narrowly to have &str (the string slice type, which is guaranteed to be UTF-8 text) at all, rather than just &[u8] (a slice of bytes) everywhere and to my mind that's a pretty serious benefit. The choice to have this in a language with the safe/ unsafe distinction works very nicely because in so many languages you'd have this promised UTF-8 type and then in practice everybody and their dog uses the unsafe assumed conversion because it's easier, but in Rust you're pulled up short because that conversion needs an unsafe block, your local style may require (and good practice certainly does) that you address this with a safety comment explaining why it's OK and... it just isn't, in most cases. So you write the safe conversion instead unless you really need not to. This is a really nice nudge, you can do the Wrong Thing™, but it's just easier not to.
- masfuerte 3y ago
  Rust still has its warts. When dealing with an archive, say, you can find yourself needing to deal with Windows strings on Unix, or vice versa, but Rust only provides the string type for the platform you are running on. It could do with UnixString and WindowsString in addition to OsString. MFC had a similar problem for years. It had a CString class which was ANSI or Unicode, depending on how your compiled your app, but moderately often you needed the other one, so it should have had a CStringA and CStringW too, with nice conversions between them.
  - tialaramex 3y ago
    OsString isn't "the native string type", it's a container for whatever was convenient for Rust's internals on that OS, there are convenience functions so it probably feels like it's "UnixString" or "WindowsString" but it's neither. In most of these file format cases what you've got is &[u8] or &[u16] and maybe it's the NonZero variant instead, so I think it's fine to be explicit that's what is going on and maybe in the process remind you to check - is this data UTF16LE? UTF16 with a BOM? UCS2 with a nod and a wink? Just arbitrary 16-bit integers and good luck? But like I said, I favoured "picking nits" long before I learned Rust, so mileage may vary.
randrus 3y ago
…languages which are mad.. Reference to “Celestial Emporium of Benevolent Knowledge”?
- rwmj 3y ago
  It was a sly dig at APL for using non-ASCII characters as regular operators. Actually I have no idea how those were implemented in the language. Presumably not as Unicode since APL predates Unicode by quite a considerable number of years. Does anyone know? As a meta-joke I was also considering: APL (j/k)
  - mlochbaum 3y ago
    In modern APLs a character scalar is just a Unicode code point, which you might consider UTF-32. It's no trouble to work with. Although interfacing with things like filenames that can be invalid UTF-8 is a bit of a mess; Dyalog encodes these using the character range that is reserved for UTF-16 surrogates and therefore unused. If you know you want to work with a byte sequence instead, you can use Unicode characters 0-255, and an array of these will be optimized to use one byte per character in dialects that care about performance.

vbezhenar 3y ago

Java is pretty painless. Never had any issues with it.

bobmcnamara 3y ago
Java doesn't read uniboms sanely. Heaven help you parsing mixed input files.
- neonsunset 3y ago
  It must be very difficult to use one of the dozens of packages that let you detect and pick correct encoding to wrap a file stream before reading into a string.
  - bobmcnamara 3y ago
    Empirically yes! Nearly everything we tried would do a passable job on European texts, and misconvert short texts in less common character sets. Because the problem is ambiguous, we had the best luck with a library that allowed us to disable possible encodings. We thought we could just remove encodings that were not widely in use in the countries we served, and that mostly fixed it, but it still came up a couple times a year. Eventually we had a large enough corpus of sample documents that we could test any changes to the character set bitmap for impact to at least some of each better supported character set. Just not my cup of tea though.

neonsunset 3y ago

Not in C/C++ which is what makes you ask this question :) Rust, C#, Java and Go are fairly straightforward in this regard. `Encoding.{Name}.GetString/GetBytes`