4 ms·

My heart goes out to anyone that ever has to work with encoding issues. I feel like I know 5% of this stuff and there looms a vast mass of chaos like a Gordian

My heart goes out to anyone that ever has to work with encoding issues. I feel like I know 5% of this stuff and there looms a vast mass of chaos like a Gordian knot whenever I need to dip my toes in it.

Motomorgen 10y ago
Agreed completely, I recently had to work with some EBCDIC output files from old mainframe systems. Packed string data in a fixed delimited file. Nail bitingly frustrating stuff.
- _JamesA_ 10y ago
  Having built countless similar conversions, EBCDIC[1] to ASCII can be as simple as a lookup table and "packed string data" is just binary coded decimal [2] with a sign. What is a fixed delimited file? A 'mainframe' file is generally fixed length or delimited. [1]: https://en.wikipedia.org/wiki/EBCDIC https://en.wikipedia.org/wiki/EBCDIC [2]: https://en.wikipedia.org/wiki/Binary-coded_decimal https://en.wikipedia.org/wiki/Binary-coded_decimal
  - JoBrad 10y ago
    I have used delimiters within a fixed-width file so that I can fit more data in a width field in a fixed-width file layout. This is usually when I have to work with a mainframe, and can't define the layout, of course.
nothrabannosir 10y ago
Every programmer has to deal with encoding; it's a fact of life. A list of bytes without encoding is just that: a list of bytes. Handling bytes as text (I.e.: "I/O")? You are now working with encodings. And the funny thing is: it's not that complex.[edit: let me rephrase: it is, but it doesn't have to be. C family is a nightmare, Python et al are delightful]. Unless you don't know how it works---then it's the mystery Gordian knot, as you describe it. The irony is that encoding is a worry precisely for those who try and stay away from it. Don't shy away from encodings; embrace them. Then you will learn to love them. (Another irony: with UTF8 gaining more and more mind share, encoding issues actually become harder to find and debug: they don't show up, and when they do, fewer and fewer people know how to deal with them. Everyone switching to UTF8 just hides the bugs, until it doesn't.)
- yoklov 10y ago
  Python is delightful? Many of the only times i've had issues were in python.
  - mixmastamyk 10y ago
    Python3 is delightful.
- masklinn 10y ago
  > let me rephrase: it is, but it doesn't have to be. C family is a nightmare, Python et al are delightful Unless you do multiplatform development, then the language has a hard time saving you (and Python definitely does not) > (Another irony: with UTF8 gaining more and more mind share, encoding issues actually become harder to find and debug: they don't show up, and when they do, fewer and fewer people know how to deal with them. Everyone switching to UTF8 just hides the bugs, until it doesn't.) That's not true at all. A ton of byte sequences (and 13 standalone bytes) are outright illegal in UTF8, there's a fair amount of error handling in a validating UTF8 decoder[0], whereas there usually isn't any invalid byte (let alone byte sequence) in 8-bit codepages or character sets. When you decode random bytes in Windows-1256 or ISO-8859-9[1] and re-encode them as UTF8, the UTF8 encoder isn't the one at fault for your garbage output, as far as I could see it got perfectly valid unicode data. People passing through unvalidated data (possibly assuming it's UTF8) isn't a problem with UTF8 either, by the way. [0] whether that's used in strict mode or in replacement mode is a different concern and not a blemish on UTF8 itself [1] which will always succeed, you can try it at home, just get a bunch of bytes from urandom and feed them to various decoders, chances are low that you'll generate anything the UTF8 decoder will accept, chances are also low that you'll generate anything an ISO-8859 character set will reject.
- bluejekyll 10y ago
  This article is trying to describe how to work on Windows specifically. I don't think it's saying ignore encodings, though honestly why -everything- isn't just encoded utf-8 I don't know. It's sad that a "modern" OS that had a mostly ground up rewrite (NT) after utf-8 was invented, doesn't have better support for it. I get in memory storage being utf-16, and I'd even accept modern OSes storing files in utf-16, but utf-8 is so elegantly backwards compatible with ascii, it's brain dead not to fix -everything- to work with it. Perhaps this is the biggest difference between a closed OS like Windows and it's more open counterparts, if Windows were open then the community could have fixed this issue long ago.
dkersten 10y ago
Some years ago I had to work with text encoded with 3GPP TS 23.038 [1], which is a 7bit packed encoding with a few quirks. It wasn't particularly fun. [1] https://en.m.wikipedia.org/wiki/GSM_03.38 https://en.m.wikipedia.org/wiki/GSM_03.38
- jfim 10y ago
  The telephony standards are all kinds of bizarre. It's a testament as to how well the specs are written that it actually works.
  - dkersten 10y ago
    The specs are incredibly dense and hard to read, but they are quite detailed. I found many of the protocols to be both incredibly clever and incredibly annoying at the same time.
Can_Not 10y ago
I've noticed that Excel seems to assert that CSVs are not in UTF8. I know the standard is actually ASCII, but nobody on the team seemed willing to understand why their smart quotes came out as "weird characters" (or even understand what smart quotes are). It came out correctly in LibreOffice.
- moogly 10y ago
  This is why I always write a BOM in CSV files (yes, even though it's UTF-8 and it shouldn't have a BOM). Excel will actually respect it. The bigger problem is when Excel tries to be helpful and reformats the data upon opening the file. I can't remember the intricacies, but it used to behave differently whether you opened it from a local path or directly from a browser. I seem to recall using an "incorrect" MIME type helped with that.
  - Symbiote 10y ago
    Adding a BOM to a CSV file then causes many libraries to add the BOM to the first column's heading name.
- TAForObvReasons 10y ago
  Excel predates Unicode by a few years. The first true Excel version on Mac was released in 1985. The first generally recognized use of Unicode dates back to 1988. Excel CSVs default to the system codepage when reading and writing, which makes some sense given that the files are plain text. You can actually control what codepage is used when importing data but you have to use the import option.
- electroly 10y ago
  It's an option that just defaults to the system ANSI codepage. After clicking Data > From Text, in the "Text Import Wizard - Step 1 of 3" screen, change the "File origin" to "65001: Unicode (UTF-8)"

Motomorgen 10y ago

Agreed completely, I recently had to work with some EBCDIC output files from old mainframe systems. Packed string data in a fixed delimited file. Nail bitingly frustrating stuff.

_JamesA_ 10y ago
Having built countless similar conversions, EBCDIC[1] to ASCII can be as simple as a lookup table and "packed string data" is just binary coded decimal [2] with a sign. What is a fixed delimited file? A 'mainframe' file is generally fixed length or delimited. [1]: https://en.wikipedia.org/wiki/EBCDIC https://en.wikipedia.org/wiki/EBCDIC [2]: https://en.wikipedia.org/wiki/Binary-coded_decimal https://en.wikipedia.org/wiki/Binary-coded_decimal
- JoBrad 10y ago
  I have used delimiters within a fixed-width file so that I can fit more data in a width field in a fixed-width file layout. This is usually when I have to work with a mainframe, and can't define the layout, of course.

nothrabannosir 10y ago

Every programmer has to deal with encoding; it's a fact of life. A list of bytes without encoding is just that: a list of bytes. Handling bytes as text (I.e.: "I/O")? You are now working with encodings. And the funny thing is: it's not that complex.[edit: let me rephrase: it is, but it doesn't have to be. C family is a nightmare, Python et al are delightful]. Unless you don't know how it works---then it's the mystery Gordian knot, as you describe it. The irony is that encoding is a worry precisely for those who try and stay away from it. Don't shy away from encodings; embrace them. Then you will learn to love them. (Another irony: with UTF8 gaining more and more mind share, encoding issues actually become harder to find and debug: they don't show up, and when they do, fewer and fewer people know how to deal with them. Everyone switching to UTF8 just hides the bugs, until it doesn't.)

yoklov 10y ago
Python is delightful? Many of the only times i've had issues were in python.
- mixmastamyk 10y ago
  Python3 is delightful.
masklinn 10y ago
> let me rephrase: it is, but it doesn't have to be. C family is a nightmare, Python et al are delightful Unless you do multiplatform development, then the language has a hard time saving you (and Python definitely does not) > (Another irony: with UTF8 gaining more and more mind share, encoding issues actually become harder to find and debug: they don't show up, and when they do, fewer and fewer people know how to deal with them. Everyone switching to UTF8 just hides the bugs, until it doesn't.) That's not true at all. A ton of byte sequences (and 13 standalone bytes) are outright illegal in UTF8, there's a fair amount of error handling in a validating UTF8 decoder[0], whereas there usually isn't any invalid byte (let alone byte sequence) in 8-bit codepages or character sets. When you decode random bytes in Windows-1256 or ISO-8859-9[1] and re-encode them as UTF8, the UTF8 encoder isn't the one at fault for your garbage output, as far as I could see it got perfectly valid unicode data. People passing through unvalidated data (possibly assuming it's UTF8) isn't a problem with UTF8 either, by the way. [0] whether that's used in strict mode or in replacement mode is a different concern and not a blemish on UTF8 itself [1] which will always succeed, you can try it at home, just get a bunch of bytes from urandom and feed them to various decoders, chances are low that you'll generate anything the UTF8 decoder will accept, chances are also low that you'll generate anything an ISO-8859 character set will reject.
bluejekyll 10y ago
This article is trying to describe how to work on Windows specifically. I don't think it's saying ignore encodings, though honestly why -everything- isn't just encoded utf-8 I don't know. It's sad that a "modern" OS that had a mostly ground up rewrite (NT) after utf-8 was invented, doesn't have better support for it. I get in memory storage being utf-16, and I'd even accept modern OSes storing files in utf-16, but utf-8 is so elegantly backwards compatible with ascii, it's brain dead not to fix -everything- to work with it. Perhaps this is the biggest difference between a closed OS like Windows and it's more open counterparts, if Windows were open then the community could have fixed this issue long ago.

dkersten 10y ago

Some years ago I had to work with text encoded with 3GPP TS 23.038 [1], which is a 7bit packed encoding with a few quirks. It wasn't particularly fun. [1] https://en.m.wikipedia.org/wiki/GSM_03.38 https://en.m.wikipedia.org/wiki/GSM_03.38

jfim 10y ago
The telephony standards are all kinds of bizarre. It's a testament as to how well the specs are written that it actually works.
- dkersten 10y ago
  The specs are incredibly dense and hard to read, but they are quite detailed. I found many of the protocols to be both incredibly clever and incredibly annoying at the same time.

Can_Not 10y ago

I've noticed that Excel seems to assert that CSVs are not in UTF8. I know the standard is actually ASCII, but nobody on the team seemed willing to understand why their smart quotes came out as "weird characters" (or even understand what smart quotes are). It came out correctly in LibreOffice.

moogly 10y ago
This is why I always write a BOM in CSV files (yes, even though it's UTF-8 and it shouldn't have a BOM). Excel will actually respect it. The bigger problem is when Excel tries to be helpful and reformats the data upon opening the file. I can't remember the intricacies, but it used to behave differently whether you opened it from a local path or directly from a browser. I seem to recall using an "incorrect" MIME type helped with that.
- Symbiote 10y ago
  Adding a BOM to a CSV file then causes many libraries to add the BOM to the first column's heading name.
TAForObvReasons 10y ago
Excel predates Unicode by a few years. The first true Excel version on Mac was released in 1985. The first generally recognized use of Unicode dates back to 1988. Excel CSVs default to the system codepage when reading and writing, which makes some sense given that the files are plain text. You can actually control what codepage is used when importing data but you have to use the import option.
electroly 10y ago
It's an option that just defaults to the system ANSI codepage. After clicking Data > From Text, in the "Text Import Wizard - Step 1 of 3" screen, change the "File origin" to "65001: Unicode (UTF-8)"