5 ms·

Every programmer has to deal with encoding; it's a fact of life. A list of bytes without encoding is just that: a list of bytes. Handling bytes as text (I.e.: "

Every programmer has to deal with encoding; it's a fact of life. A list of bytes without encoding is just that: a list of bytes. Handling bytes as text (I.e.: "I/O")? You are now working with encodings. And the funny thing is: it's not that complex.[edit: let me rephrase: it is, but it doesn't have to be. C family is a nightmare, Python et al are delightful]. Unless you don't know how it works---then it's the mystery Gordian knot, as you describe it. The irony is that encoding is a worry precisely for those who try and stay away from it. Don't shy away from encodings; embrace them. Then you will learn to love them. (Another irony: with UTF8 gaining more and more mind share, encoding issues actually become harder to find and debug: they don't show up, and when they do, fewer and fewer people know how to deal with them. Everyone switching to UTF8 just hides the bugs, until it doesn't.)

yoklov 11y ago
Python is delightful? Many of the only times i've had issues were in python.
- mixmastamyk 11y ago
  Python3 is delightful.
masklinn 11y ago
> let me rephrase: it is, but it doesn't have to be. C family is a nightmare, Python et al are delightful Unless you do multiplatform development, then the language has a hard time saving you (and Python definitely does not) > (Another irony: with UTF8 gaining more and more mind share, encoding issues actually become harder to find and debug: they don't show up, and when they do, fewer and fewer people know how to deal with them. Everyone switching to UTF8 just hides the bugs, until it doesn't.) That's not true at all. A ton of byte sequences (and 13 standalone bytes) are outright illegal in UTF8, there's a fair amount of error handling in a validating UTF8 decoder[0], whereas there usually isn't any invalid byte (let alone byte sequence) in 8-bit codepages or character sets. When you decode random bytes in Windows-1256 or ISO-8859-9[1] and re-encode them as UTF8, the UTF8 encoder isn't the one at fault for your garbage output, as far as I could see it got perfectly valid unicode data. People passing through unvalidated data (possibly assuming it's UTF8) isn't a problem with UTF8 either, by the way. [0] whether that's used in strict mode or in replacement mode is a different concern and not a blemish on UTF8 itself [1] which will always succeed, you can try it at home, just get a bunch of bytes from urandom and feed them to various decoders, chances are low that you'll generate anything the UTF8 decoder will accept, chances are also low that you'll generate anything an ISO-8859 character set will reject.
bluejekyll 11y ago
This article is trying to describe how to work on Windows specifically. I don't think it's saying ignore encodings, though honestly why -everything- isn't just encoded utf-8 I don't know. It's sad that a "modern" OS that had a mostly ground up rewrite (NT) after utf-8 was invented, doesn't have better support for it. I get in memory storage being utf-16, and I'd even accept modern OSes storing files in utf-16, but utf-8 is so elegantly backwards compatible with ascii, it's brain dead not to fix -everything- to work with it. Perhaps this is the biggest difference between a closed OS like Windows and it's more open counterparts, if Windows were open then the community could have fixed this issue long ago.
- jameshart 11y ago
  The problem with elegant backwards compatibility with ASCII is that few systems ever stuck to ASCII - everybody used different, incompatible extensions to ASCII, like the ISO 8859 family and codepage 1252. So by being elegantly backwards compatible with ASCII, UTF-8 also manages to be subtly incompatible with the majority of almost-aSCII data that exists, in ways that sometimes don't matter. Until they do.
  - kps 11y ago
    ISO 8859 (or big brother ISO 10367) with ISO 4873 is fairly sane, backwards compatible with ASCII, distinguishable from UTF-8, and historically supported by X terminals… but not much else. ‘Plain ASCII’ is not often fully supported either; how many people even know that the standard included composing accented characters by overstriking (e.g. a BS " → ä)?
    - jameshart 11y ago
      The ASCII standard has nothing to say about backspace overstriking - that would be something defined by a terminal, file format, or wire specification. In a more common example, ASCII has nothing to say about how you move the cursor to the start of a new line. Some terminal standards will do so on an LF, others just move the cursor down on LF; file and wire specs need to take a position on how they want to represent a line break; and now we get to the point where CRLF is a magic sequence needed to trigger a line break in some formats and it is completely unrelated to how a particular terminal behaves.
jameshart 11y ago
Yes, the danger is when people think 'Encodings are too complicated! Surely my framework/platform/library should be abstracting all this stuff for me?' and then proceed to assume that because it should be dealt with at a lower level, that is is dealt with at a lower level. All I/O encoding abstractions leak sooner or later. For which reason, all programmers need to stop pretending encodings don't affect them and read this: http://www.joelonsoftware.com/articles/Unicode.html http://www.joelonsoftware.com/articles/Unicode.html
77pt77 11y ago
> let me rephrase: it is, but it doesn't have to be. C family is a nightmare, Python et al are delightful No. In python you have the absolutely retarded behaviour of a program running fine in the command line but crashing if you redirect the output to a file.
revelation 11y ago
Python (the 90% of programs 2.x branch) is horrible. It makes the cardinal mistake of mixing binary data and encoded data in one String type. C is much better, because at least it doesn't pretend to support encodings. So you're forced to use a 3rd party library anyway.
- uxp 11y ago
  > C is much better, because at least it doesn't pretend to support encodings. So you're forced to use a 3rd party library anyway. C is better because it _doesnt_ provide some kind of interface into encoding. A byte is a byte is a byte. Whether that's some slice of a binary blob or the first byte in a multi-byte unicode string is completely irrelevant. The most infuriating thing I've found with higher level languages (Ruby/Python/etc.) is their abstractions on top of string encodings that fail to cover the hundreds of thousands of edge-cases (which, they rightfully shouldn't cover either), meaning that the only time you do end up having to wrangle encodings is when things go south and you have to fight the language's defaults just to get some stupid email address that most likely had a bit flip in transit somewhere to properly write to a CSV.
hsivonen 11y ago
Python delightful? In Python 2, the behavior of Unicode strings (UTF-16 vs. UTF-32) depends on what options the interpreter was compiled with. Then they made a backward-incompatible change to "fix" Unicode but did the exact wrong thing (UTF-32, UCS-2 or ISO-8859-1 depending on string content) instead of the right thing (UTF-8 storage with UTF-32 iterators).
- ygra 11y ago
  I got the impression that the internal encoding Python chooses for a string is an implementation detail you shouldn't (and cannot) care about. All the public ways if accessing string contents operate on code points, unless you convert the string to a byte array first.

yoklov 11y ago

Python is delightful? Many of the only times i've had issues were in python.

mixmastamyk 11y ago
Python3 is delightful.

masklinn 11y ago

> let me rephrase: it is, but it doesn't have to be. C family is a nightmare, Python et al are delightful Unless you do multiplatform development, then the language has a hard time saving you (and Python definitely does not) > (Another irony: with UTF8 gaining more and more mind share, encoding issues actually become harder to find and debug: they don't show up, and when they do, fewer and fewer people know how to deal with them. Everyone switching to UTF8 just hides the bugs, until it doesn't.) That's not true at all. A ton of byte sequences (and 13 standalone bytes) are outright illegal in UTF8, there's a fair amount of error handling in a validating UTF8 decoder[0], whereas there usually isn't any invalid byte (let alone byte sequence) in 8-bit codepages or character sets. When you decode random bytes in Windows-1256 or ISO-8859-9[1] and re-encode them as UTF8, the UTF8 encoder isn't the one at fault for your garbage output, as far as I could see it got perfectly valid unicode data. People passing through unvalidated data (possibly assuming it's UTF8) isn't a problem with UTF8 either, by the way. [0] whether that's used in strict mode or in replacement mode is a different concern and not a blemish on UTF8 itself [1] which will always succeed, you can try it at home, just get a bunch of bytes from urandom and feed them to various decoders, chances are low that you'll generate anything the UTF8 decoder will accept, chances are also low that you'll generate anything an ISO-8859 character set will reject.

bluejekyll 11y ago

This article is trying to describe how to work on Windows specifically. I don't think it's saying ignore encodings, though honestly why -everything- isn't just encoded utf-8 I don't know. It's sad that a "modern" OS that had a mostly ground up rewrite (NT) after utf-8 was invented, doesn't have better support for it. I get in memory storage being utf-16, and I'd even accept modern OSes storing files in utf-16, but utf-8 is so elegantly backwards compatible with ascii, it's brain dead not to fix -everything- to work with it. Perhaps this is the biggest difference between a closed OS like Windows and it's more open counterparts, if Windows were open then the community could have fixed this issue long ago.

jameshart 11y ago
The problem with elegant backwards compatibility with ASCII is that few systems ever stuck to ASCII - everybody used different, incompatible extensions to ASCII, like the ISO 8859 family and codepage 1252. So by being elegantly backwards compatible with ASCII, UTF-8 also manages to be subtly incompatible with the majority of almost-aSCII data that exists, in ways that sometimes don't matter. Until they do.
- kps 11y ago
  ISO 8859 (or big brother ISO 10367) with ISO 4873 is fairly sane, backwards compatible with ASCII, distinguishable from UTF-8, and historically supported by X terminals… but not much else. ‘Plain ASCII’ is not often fully supported either; how many people even know that the standard included composing accented characters by overstriking (e.g. a BS " → ä)?
  - jameshart 11y ago
    The ASCII standard has nothing to say about backspace overstriking - that would be something defined by a terminal, file format, or wire specification. In a more common example, ASCII has nothing to say about how you move the cursor to the start of a new line. Some terminal standards will do so on an LF, others just move the cursor down on LF; file and wire specs need to take a position on how they want to represent a line break; and now we get to the point where CRLF is a magic sequence needed to trigger a line break in some formats and it is completely unrelated to how a particular terminal behaves.

jameshart 11y ago

Yes, the danger is when people think 'Encodings are too complicated! Surely my framework/platform/library should be abstracting all this stuff for me?' and then proceed to assume that because it should be dealt with at a lower level, that is is dealt with at a lower level. All I/O encoding abstractions leak sooner or later. For which reason, all programmers need to stop pretending encodings don't affect them and read this: http://www.joelonsoftware.com/articles/Unicode.html http://www.joelonsoftware.com/articles/Unicode.html

77pt77 11y ago

> let me rephrase: it is, but it doesn't have to be. C family is a nightmare, Python et al are delightful No. In python you have the absolutely retarded behaviour of a program running fine in the command line but crashing if you redirect the output to a file.

revelation 11y ago

Python (the 90% of programs 2.x branch) is horrible. It makes the cardinal mistake of mixing binary data and encoded data in one String type. C is much better, because at least it doesn't pretend to support encodings. So you're forced to use a 3rd party library anyway.

uxp 11y ago
> C is much better, because at least it doesn't pretend to support encodings. So you're forced to use a 3rd party library anyway. C is better because it _doesnt_ provide some kind of interface into encoding. A byte is a byte is a byte. Whether that's some slice of a binary blob or the first byte in a multi-byte unicode string is completely irrelevant. The most infuriating thing I've found with higher level languages (Ruby/Python/etc.) is their abstractions on top of string encodings that fail to cover the hundreds of thousands of edge-cases (which, they rightfully shouldn't cover either), meaning that the only time you do end up having to wrangle encodings is when things go south and you have to fight the language's defaults just to get some stupid email address that most likely had a bit flip in transit somewhere to properly write to a CSV.

hsivonen 11y ago

Python delightful? In Python 2, the behavior of Unicode strings (UTF-16 vs. UTF-32) depends on what options the interpreter was compiled with. Then they made a backward-incompatible change to "fix" Unicode but did the exact wrong thing (UTF-32, UCS-2 or ISO-8859-1 depending on string content) instead of the right thing (UTF-8 storage with UTF-32 iterators).

ygra 11y ago
I got the impression that the internal encoding Python chooses for a string is an implementation detail you shouldn't (and cannot) care about. All the public ways if accessing string contents operate on code points, unless you convert the string to a byte array first.