> Locales are a generic mechanism for dealing with encodings. What makes UTF-8 special in that regard, and why is an UTF-8 locale a hack? Locales are a generic

> Locales are a generic mechanism for dealing with encodings. What makes UTF-8 special in that regard, and why is an UTF-8 locale a hack? Locales are a generic mechanism for dealing with code that should run differently in different countries. There have been other HN discussions recently about why this is terrible in the present day. Encodings are just one aspect of that. The whole thing where you name a locale, then put a dot and tell it what encoding you really wanted, is what I'm referring to as a hack. There is no standard for locales with dots in them. But the locale system was created at a time when, say, the US was using ISO-8859-1 and Poland was using ISO-8859-2, and this was just a fact about how you had to deal with text. But that's exactly what Unicode got rid of! You don't make Unicode decisions by country (with terribly awkward exceptions such as Japan, where Unicode itself is unpopular, and Python is too). It's not like the US uses UTF-8 and Canada uses UTF-16. You make Unicode decisions based on the OS and APIs that you're interacting with. And that's why we have this Linux dot convention for overriding what the locale would otherwise say so we can use Unicode. So your recommendation to use locales is actually a recommendation to mostly ignore locales, and just use the part after the dot as the name of the encoding you should be using. And to make wild-ass wrong guesses if there's no dot. Taking the locale "C" and interpreting it as the encoding "ASCII" is an example of a wild-ass wrong guess. But there are already environment variables that configure Python to use a particular encoding, without hacking it on top of archaic shit like locales. And harm is done by trying to infer the encoding from the locale, because of the complete wrongness of assuming the "C" locale means to use Python's "ASCII" encoding. The resulting behavior is not correct, and the reasoning for it is not correct. It's a bug. It will probably be fixed in one of the next two versions of Python. People run Python from cron jobs, from IDEs, from all sorts of places that don't set the locale the way the Ubuntu shell does, and get bafflingly inconsistent results. You can say "fix your locales then" all you want, but this is not an answer that makes Python more usable, and the developers have acknowledged this. Here's a particular example of where I think you're coming at this from the wrong direction, where I think you're taking the current behavior of Python as if it were actually some sort of intentionally-designed standard: > That's true for any locale, if a character comes up in the output that cannot be encoded in it - it should just use some reasonable substitution. And if you're actually dealing with binary data, then you should be reading and writing bytes objects, not printing strings, and then the whole question of encoding is moot. You don't encode things in a locale! You encode things in encodings! I'm definitely not talking about binary data, I'm talking about printing out perfectly normal characters like "ü". The locale "C" does not tell you anything about what encoding to use. Which is very different from Python's current assumption (which may go away in 3.7) that it's telling you to use ASCII and explode.