4 ms·

Locales are not a real way of specifying an encoding for standard in and out. They were not designed for a world with Unicode and UTF-8 in it; they were designe

• int_19h 9y ago
  > Locale suffixes ".utf8" and ".UTF-8" are hacks by specific Linux distributions, and people want their Python code to work even if their system has not implemented this hack. I do want my Python code to work, yes. But I assure you that if my locale says ru_RU.KOI8-R, when I say "work", I don't mean "dump garbled stuff on my screen, because you output UTF-8 when I specifically asked you to use KOI8-R". I also don't see why the UTF-8 encoding specifier in locale is a "hack". UTF-8 is just that, an encoding. Locales are a generic mechanism for dealing with encodings. What makes UTF-8 special in that regard, and why is an UTF-8 locale a hack? > The locale "C" does not really mean you want your program to explode when it sees a non-ASCII byte. What Python does here does not promote compatibility in any way. That's true for any locale, if a character comes up in the output that cannot be encoded in it - it should just use some reasonable substitution. And if you're actually dealing with binary data, then you should be reading and writing bytes objects, not printing strings, and then the whole question of encoding is moot. > BSD has no UTF-8 locales and has never pretended that locales work with Unicode. That would be a surprise to my FreeBSD installation, which is running with en_US.UTF-8. Do you mean that the locales don't come generated by default? I'm not sure why it's a big deal ... I mean, Arch doesn't come with any locales generated by default, but I don't think anyone would claim that it doesn't support UTF-8. One other thing BSD does, is that it doesn't guarantee that wchar_t is Unicode in all locales (Linux/glibc does). But that's a different and orthogonal issue. > Windows' equivalent of locales is extremely deprecated and never actually supported UTF-8. The modern Windows APIs that deal with standard in and out correspond to Python's Unicode string type, not its bytes type; encoding the Unicode I/O into bytes is not Python's responsibility. Win32 doesn't have any string-based I/O functions, except for console. So if you're printing directly to the screen (and ignoring / not supporting redirection!), then yes, you can just invoke WriteConsole, give it a UTF-16 string, and be done with it. But if you're printing to a file, or to stdin/out as a handle, then you only have ReadFile/WriteFile, which require a byte array; you're expected to do your own encoding. By the way, and speaking of proper Unicode handling APIs - because of this disparity, and the desire to avoid a conversion unnecessarily (due to the lack of UTF-8 locale making it lossy) so as to allow printing out Unicode, Python actually has to resort to hacks to do this on Win32. Specifically, it has a special class - _io._Win32ConsoleIO - that is implemented using ReadConsole/WriteConsole. If output is a Win32 console, then that class is used, and Unicode is fully supported in the output. If output is redirected, then the standard implementation (that does encoding + Read/WriteFile) is used instead. > Really every operating system but Linux has this figured out, and in practice, Linux users want UTF-8 regardless of what their locale says. I strongly disagree. The users who want it but don't know how to configure it, are already using a distro that says that it should be UTF-8. The users who are using something that doesn't, either do it for a reason, or are knowledgeable enough to know what's going on and why, and deal with it accordingly. Either way, no harm is done by just using the locale setting. On Ubuntu etc, it will just be UTF-8, and everyone will be happy.
  • rspeer 9y ago
    > Locales are a generic mechanism for dealing with encodings. What makes UTF-8 special in that regard, and why is an UTF-8 locale a hack? Locales are a generic mechanism for dealing with code that should run differently in different countries. There have been other HN discussions recently about why this is terrible in the present day. Encodings are just one aspect of that. The whole thing where you name a locale, then put a dot and tell it what encoding you really wanted, is what I'm referring to as a hack. There is no standard for locales with dots in them. But the locale system was created at a time when, say, the US was using ISO-8859-1 and Poland was using ISO-8859-2, and this was just a fact about how you had to deal with text. But that's exactly what Unicode got rid of! You don't make Unicode decisions by country (with terribly awkward exceptions such as Japan, where Unicode itself is unpopular, and Python is too). It's not like the US uses UTF-8 and Canada uses UTF-16. You make Unicode decisions based on the OS and APIs that you're interacting with. And that's why we have this Linux dot convention for overriding what the locale would otherwise say so we can use Unicode. So your recommendation to use locales is actually a recommendation to mostly ignore locales, and just use the part after the dot as the name of the encoding you should be using. And to make wild-ass wrong guesses if there's no dot. Taking the locale "C" and interpreting it as the encoding "ASCII" is an example of a wild-ass wrong guess. But there are already environment variables that configure Python to use a particular encoding, without hacking it on top of archaic shit like locales. And harm is done by trying to infer the encoding from the locale, because of the complete wrongness of assuming the "C" locale means to use Python's "ASCII" encoding. The resulting behavior is not correct, and the reasoning for it is not correct. It's a bug. It will probably be fixed in one of the next two versions of Python. People run Python from cron jobs, from IDEs, from all sorts of places that don't set the locale the way the Ubuntu shell does, and get bafflingly inconsistent results. You can say "fix your locales then" all you want, but this is not an answer that makes Python more usable, and the developers have acknowledged this. Here's a particular example of where I think you're coming at this from the wrong direction, where I think you're taking the current behavior of Python as if it were actually some sort of intentionally-designed standard: > That's true for any locale, if a character comes up in the output that cannot be encoded in it - it should just use some reasonable substitution. And if you're actually dealing with binary data, then you should be reading and writing bytes objects, not printing strings, and then the whole question of encoding is moot. You don't encode things in a locale! You encode things in encodings! I'm definitely not talking about binary data, I'm talking about printing out perfectly normal characters like "ü". The locale "C" does not tell you anything about what encoding to use. Which is very different from Python's current assumption (which may go away in 3.7) that it's telling you to use ASCII and explode.