4 ms·

The Python developers plan to never break syntax or stdlib compatibility again. Core developer Nick Coghlan wants to see 4.0 within the decade but he wants it t

by rspeer 9y ago

The Python developers plan to never break syntax or stdlib compatibility again. Core developer Nick Coghlan wants to see 4.0 within the decade but he wants it to be "uneventful". [1] In that way, it would be a lot like Python 2.0, which only broke compatibility in obscure ways, but added new ways to do things. Here are some of my wild guesses at what features Python would want to commemorate with a 4.0 release: - An easy, Pythonic way to opt out of the Global Interpreter Lock. (The GIL clearly isn't going away because so much extension code needs it, but that doesn't mean all code has to suffer from the GIL for eternity.) - UTF-8 as the guaranteed default encoding, instead of trying to infer anything from stupid locale shit. (This wouldn't break any code that wasn't broken already.) - More built-in syntax to support NumPy. [1] https://www.curiousefficiency.org/posts/2014/08/python-4000.html https://www.curiousefficiency.org/posts/2014/08/python-4000....

int_19h 9y ago
> UTF-8 as the guaranteed default encoding, instead of trying to infer anything from stupid locale shit. This is already the case for source code, right? And I don't see where else it could apply. If you mean stdin/out, then those have to use locale to be compatible with other text-processing software (so that you can pipe things etc).
- takluyver 9y ago
  Reading and writing files. This catches people out, because on most Linux and Mac systems it defaults to UTF-8, so people assume UTF-8 is the default. But it's based on the locale, so one day someone runs your code on Windows, or on Linux with the basic ANSI C locale, and suddenly writing text to a file blows up. This is the number one thing I'd change if I were in charge of Python. ;-)
  - int_19h 9y ago
    I would actually rather change it to make the encoding required when opening a text file. Sometimes you do want UTF-8, but other times you do actually want locale encoding (because other text processing tools expect locale). And then "Explicit is better than implicit". That said, .NET has been defaulting to UTF-8 for text files since 1.0, and it seems to be working well in practice, so maybe it is a reasonable default after all.
- rspeer 9y ago
  Locales are not a real way of specifying an encoding for standard in and out. They were not designed for a world with Unicode and UTF-8 in it; they were designed for a world with limited character sets where your text data would probably not be sent outside of your country. Here are some reasons not to try to get your locale to tell you about UTF-8: - There is no standard for this. - Locale suffixes ".utf8" and ".UTF-8" are hacks by specific Linux distributions, and people want their Python code to work even if their system has not implemented this hack. - The locale "C" does not really mean you want your program to explode when it sees a non-ASCII byte. What Python does here does not promote compatibility in any way. - BSD has no UTF-8 locales and has never pretended that locales work with Unicode. - Windows' equivalent of locales is extremely deprecated and never actually supported UTF-8. The modern Windows APIs that deal with standard in and out correspond to Python's Unicode string type, not its bytes type; encoding the Unicode I/O into bytes is not Python's responsibility. - Really every operating system but Linux has this figured out, and in practice, Linux users want UTF-8 regardless of what their locale says. To be compatible with other text-processing software in 2017, you don't use locales, you use Unicode APIs and UTF-8.
  - int_19h 9y ago
    > Locale suffixes ".utf8" and ".UTF-8" are hacks by specific Linux distributions, and people want their Python code to work even if their system has not implemented this hack. I do want my Python code to work, yes. But I assure you that if my locale says ru_RU.KOI8-R, when I say "work", I don't mean "dump garbled stuff on my screen, because you output UTF-8 when I specifically asked you to use KOI8-R". I also don't see why the UTF-8 encoding specifier in locale is a "hack". UTF-8 is just that, an encoding. Locales are a generic mechanism for dealing with encodings. What makes UTF-8 special in that regard, and why is an UTF-8 locale a hack? > The locale "C" does not really mean you want your program to explode when it sees a non-ASCII byte. What Python does here does not promote compatibility in any way. That's true for any locale, if a character comes up in the output that cannot be encoded in it - it should just use some reasonable substitution. And if you're actually dealing with binary data, then you should be reading and writing bytes objects, not printing strings, and then the whole question of encoding is moot. > BSD has no UTF-8 locales and has never pretended that locales work with Unicode. That would be a surprise to my FreeBSD installation, which is running with en_US.UTF-8. Do you mean that the locales don't come generated by default? I'm not sure why it's a big deal ... I mean, Arch doesn't come with any locales generated by default, but I don't think anyone would claim that it doesn't support UTF-8. One other thing BSD does, is that it doesn't guarantee that wchar_t is Unicode in all locales (Linux/glibc does). But that's a different and orthogonal issue. > Windows' equivalent of locales is extremely deprecated and never actually supported UTF-8. The modern Windows APIs that deal with standard in and out correspond to Python's Unicode string type, not its bytes type; encoding the Unicode I/O into bytes is not Python's responsibility. Win32 doesn't have any string-based I/O functions, except for console. So if you're printing directly to the screen (and ignoring / not supporting redirection!), then yes, you can just invoke WriteConsole, give it a UTF-16 string, and be done with it. But if you're printing to a file, or to stdin/out as a handle, then you only have ReadFile/WriteFile, which require a byte array; you're expected to do your own encoding. By the way, and speaking of proper Unicode handling APIs - because of this disparity, and the desire to avoid a conversion unnecessarily (due to the lack of UTF-8 locale making it lossy) so as to allow printing out Unicode, Python actually has to resort to hacks to do this on Win32. Specifically, it has a special class - _io._Win32ConsoleIO - that is implemented using ReadConsole/WriteConsole. If output is a Win32 console, then that class is used, and Unicode is fully supported in the output. If output is redirected, then the standard implementation (that does encoding + Read/WriteFile) is used instead. > Really every operating system but Linux has this figured out, and in practice, Linux users want UTF-8 regardless of what their locale says. I strongly disagree. The users who want it but don't know how to configure it, are already using a distro that says that it should be UTF-8. The users who are using something that doesn't, either do it for a reason, or are knowledgeable enough to know what's going on and why, and deal with it accordingly. Either way, no harm is done by just using the locale setting. On Ubuntu etc, it will just be UTF-8, and everyone will be happy.

by rspeer 9y ago

int_19h 9y ago

> UTF-8 as the guaranteed default encoding, instead of trying to infer anything from stupid locale shit. This is already the case for source code, right? And I don't see where else it could apply. If you mean stdin/out, then those have to use locale to be compatible with other text-processing software (so that you can pipe things etc).

takluyver 9y ago
Reading and writing files. This catches people out, because on most Linux and Mac systems it defaults to UTF-8, so people assume UTF-8 is the default. But it's based on the locale, so one day someone runs your code on Windows, or on Linux with the basic ANSI C locale, and suddenly writing text to a file blows up. This is the number one thing I'd change if I were in charge of Python. ;-)
- int_19h 9y ago
  I would actually rather change it to make the encoding required when opening a text file. Sometimes you do want UTF-8, but other times you do actually want locale encoding (because other text processing tools expect locale). And then "Explicit is better than implicit". That said, .NET has been defaulting to UTF-8 for text files since 1.0, and it seems to be working well in practice, so maybe it is a reasonable default after all.
rspeer 9y ago
Locales are not a real way of specifying an encoding for standard in and out. They were not designed for a world with Unicode and UTF-8 in it; they were designed for a world with limited character sets where your text data would probably not be sent outside of your country. Here are some reasons not to try to get your locale to tell you about UTF-8: - There is no standard for this. - Locale suffixes ".utf8" and ".UTF-8" are hacks by specific Linux distributions, and people want their Python code to work even if their system has not implemented this hack. - The locale "C" does not really mean you want your program to explode when it sees a non-ASCII byte. What Python does here does not promote compatibility in any way. - BSD has no UTF-8 locales and has never pretended that locales work with Unicode. - Windows' equivalent of locales is extremely deprecated and never actually supported UTF-8. The modern Windows APIs that deal with standard in and out correspond to Python's Unicode string type, not its bytes type; encoding the Unicode I/O into bytes is not Python's responsibility. - Really every operating system but Linux has this figured out, and in practice, Linux users want UTF-8 regardless of what their locale says. To be compatible with other text-processing software in 2017, you don't use locales, you use Unicode APIs and UTF-8.
- int_19h 9y ago
  > Locale suffixes ".utf8" and ".UTF-8" are hacks by specific Linux distributions, and people want their Python code to work even if their system has not implemented this hack. I do want my Python code to work, yes. But I assure you that if my locale says ru_RU.KOI8-R, when I say "work", I don't mean "dump garbled stuff on my screen, because you output UTF-8 when I specifically asked you to use KOI8-R". I also don't see why the UTF-8 encoding specifier in locale is a "hack". UTF-8 is just that, an encoding. Locales are a generic mechanism for dealing with encodings. What makes UTF-8 special in that regard, and why is an UTF-8 locale a hack? > The locale "C" does not really mean you want your program to explode when it sees a non-ASCII byte. What Python does here does not promote compatibility in any way. That's true for any locale, if a character comes up in the output that cannot be encoded in it - it should just use some reasonable substitution. And if you're actually dealing with binary data, then you should be reading and writing bytes objects, not printing strings, and then the whole question of encoding is moot. > BSD has no UTF-8 locales and has never pretended that locales work with Unicode. That would be a surprise to my FreeBSD installation, which is running with en_US.UTF-8. Do you mean that the locales don't come generated by default? I'm not sure why it's a big deal ... I mean, Arch doesn't come with any locales generated by default, but I don't think anyone would claim that it doesn't support UTF-8. One other thing BSD does, is that it doesn't guarantee that wchar_t is Unicode in all locales (Linux/glibc does). But that's a different and orthogonal issue. > Windows' equivalent of locales is extremely deprecated and never actually supported UTF-8. The modern Windows APIs that deal with standard in and out correspond to Python's Unicode string type, not its bytes type; encoding the Unicode I/O into bytes is not Python's responsibility. Win32 doesn't have any string-based I/O functions, except for console. So if you're printing directly to the screen (and ignoring / not supporting redirection!), then yes, you can just invoke WriteConsole, give it a UTF-16 string, and be done with it. But if you're printing to a file, or to stdin/out as a handle, then you only have ReadFile/WriteFile, which require a byte array; you're expected to do your own encoding. By the way, and speaking of proper Unicode handling APIs - because of this disparity, and the desire to avoid a conversion unnecessarily (due to the lack of UTF-8 locale making it lossy) so as to allow printing out Unicode, Python actually has to resort to hacks to do this on Win32. Specifically, it has a special class - _io._Win32ConsoleIO - that is implemented using ReadConsole/WriteConsole. If output is a Win32 console, then that class is used, and Unicode is fully supported in the output. If output is redirected, then the standard implementation (that does encoding + Read/WriteFile) is used instead. > Really every operating system but Linux has this figured out, and in practice, Linux users want UTF-8 regardless of what their locale says. I strongly disagree. The users who want it but don't know how to configure it, are already using a distro that says that it should be UTF-8. The users who are using something that doesn't, either do it for a reason, or are knowledgeable enough to know what's going on and why, and deal with it accordingly. Either way, no harm is done by just using the locale setting. On Ubuntu etc, it will just be UTF-8, and everyone will be happy.