4 ms·

> No, it's not even remotely worth the amount of confusion and work it caused. Have you worked with pervasively non-ASCII text in Python 2? Like on a machine w

> No, it's not even remotely worth the amount of confusion and work it caused. Have you worked with pervasively non-ASCII text in Python 2? Like on a machine where any file might suddenly turn out to be non-ASCII (even if it’s only in a comment), not just data inside a few carefully-patrolled fences? Outside of a few well-behaved libraries (Flask), my experience was that it was utterly impossible. More than half of my time debugging straightforward code was spent chasing down and fixing UnicodeErrors, knowing that some will still remain. And that includes code that used nothing but the standard library. Now you might think that’s not important, and though I disagree (especially when talking about “most users”) that would be fair. But it is a substantial improvement going from 2 to 3.

zarzavat 4y ago
There is another universe where Python 3 only fixed the unicode strings, and the transition was a huge success and over in a couple of years. The print() function was absolutely not worth all the carnage it caused. They could have just kept the print statement around and introduced a printing function with a different name, for example. The fact that they removed syntax for Python 3 meant that all kinds of scientific computing code had to be transitioned even if it didn't use any strings!
- chaosite 4y ago
  Wait, if it didn't use any strings, then it didn't use the print statement either, right?
  - zarzavat 4y ago
    For printing it's not a problem because a string containing unicode will be transparently printed in both Python 2 and 3 (at least on Linux/Mac, not sure how Windows deals with it). It's also not a problem if you only have ASCII strings.
- IgorPartola 4y ago
  The print function was the simplest part of the transition. For one you could find and replace all instances of it fairly safely across your codebase and you could quickly discover any places where things broke because of a bad replace. That part took like an hour for a large codebase. The Unicode handling on the other hand took ages because string handling is not easily searchable and replacing it requires understanding what the piece of code actually does. So you have it exactly backwards: if the only migration was the print function then it would be a quick and relatively painless process. But fixing the hard thing which was the broken str/byte model was the hard transition. I am curious if you actually went through this transition or if your opinion on this is more based on observation of others’ work.
  - zarzavat 4y ago
    Python is a scripting language. Imagine if bash suddenly changed its syntax so that that instead of printing using echo, you now have to write echo(). Sure it’s just a pair of parentheses. But you probably have thousands of shell scripts on your computer. Are you really going to get your hands dirty and risk editing them without knowing what they do? Of course not, you’re just going to revert back to the version of bash that doesn’t require parentheses and sit on it for as long as you possibly can. Code exists in three states: actively maintained, done/mature, and obsolete. Python 3 attacked mature code, forcing developers to either go back into active maintenance or to obsolete it early. I did transition some Python 2 code to 3. But mostly it was uneconomical and risky, so the vast majority of it sat on Python 2.7 until it was no longer needed. I feel that most of the people who had a positive experience with Python 3 were people doing webdev using Django. They were working on monolithic actively maintained projects and so transitioning to v3 was just a normal maintenance activity.
    - IgorPartola 4y ago
      You are correct that my experience is mostly though not exclusively we Django projects. I guess maintaining your code pays off in the long run. Again, you are arguing that running 2to3 and testing your code was hard. I guess that may be true. But having hunted for a ton of subtle Unicode bugs that problem was very worth fixing.
- josefx 4y ago
  > There is another universe where Python 3 only fixed the unicode strings, and the transition was a huge success and over in a couple of years. Didn't the unicode changes go on until at least Python 3.6? The issue that most C APIs did not care about their byte strings having any encoding at all seemed to cause one headache after another, especially on Posix, where everything touching the operating system could return random bytes. Reducing the scope to unicode wouldn't have sped up the transition one iota.
wink 4y ago
Not the one you replied to, but we were doing things mostly without strings (audio processing, stuff with bits) and python 2 worked perfectly fine. We had started this project as late as 2013 because the important libraries had not been ported yet and when the stuff wrapped up in 2017... I'm not sure if we could have switched by then. So personally I'm ok with Python 3 but I actually never ran into the described problems despite using Python 2 for years, so it was at least not a universal problem that needed to be solved in such a breaking way.
- harriet1 4y ago
  [dead]
- mananaysiempre 4y ago
  So not a single Cyrillic path in your INI or CSV files, or in the user input (CP_ANSI) to be printed onto the Windows console (CP_OEM, unless you’re in IDLE or redirecting the output, in which case CP_ANSI)? (The standard library’s CSV module was particularly atrocious at Unicode as I remember.) Not a single Chinese comment in your hardware descriptions? I envy you, but that’s a still a very specific corner of the world.
  - wink 4y ago
    We were working exclusively with filenames (and files) that we generated ourselves, moving them around in our own infrastructure, and writing all of our own configs, yes :P I didn't insinuate that there weren't problems or that I wanted to keep Python 2 - just that it was perfectly normal to use it and not run into problems. I think I only remember a handful of encoding problems with my other python 2 adventures...
- harriet1 4y ago
  [dead]
wokwokwok 4y ago
Look, the breaking change to python3 was a disaster. There’s no need to sugar coat it. It was incompetently managed, in a way that made the technical success of the work look like a failure. That’s on them. They screwed up. I lived through it; I feel no particular need to smile and nod and say “it wasn’t that bad”. It was bad. Things are good now! Python 3 is great, it’s well supported and the people involved all learnt a lot about how important managing communities is, as well as writing good code. So, all round, a success! …but, if you don’t acknowledge failures in the past, you’re doomed to repeat them. > Now you might think that’s not important I don’t think your experience with Python 2 is not important; I had similar issues with it. No one is seriously going back to Python 2 at this point. However, the process is a lesson worth studying; and the take away is not “what a technical success!” … it is: never do this. Big scary breaking migrations are bad, they hurt your community, and leave bitterness behind them. Even if the result is better, technically; there are other factors that need to be considered too.
- newjersey 4y ago
  > No one is seriously going back to Python 2 at this point. No, the community is stupid and wrong. My machine learning professor (actual professor, not a teaching assistant) only used Python 2 in 2017. Imagine how many people had their opinions shaped by someone like them. > … it is: never do this. By that standard, we would never get things like angular 2 or Raku. Would it be better if we renamed Python 3 to some other name and pretended it was a completely new language?
  - IgorPartola 4y ago
    IMO Angular 2 was a huge fail in usability. Angular 1 was a joy to use and 2 was a disaster in terms of writing code in it. However, I am firmly in the Python 3 camp. It is a better language (all the newfangled bullshit notwithstanding).
  - pjc50 4y ago
    > Would it be better if we renamed Python 3 to some other name and pretended it was a completely new language? Quite possibly yes? It doesn't run python2 code, it can't be inter-linked with python2 code, so is it really the same language? The hybrid dialect that works in both does exist but you have to have compatibility shims in a few places. By comparison, most C compilers will let you use C89 and C17 code in the same project provided you set the compiler options, and C# provided the "netstandard" target for libraries to smooth the transition between Framework and Core runtimes.
KptMarchewa 4y ago
Agreed. Unicode change alone would be sufficient do to Python 3.
jeltz 4y ago
Yes, I have and it was simple but with plenty of footguns for developers who had not groked how you were supposed to work with strings. I am happy we moved away from that but I am not happy with the solution Python 3 picked. I think e.g. Ruby 1.9 came up with a nicer solution to the problem and one which broke much less code.
mort96 4y ago
Python 2 strings pretty much just worked with UTF-8, didn't it? Or did they manage to make Python 2's string handling worse than C's?
naniwaduni 4y ago
Uh, Python 2 was aboslutely fine at working with non-ASCII text as long as the only operations you try to do on it are "take the length" and "reproduce a string verbatim". Notably, it did not forcibly explode when encountering a character appearing in a comment or string literal or whatever that couldn't be interpreted as utf-8. This was a real step back for code that, oh, say, used filesystem interfaces to look at untrusted filenames.
- mananaysiempre 4y ago
  Not if “reproduce the [possibly Unicode] string verbatim” includes concatenating it with others in order to insert it into some sort of context, that will absolutely blow up if you’re not careful and unfortunate input data comes in (I hope that happens before the code enters production!). For comments and string literals in Python code, note that Python 3 changed the default from Latin-1 to UTF-8, but you can still use -*- coding: -*- to change that. Bytes literals outside of ASCII range you’ll still have to escape. As for program input that’s possibly invalid UTF-8 but not in places you care about, you’ll need to set errors='surrogateescape' or similar explicitly (or possibly set an 8-bit encoding like Latin-1, though not if there’s a chance of UTF-8 in places you do care about). Paths are a bit painful, yes, both because there’s nothing on Unix systems that precludes /home/alice having filenames in UTF-8 and /home/boris in KOI8-R, and because NT paths are not byte sequences at all (they are WCHAR, that is 16-byte-number, sequences, in no way required to be representable in the current CP_ANSI). Having uninterpreted byte sequences as filenames would work to solve the former issue, and it does: I think most functions in os will accept bytes paths as input and treat that as a signal that you want bytes paths as output (if any). For the latter the cleanest solution is probably an abstract Path type—and you get one, in the standard library’s pathlib. That didn’t work all that well in early Python 3 versions, but it does now. (And IIUC inspired Rust’s abstract-type solution to the problem.)
  - naniwaduni 4y ago
    This is true, almost all "string" operations require you to know extra information about the string's "type"—which, yes, includes its encoding. Python 3 takes this fact and runs with the stance that strings should, by default, carry their encoding with them (and that encoding is specifically, more or less, a sequence of quasi-Unicode code points with unspecified internal representation). But this is an intensely half-baked solution to the wrong problem, because you need to know much more about a string than its encoding for just about any string operation you'd want to do. Just with Unicode code point concatenation, you can run into plenty of trouble with: - strings containing control characters - strings with unpaired surrogates (which okay, shouldn't really be there, but get used for various reasons anyway) - strings containing combining characters - strings in different natural languages - strings in languages with ligatures - strings with ligature-like emoji (and this is obviously not remotely close to exhaustive.)

zarzavat 4y ago

There is another universe where Python 3 only fixed the unicode strings, and the transition was a huge success and over in a couple of years. The print() function was absolutely not worth all the carnage it caused. They could have just kept the print statement around and introduced a printing function with a different name, for example. The fact that they removed syntax for Python 3 meant that all kinds of scientific computing code had to be transitioned even if it didn't use any strings!

chaosite 4y ago
Wait, if it didn't use any strings, then it didn't use the print statement either, right?
- zarzavat 4y ago
  For printing it's not a problem because a string containing unicode will be transparently printed in both Python 2 and 3 (at least on Linux/Mac, not sure how Windows deals with it). It's also not a problem if you only have ASCII strings.
IgorPartola 4y ago
The print function was the simplest part of the transition. For one you could find and replace all instances of it fairly safely across your codebase and you could quickly discover any places where things broke because of a bad replace. That part took like an hour for a large codebase. The Unicode handling on the other hand took ages because string handling is not easily searchable and replacing it requires understanding what the piece of code actually does. So you have it exactly backwards: if the only migration was the print function then it would be a quick and relatively painless process. But fixing the hard thing which was the broken str/byte model was the hard transition. I am curious if you actually went through this transition or if your opinion on this is more based on observation of others’ work.
- zarzavat 4y ago
  Python is a scripting language. Imagine if bash suddenly changed its syntax so that that instead of printing using echo, you now have to write echo(). Sure it’s just a pair of parentheses. But you probably have thousands of shell scripts on your computer. Are you really going to get your hands dirty and risk editing them without knowing what they do? Of course not, you’re just going to revert back to the version of bash that doesn’t require parentheses and sit on it for as long as you possibly can. Code exists in three states: actively maintained, done/mature, and obsolete. Python 3 attacked mature code, forcing developers to either go back into active maintenance or to obsolete it early. I did transition some Python 2 code to 3. But mostly it was uneconomical and risky, so the vast majority of it sat on Python 2.7 until it was no longer needed. I feel that most of the people who had a positive experience with Python 3 were people doing webdev using Django. They were working on monolithic actively maintained projects and so transitioning to v3 was just a normal maintenance activity.
  - IgorPartola 4y ago
    You are correct that my experience is mostly though not exclusively we Django projects. I guess maintaining your code pays off in the long run. Again, you are arguing that running 2to3 and testing your code was hard. I guess that may be true. But having hunted for a ton of subtle Unicode bugs that problem was very worth fixing.
josefx 4y ago
> There is another universe where Python 3 only fixed the unicode strings, and the transition was a huge success and over in a couple of years. Didn't the unicode changes go on until at least Python 3.6? The issue that most C APIs did not care about their byte strings having any encoding at all seemed to cause one headache after another, especially on Posix, where everything touching the operating system could return random bytes. Reducing the scope to unicode wouldn't have sped up the transition one iota.

wink 4y ago

Not the one you replied to, but we were doing things mostly without strings (audio processing, stuff with bits) and python 2 worked perfectly fine. We had started this project as late as 2013 because the important libraries had not been ported yet and when the stuff wrapped up in 2017... I'm not sure if we could have switched by then. So personally I'm ok with Python 3 but I actually never ran into the described problems despite using Python 2 for years, so it was at least not a universal problem that needed to be solved in such a breaking way.

harriet1 4y ago
[dead]
mananaysiempre 4y ago
So not a single Cyrillic path in your INI or CSV files, or in the user input (CP_ANSI) to be printed onto the Windows console (CP_OEM, unless you’re in IDLE or redirecting the output, in which case CP_ANSI)? (The standard library’s CSV module was particularly atrocious at Unicode as I remember.) Not a single Chinese comment in your hardware descriptions? I envy you, but that’s a still a very specific corner of the world.
- wink 4y ago
  We were working exclusively with filenames (and files) that we generated ourselves, moving them around in our own infrastructure, and writing all of our own configs, yes :P I didn't insinuate that there weren't problems or that I wanted to keep Python 2 - just that it was perfectly normal to use it and not run into problems. I think I only remember a handful of encoding problems with my other python 2 adventures...
harriet1 4y ago
[dead]

wokwokwok 4y ago

Look, the breaking change to python3 was a disaster. There’s no need to sugar coat it. It was incompetently managed, in a way that made the technical success of the work look like a failure. That’s on them. They screwed up. I lived through it; I feel no particular need to smile and nod and say “it wasn’t that bad”. It was bad. Things are good now! Python 3 is great, it’s well supported and the people involved all learnt a lot about how important managing communities is, as well as writing good code. So, all round, a success! …but, if you don’t acknowledge failures in the past, you’re doomed to repeat them. > Now you might think that’s not important I don’t think your experience with Python 2 is not important; I had similar issues with it. No one is seriously going back to Python 2 at this point. However, the process is a lesson worth studying; and the take away is not “what a technical success!” … it is: never do this. Big scary breaking migrations are bad, they hurt your community, and leave bitterness behind them. Even if the result is better, technically; there are other factors that need to be considered too.

newjersey 4y ago
> No one is seriously going back to Python 2 at this point. No, the community is stupid and wrong. My machine learning professor (actual professor, not a teaching assistant) only used Python 2 in 2017. Imagine how many people had their opinions shaped by someone like them. > … it is: never do this. By that standard, we would never get things like angular 2 or Raku. Would it be better if we renamed Python 3 to some other name and pretended it was a completely new language?
- IgorPartola 4y ago
  IMO Angular 2 was a huge fail in usability. Angular 1 was a joy to use and 2 was a disaster in terms of writing code in it. However, I am firmly in the Python 3 camp. It is a better language (all the newfangled bullshit notwithstanding).
- pjc50 4y ago
  > Would it be better if we renamed Python 3 to some other name and pretended it was a completely new language? Quite possibly yes? It doesn't run python2 code, it can't be inter-linked with python2 code, so is it really the same language? The hybrid dialect that works in both does exist but you have to have compatibility shims in a few places. By comparison, most C compilers will let you use C89 and C17 code in the same project provided you set the compiler options, and C# provided the "netstandard" target for libraries to smooth the transition between Framework and Core runtimes.

KptMarchewa 4y ago

Agreed. Unicode change alone would be sufficient do to Python 3.

jeltz 4y ago

Yes, I have and it was simple but with plenty of footguns for developers who had not groked how you were supposed to work with strings. I am happy we moved away from that but I am not happy with the solution Python 3 picked. I think e.g. Ruby 1.9 came up with a nicer solution to the problem and one which broke much less code.

mort96 4y ago

Python 2 strings pretty much just worked with UTF-8, didn't it? Or did they manage to make Python 2's string handling worse than C's?

naniwaduni 4y ago

Uh, Python 2 was aboslutely fine at working with non-ASCII text as long as the only operations you try to do on it are "take the length" and "reproduce a string verbatim". Notably, it did not forcibly explode when encountering a character appearing in a comment or string literal or whatever that couldn't be interpreted as utf-8. This was a real step back for code that, oh, say, used filesystem interfaces to look at untrusted filenames.

mananaysiempre 4y ago
Not if “reproduce the [possibly Unicode] string verbatim” includes concatenating it with others in order to insert it into some sort of context, that will absolutely blow up if you’re not careful and unfortunate input data comes in (I hope that happens before the code enters production!). For comments and string literals in Python code, note that Python 3 changed the default from Latin-1 to UTF-8, but you can still use -*- coding: -*- to change that. Bytes literals outside of ASCII range you’ll still have to escape. As for program input that’s possibly invalid UTF-8 but not in places you care about, you’ll need to set errors='surrogateescape' or similar explicitly (or possibly set an 8-bit encoding like Latin-1, though not if there’s a chance of UTF-8 in places you do care about). Paths are a bit painful, yes, both because there’s nothing on Unix systems that precludes /home/alice having filenames in UTF-8 and /home/boris in KOI8-R, and because NT paths are not byte sequences at all (they are WCHAR, that is 16-byte-number, sequences, in no way required to be representable in the current CP_ANSI). Having uninterpreted byte sequences as filenames would work to solve the former issue, and it does: I think most functions in os will accept bytes paths as input and treat that as a signal that you want bytes paths as output (if any). For the latter the cleanest solution is probably an abstract Path type—and you get one, in the standard library’s pathlib. That didn’t work all that well in early Python 3 versions, but it does now. (And IIUC inspired Rust’s abstract-type solution to the problem.)
- naniwaduni 4y ago
  This is true, almost all "string" operations require you to know extra information about the string's "type"—which, yes, includes its encoding. Python 3 takes this fact and runs with the stance that strings should, by default, carry their encoding with them (and that encoding is specifically, more or less, a sequence of quasi-Unicode code points with unspecified internal representation). But this is an intensely half-baked solution to the wrong problem, because you need to know much more about a string than its encoding for just about any string operation you'd want to do. Just with Unicode code point concatenation, you can run into plenty of trouble with: - strings containing control characters - strings with unpaired surrogates (which okay, shouldn't really be there, but get used for various reasons anyway) - strings containing combining characters - strings in different natural languages - strings in languages with ligatures - strings with ligature-like emoji (and this is obviously not remotely close to exhaustive.)