20 ms·
About every other month I have to help a biologist who has pulled down an older program from Github debug why it broke in their data only to find that they were
by epistasis 6y ago
About every other month I have to help a biologist who has pulled down an older program from Github debug why it broke in their data only to find that they were trying to run in Python 3 instead of Python 2. These are people that can write shell scripts, but don't program. It makes me aware of all the old code out there that is not used super frequently, but still solves people's problems, and would continue to if there hadn't been breaking changes.
As I write Python 3 now, I am less than impressed with the reasons for breaking compatibility. I'm told that it was necessary for proper bug-free Unicode code, but there's enough difficulty when using, say, gzip, and little enough documentation about the "proper" way to handle Unicode that I doubt that it will actually reduce Unicode-related bugs.
Python is still a great tool, but Python 3 really took the shine off for me.
- tyingq 6y agoI wonder if there's any merit to python3 kicking off a "is this script python2?" on some subset of common errors that crop up. Then it could tack on a "looks like you're trying to run a python 2 script on python 3" final error to stderr.
- minot 6y agoThat’s a great idea. Perhaps even offer to 2to3...
- hprotagonist 6y agohttps://www.python.org/dev/peps/pep-0394/ https://www.python.org/dev/peps/pep-0394/ so “helpful”!
- Alex3917 6y ago> little enough documentation about the "proper" way to handle Unicode that I doubt that it will actually reduce Unicode-related bugs. There are hundreds if not thousands of pages of unicode documentation. If anything the problem is that there is way too much for any normal person to actually understand it. For most people though, knowing when to normalize strings to NFC versus NFKC is all they need to not unknowingly accumulate tons of technical debt.
- epistasis 6y agoYou are right, there's tons of Unicode documentation out there. The problem is how to use encodings in Python. What parts of the standard library are going to be OK with Unicode, and which parts are OK with bytes, and more importantly which parts reject bytes/Unicode? For example, I coded along in Python 3, not changing and IO handling for the longest time, but when I Tried to use the gzip module I suddenly had to learn a whole bunch more knowledge about Unicode. I looked around for about 10 minutes for a some sort of best practices guide, but ij lieu of that just mucked with encode/decode functions until it worked. I don't know if I did conversions at the right spot, or where the guidelines are for changing between bytes and characters, but it didn't matter for my data at the time and o had a deadline so I couldn't really justify spending another hour at the time to figure it out. Maybe I'm just trying to justify my behavior to myself, but I think this sort of problem solving is the norm. I tend to try to adopt best practices and educate myself more than the other programmers I have worked with, if not more, and am the one introducing flake8 etc. to code reviews, so I'm not the worst sort of programmer, though I certainly wouldn't claim to be top-tier either.
- dehrmann 6y ago> What parts of the standard library are going to be OK with Unicode, and which parts are OK with bytes, and more importantly which parts reject bytes/Unicode? This is coming from someone who's written a lot of Java (Java's had the byte/char distinction since forever); the way Python 2 pretended byte strings are strings really was broken and not what you want. What the changes in Python 3 mean are that you have to actually think about if something is bytes or a string, and you have to be intentional in choosing encodings.
- csande17 6y agoI've recently been using some other languages that use a Python 2-esque "strings are byte-strings that are probably UTF-8" model, and I actually think it's a better choice than what Python 3 went with. Sure, you can argue that iterating over bytes is less "correct" than iterating over Unicode code-points. But iterating over code-points isn't very useful either -- maybe that code-point represents the letter B, but maybe it represents the third of seven characters and modifiers and zero-width joiners that make up one family emoji.
- duckerude 6y ago> little enough documentation about the "proper" way to handle Unicode that I doubt that it will actually reduce Unicode-related bugs I won't claim here that the compatibility break was actually worth it, but I am sure that it reduces Unicode-related bugs. It's easy to write Python 2 code that seems to work fine but immediately crashes when it has to process Unicode. But if your Python 3 code works at all it'll probably handle Unicode fine, or at least handle it better than naive Python 2 code would. Whenever I have to write Python 2 code that has to handle Unicode, I mainly just try to write it so that it would be able to run in Python 3.
- takeda 6y ago> It's easy to write Python 2 code that seems to work fine but immediately crashes when it has to process Unicode. But if your Python 3 code works at all it'll probably handle Unicode fine, or at least handle it better than naive Python 2 code would. Yeah, that's the biggest benefit, and the reason python 3 handles it better, is because it crashes early. When it it sees that you are making an incompatible conversion, it refuses to run without waiting for some specific unicode character to show it. > Whenever I have to write Python 2 code that has to handle Unicode, I mainly just try to write it so that it would be able to run in Python 3. Thankfully I no longer have to write python 3 code, but from my experience, it works the best to write the code for python 3, then adapting it (sometimes with help of six package and imports from `__future__`) to work with 2.7. It is easier than do it the other way.
- sys_64738 6y agoYou should never break backwards compatibility in a language. It's its undoing.
- formerly_proven 6y agoYet. https://www.jetbrains.com/lp/python-developers-survey-2019/#PythonVersions https://www.jetbrains.com/lp/python-developers-survey-2019/#... https://www.jetbrains.com/research/python-developers-survey-2017/#python-3-adoption https://www.jetbrains.com/research/python-developers-survey-...
- sys_64738 6y agoProduction code must be maintained. I've seen many shops with legacy 2.7 code that is stuck due to breaking compatibility in 3.x. Put it this way, if my production code suddenly needs to be patched due to non-security changes then that has to come out of somebody's budget. What else could we be doing with that budget than patching code from 2.7 to 3.8. How do you justify that to your budget director?
- joshuamorton 6y ago"We can either do this now, slowly and carefully, or do it quickly and haphazardly when some dependency has a cve only patched in 3.x"
- gtaylor 6y agoAs long as you are willing to spend some of that budget continuously patching and maintaining the old 2.x interpreter to address security vulns as they are discovered (IF they are discovered), sure. No need to upgrade!
- formerly_proven 6y agoIf you haven't had the resources to port a codebase to a new language version over the course of twelve years, you clearly haven't had the resources to maintain it.
- 6y ago
- jbay808 6y agoWhen I was first starting, I almost gave up on Python because the print "Hello world" script that I copy-pasted from an online tutorial failed to run. The hardest part (for me) of getting started with a new language is setting up the environment to run properly... Installing libraries, configuring environment variables, pointing the IDE to the interpreter, etc. So of course that's where I assumed my mistake was. It was only much later that I realized that I was running Python 3 (newer is better right?) and the tutorial was written when Python 2 was the latest version. So the print statements changed. And of course, "print" is in everyone's first program. So it broke the experience for all first time users.
- norrius 6y ago> So it broke the experience for all first time users. To be fair, Python 3 provides a very clear error message if you try to use print as a statement, although I don't know when exactly that was added: $ echo 'print "Hello World"' > test.py && python3 test.py File "test.py", line 1 print "Hello World" ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Hello World")?
- NullPrefix 6y ago>To be fair, Python 3 provides a very clear error message To be fair, nobody reads error or warning messages.
- mylons 6y agothen those people are helpless? you can lead a horse to water.
- kshacker 6y agoI agree with the parent actually. My first python program was a json read and print something. 4 lines, maybe 5. I had a need to parse json and print something specific and I suspected I would be able to do it in python. After 4 hours, I was able to and I felt happy. This is not to complain about that 4 hours. But when I think back at that 4 hours and how much time will it take to do it today, it reminds me of all the learnings. Is it load or loads? Which one is from file? What is a json object? Do I need to quote? If so how? Why does it work with the small block but not the big block? Is it the nesting? Am I doing something wrong? I think the same is true of hello world. Hello world is a newbie presumably who has not written a single line of code. Well you copy and paste and it does not run. Yes the error is there but hey I copied working code : did I install something wrong? Should I check some website? It says it is python but is it? You can't imagine the questions that will come to a newbie. We need to be mindful of this rather than have a stack overflow mindset.
- donor20 6y agoAgreed completely. There were so many routes forward that didn't involve breaking everything but still would have allowed fewer unicode bugs (maybe). And they could have done a lot more to make the transition easier if they wanted to - some of it was just in your face type breakage to a prove a point without considering some underlying use cases (a lot of internet protocols, headers etc still ASCII). What is nuts to think about - if tweaking some unicode handling is worth creating an incompatible version, what happens when some random python developer has another itch to scratch that they consider so important it is worth break a language over. The idea that these types of fixes require breaking compatibility is absolutely insane if you want to address a larger / longer term market. Complain all you want about Microsoft, C etc. Even go of all languages (because google has an otherwise terrible rep in this area!!). But some of these languages and companies really do seem to be able to deliver or intend to deliver longer term compatibility. When businesses and lifetimes are measured in decades (not months) it is VERY nice to build on stuff that has longer support horizons.
- joshuamorton 6y agoPython 2.7 was supported for 9+ years, python 3 was released in 2008, with development beginning before then. You had almost 15 years of warning from initial announcement to eol. Python3 predates golang. > And they could have done a lot more to make the transition easier if they wanted to - some of it was just in your face type breakage to a prove a point without considering some underlying use case Such as? > (a lot of internet protocols headers etc still ASCII). What do you mean here? Py3 correctly handles http headers encoded as ascii bytes. Yes the decision to make http headers ascii bytes and not basically anything else is bad in hindsight, but that ships sailed.
- ornxka 6y ago>You had almost 15 years of warning from initial announcement to eol. Why should I have to do labor to fix things that already work?
- CogitoCogito 6y ago
- jghn 6y agoI struggle with this whenever I think about it. On one hand I agree with all of the points you make. But on the other hand, it didn't have to be this way. Python 3 has been out for 12 years now. If people hadn't dragged their feet on converting the mess wouldn't be nearly as large as it wound up being.
- epistasis 6y agoIMHO 12 years is a very short time! Of the user-facing programs I use (mostly CLI), the vast majority of the code That gets executed was written far longer ago than that. Writing some code and not have it executable a dozen years later is a terrible terrible situation. And it wouldn't be this way if Python 2 wasn't EOL. I'm going to always need to keep a copy of Python 2 around. Always. And if I'm going to change old, working code, I'm probably going to rewrite it in a new language.
- jghn 6y agoAs with my previous post I'm torn on topics like this as again I agree with everything stated. And yet, Python was relatively niche in the early aughts so had only really been in the mainstream for a handful of years before 3.0 came out. What %age of live python code in the wild was written after 3.0 was released? I'd wager that's a very large number. But yet I get why things are the way they are. Heck, it was only last year that I switched to using Python 3. I use it so rarely that it was just easier to stick with the Python 2 which was already installed on my system.
- cortesoft 6y agoI am not sure why the length of time matters. They were spending the time doing other important things. They were making the rational decision to work on something that was important and WASN'T already working. The python 2 code was working fine, it didn't have bugs and it didn't need any new features. Other code either didn't exist and needed to be written or had bugs that needed to be fixed. It makes sense that changing working code never made it high enough on the priority list to get worked on.
- umvi 6y ago
- mtalantikite 6y agoThe whole Python 2 vs 3 thing turned me off when the split first happened when it was “Python 3000” back in, what, 2008 or so. Every time I‘ve had to do something with Python since then there has inevitably been some sort of incompatibility issue that popped up, although it’s been quite a lot better these past few years. It always left a bitter taste in my mouth and I always feel like I’m fighting with Python every time a project comes up where I have to use it. There are so many other languages that are a pleasure to use and I’ve always wanted to like Python, I just feel like I’m cursed to always find Python needlessly annoying.
- Ankaios 6y agoPersonally, I don't mind some inconvenience if language designers occasionally need to make breaking changes to allow improvements, and I've had to port enough scientific Fortran code to newer languages to limit my sympathy for porting from Python 2 to 3. Computer languages aren't static, and sometimes I need to do some work to keep up as they evolve.
- hdjdbtbgwjsn 6y agoI bet the old Fortran versions are supported and compile decades after the 'newer' languages stop being supported...
- tjalfi 6y agoI have compiled Fortran programs from the 70s on modern platforms without changing a line. The compiler, OS, and CPU architecture had all disappeared but the programs still worked correctly.
- hdjdbtbgwjsn 6y agoI've 'modernised' such an old program then come back a few years later, realised the 'modern' ones don't run anymore and reverted to the old Fortran version.
- TheRealKing 6y agoThat is likely a compiler issue. Besides being modern, Fortran 2018, 2008, 2003 have no backward incompatibility issues.
- hdjdbtbgwjsn 6y agoYou misunderstand. I 'modernized' to Python (similar to the GP). Then the Python version stopped being supported so I reverted to Fortran..
- occamrazor 6y ago
- takeda 6y ago> As I write Python 3 now, I am less than impressed with the reasons for breaking compatibility. I'm told that it was necessary for proper bug-free Unicode code, but there's enough difficulty when using, say, gzip, and little enough documentation about the "proper" way to handle Unicode that I doubt that it will actually reduce Unicode-related bugs. This was absolutely essential. Python 2 tried to fix by introducing unicode type. It was hacky, and because they looked for a shortcut, they did a very bad thing by introducing implicit casting (so python 2 automatically converted between str and unicode). They did implicit casting because they didn't made the effort of fixing the stdlib to properly handle unicode. At that point not only there was a problem that str represent two different things (bytes and text), but also authors that were using unicode (to make their code futureproof) make their life actually harder (in python2 even when you know how unicode works, implementing it correctly is a minefield, it is so easy to do it incorrectly and end up with code that works, but blow in special circumstances). Python 3 basically decided to rip the bandaid off and do it correctly. And that's what happened. When you program in python 3 you no longer have to think much about unicode. To convert between byes you need to explicitly convert from text to bytes (.encode()) and bytes to text (.decode()) but this is not a concern unless you are doing some kind of serialization. While the breaking unfortunately was necessary (a lot of python 2 code, while it runs it is broken), I'm wondering if it was possible to do it a bit more developer friendly. I think things would be much easier if instead of changing the behavior of `str` type, Guido would introduce a `text` type (along with `bytes` type), then later deprecate it. I think that would help a lot. Although in that case modifying stdlib would be unavoidable, so it would be a lot of work for them. BTW: MyPy was also a tool that could help with the migration, but once again they IMO dropped a ball. Basically when MyPy is used in Python 2 mode, it follows Python 2 typing system, which means it still allows for implicit casting. I think what was really needed is to highlight these issues to allow for fixing python 2 code before doing the migration. The way mypy worked required you to first migrate your code to Python 3 and only after that you could start fixing the text/bytes issues.
- golergka 6y ago> These are people that can write shell scripts, but don't program. Being able to figure out that you're using a different major version of Python is definetly expected of today's biologist. This is part of the profession now.
- korijn 6y agoAs if all those shell scripts are portable across time and distributions. Or PHP code. Or really any other code. How many builds of your software are truly reproducible? Python 2/3 incompatibility is always magnified for some reason, but it's nothing out of the ordinary in my experience. Didn't we just recently have a highly upvoted/heavily debated HN submission on semver? Enough said...
- veselin 6y agoOn a similar note, both Java and C# made an early and mostly incorrect decision to support utf-16 in the native string and char types. This choice also made some programmers to believe that every unicode character fits in one char, causing bugs, etc. Yet, I have not seen a rush to "fix" any of them the Python way.
- takeda 6y agoYou are mixing the internal representation of text with the change to make distinction between text and bytes. The python incompatibility wasn't really the Unicode it was that older versions of python mixed text with bytes, and because of that a lot of python 2 code is actually broken. It might run fine, but as soon as input are Unicode characters then many python 2 applications crash. Python 2 tried to implement Unicode and implemented an `unicode` type. But it actually made things worse due to them adding implicit casting. The proper way to fix the issue was actually clearly making the distinction what is text and what are bytes, something that was in both Java, C# (both of them are younger than Python BTW, the Unicode was in works when Python was released). he UTF-16 used by Java and C# as I said is just an internal representation of the text, and that's actually trivially changed since normally those things are not accessible for developers. The only way where it could somewhat leak is default encoding, and maybe names of the methods from stdlib. But those things are really easy to change without much breakage. Side note, Python internally stores text in many different ways if the text is only out bytes it uses traditional C-string (for performance). Otherwise it also stores as UTF-8, UTF-16, UTF-32 based on characters and requested conversion. So those conversion you said Java or C# did Python 3 does all the time (based on need) without breaking running code.
- atoav 6y agoAs somebody who programs a lot of python for roughly a decade now I really like Python 3 (although I tone quite a while to switch over completely). Unicode handling really got much more straightforward (which you might notice only if you need to deal with Unicode, which many people in the anglosphere don't do all that often). I like the new formatting options, type annotations all great stuff. And quite frankly most python 27 code I wrote worked after adding brackets to the print statements.
- bsder 6y ago> Python is still a great tool, but Python 3 really took the shine off for me. People forget that part of the reason why the Python maintainers were so snotty about Python 2 to Python 3 was because they went through this before. Anyone remember the gnashing of teeth that occurred when trying to drag people off of Python 1.5.2? I sure do. I want to throw anybody complaining about backward compatibility in 2020 into Python 1.5.2 because THAT'S what you're advocating for.