13 ms·

Incrementally migrating over one million lines of code from Python 2 to Python 3

91 points by el_duderino 8y ago

danpalmer 8y ago
Nice write up. At Thread we did a similar thing. Admittedly our codebase is ~10% as big, but incrementally adding linters for incompatible code, and keeping the CI green, helped loads. We only had about 2-3 days of engineering time to ship the final version, the rest was done in 10% time.
jsilence 8y ago
Why does the desktop client have one million lines of code?
- danpalmer 8y ago
  Not to be facetious, but why wouldn't it? It has to find and diff files, coordinate a very reliable file upload of many files, as fast as possible. It has to understand enough about the content of those files to be able to do useful things with them. It has to reliably update itself again and again, from possibly very old versions, it has to communicate with an ever changing API. It has to have enough analytics in it to support product development, error reporting, understanding how users use it, how the product needs to evolve over time. It needs to integrate deeply with all major OSes. ...I'm sure I've missed some things. 1 million lines of code sounds like the right ballpark to me though.
  - sametmax 8y ago
    Plus the UI, the file manager intagration, the daemon, the authentification, the cache... All that cross platform.
    - saagarjha 8y ago
      None of those should be written in Python, except for perhaps the cache handling code. The rest should be platform-specific.
      - rmdashrfstar 8y ago
        Why?
        saagarjha 8y ago
        Because they're platform specific. For example, on macOS, the UI and file manager integration should be done in Swift or Objective-C(++), the authentication in some C-compatible language to match the security frameworks, and the daemon in some language that isn't single threaded or garbage collected.
        chrisseaton 8y ago
        You’re talking about that it uses platform specific APIs. You can use these APIs from Python, and then there is all the logic which decides how to use the APIs and that can be in Python as well.
        saagarjha 8y ago
        > You can use these APIs from Python You can do it hand-written assembly too, but usually it's more trouble than it's worth.
        chrisseaton 8y ago
        I think what we've worked out what you mean is just that if you were doing it you'd chose to do it in Objective C etc. That's not quite the same as anyone else 'should' do it.
        sametmax 8y ago
        Not only you can do all that confortably and productively in python, but you should. It's crossplatform, the task has not bottleneck for this vm, and it takes complexity down to one language. One language that as the very specific characteristic of being the best at nothing, but pretty good at everything, which in such an heterogenous software is what you want.
- bdcravens 8y ago
  I wonder if their code is over 1M, or if they're doing what you see in a lot of marketing, which is count all the lines from included packages.
  - rhizome 8y ago
    Who markets on the basis of LOC?
    - chillydawg 8y ago
      Dropbox, right now. These blogs are always with an eye to recruitment.
      - chrisseaton 8y ago
        Why would a massive legacy codebase be a recruitment turn-on?
        pmoriarty 8y ago
        Because then you can put on your resume that you have experience working on a massive codebase.
        deleted 8y ago
        [deleted]
- barkingcat 8y ago
  million lines isn't really that much
rhacker 8y ago
Is python finally getting past the 2/3 jump?
- dangoor 8y ago
  Python 2 EOL on January 1, 2020 is pushing a lot of folks to seriously look at this. This is a big thing we're looking at right now at Khan Academy.
- Alex3917 8y ago
  It's already been since 2015. Every major Python dependency has been Python3 compatible for several years, and many have already gone Python3 only. If you have a good test suite it's pretty easy to migrate, you can easily do a couple thousand lines per day. There were a few third-party testing tools that used to be fairly popular that are no longer well maintained, so if your tests are written in those then you might be looking at re-writing the app from scratch. But otherwise it's fairly straightforward to port. Even if you did your string encoding just using guess-and-check, which most people did, it still doesn't take all that long to upgrade as long as there are tests.
- lincolnq 8y ago
  I would say yes. I'm pretty sure most people are writing all new projects exclusively in python3, except if you need them to run on old distros without any dependencies (lots of distributions shipped for years with python2 but not python3 installed by default). The ecosystem of 3 is very mature - I've been running my servers' production code for 5+ years in python3 and had no major issues with that choice. The only thing left is "legacy codebases".
  - Jare 8y ago
    IIRC osx still ships python2 only with the OS.
- sametmax 8y ago
  It has for 2 years now. 3.6 was the tipping point, such a fantastic release. See https://www.jetbrains.com/research/devecosystem-2018/python/ https://www.jetbrains.com/research/devecosystem-2018/python/ For 2018 stats
lincolnq 8y ago
It's worth noting that Guido van Rossum, creator of Python, is working at Dropbox on this project, and that Mypy is funded primarily by Dropbox (I think). I'm glad that they are doing it, because it's contributed hugely to the Python ecosystem -- especially making it easier for other companies to make the switch to Python 3. I highly recommend mypy for new and current python3 users. It's one of the biggest and best reasons to be using python3 (not that you should need any more :) )
- xvector 8y ago
  I've never used mypy and I used Python a lot until I recently got my first job. It kind of looks like a band-aid. Could you elaborate on why I would use, say, mypy as opposed to Golang if I don't need the benefits of a scripting language? edit: I can understand why mypy would be useful to refactor an existing (large) project to bring forth type safety guarantees as you go forwards, but surely for new projects there are other languages of choice if type safety is what you want?
  - joshuamorton 8y ago
    If python offers type safety, and language x offers type safety, what reason do I have to pick language x over python?
    - rantanplan 8y ago
      5x(probably more) reduction on server costs because of less memory/cpu requirements, if you choose Go for example.
      - joshuamorton 8y ago
        Ah, so your reasons to pick a different language are based not on type safety, but on performance. Indeed there may be a plethora of valid reasons to pick another language over python for some project. But that wasn't the question.
nvr219 8y ago
That "learn python the hard way" guy says don't use python 3. As someone who is at the "Hello World" stage, should I use 2 or 3?
- qgsrw3fd 8y ago
  Whoever says to use 2 today is a retard.
  - klardotsh 8y ago
    I can't downvote, so instead I'll call you out publicly. This is neither a productive nor appropriate response. We can discuss the merits of Python 3 over 2 (almost all of which I full-heartedly would agree with!) without resorting to name-calling of "the other camp".
    - qgsrw3fd 8y ago
      You are right. My apologies. Whoever tells someone who is learning Python to use 2, which will be EOL in less than one year is not a retard, but a fucking retard.
  - sametmax 8y ago
    After 12 years since v3 came out and the annual python survey showing 84% v3, i think it's really disrespectful from anyone to advice v2. But retard is not a convincing argument, even if i'm tempted to use it too.
- carbocation 8y ago
  I don't write a lot of Python (use it mostly when deep learning calls for it), but my understanding is that it's time to start with Python 3. I try to write Python 3 whenever I have to write Python. https://pythonclock.org/ https://pythonclock.org/
- phowat 8y ago
  Python 2 will be EOL'd on the 1st of January 2020. So you should definitely go with python 3.
deleted 8y ago
[deleted]
saltcured 8y ago
At work, we have much smaller codebases that have fractional FTEs allocated to their ongoing maintenance. In spite of the difference in scale, my experience is similar to what was described in the article. Because we had focused on getting the unicode to work right in those old code bases, we had good test coverage for those features as a result. The other common legacy problems to address were: - Other implicit encode/decode behaviors in py2 that need to be explicit in py3 - Old 'print' and 'except' statements not valid in py3, easily rewritten - Implicitly relative 'import' statements not valid in py3, rewritten with a little care - Arithmetic needing a change to the '//' operator for integer division - Waiting for py3 support in all third-party dependencies - Dealing with restructuring in standard lib packages and in upgraded third-party libs w/ py3 support As I reviewed the techniques for straddling py2 and py3, I was displeased with how many seemed to involve a third dialect which was not really idiomatic py2 nor py3, particularly for the unicode/bytes handling. Many third-party libraries and frameworks also made different choices for how they handled this. Trying to integrate those approaches looked to produce even uglier code. Also, some of our code had evolved since the Python 2.2 days and had accumulated cruft to import and wrap multiple generations of older standard lib and add-on packages which we have not cared about for 5+ years. The additional package restructuring in py3 would have made this even more bizarre. I wanted to see the code reset to use standard libs where possible and cull these legacy third-party dependencies. I also wanted the code to become more idiomatically py3, so whoever visited it for future maintenance would not need to work so hard to understand it. So, we chose a clean break where we finally clean up and modernize the code to py3-only without the added burden of supporting py2 deployments from the same code. The declared 2020 deadline helped this decision. We branched our repos and worked on py3 ports and integration testing in parallel while continuing to run py2 in production. We declared a feature freeze on the py2 code, so we would not have a merging nightmare later, and so that we could use that as pressure to prevent procrastination on scheduling the flag day where we merge PRs and convert all our repos to a py3-only worldview.
raverbashing 8y ago
Python 2 to 3 migration is straightforward for the most part but in the end it has some nooks and crannies Trickiest one that we tripped: strings in Python 3 have the __iter__ method
dekhn 8y ago
While adding Python3 support for numpy at a time when I worked for a large Ad-serving company, we managed to uncover a bug in python 2/3's import code that had been latent (no crash) in Python 2 for 15+ years. The problem was unique to people who had made it possible to import the same library from two filesystem locations. In python2, it was silent, in python3 it was a segfault. There was really only one person in the company truly qualified to understand and fix the bug. I'm finally moved over to Python3 but boy, was that an unwanted transition.
ilovecaching 8y ago
I've noticed a trend that migrating from Python 2 to Python 3 includes adding mypy annotations and going async. At that point, why not Go? You're trying to correct for a language that was designed for scripting, not application software. Go already has a type system, coroutines are a simpler model of concurrency than async (and Go can actually use multithreading), you don't have to choose between writing and async or nonasync library code, built in formatter (wheras Black is still experimental), and the code will run 10x faster.
- ApolloFortyNine 8y ago
  >You're trying to correct for a language that was designed for scripting, not application software. Well, that's not true. Are you basing this solely over a lack of a type system?
  - eesmith 8y ago
    As an historical observation, Python 0.9p1's README said (emphasis mine): > This is Python, an extensible interpreted programming language that combines remarkable power with very clear syntax. > This is version 0.9 (the first beta release), patchlevel 1. > Python can be used instead of shell, Awk or Perl scripts, to write prototypes of real applications, or as an extension language of large systems, you name it. Note the emphasis on "prototypes of real applications." That would suggest that version 0.9 was not really designed for "real applications". However, by the 1.0 release this timid statement disappeared. For example, by the 1.0.3 release the README was telling "For a quick summary of what Python can mean for a UNIX/C programmer, read Misc/BLURB.LUTZ." That quotes an email dated Thu, 14 Oct 1993 17:10:37 GMT which said: > It's central goal is to provide the best of both worlds: the dynamic nature of scripting languages like Perl/TCL/REXX, but also support for general programming found in the more traditional languages like Icon, C, Modula,... > As such, it can function as a scripting/extension language, as a rapid prototyping language, and as a serious software development language. Python is suitable for fast development of large programs, but also does well at throw-away shell coding. Thus, in a very narrow and irrelevantly technical sense, ilovecaching is correct - the very earliest versions of Python were not described as being for full-blown application software. As it turns out, despite not being designed for 'real applications', it turned out to be useful for 'serious software development', and the evolution for the last 27 or so years has kept that goal in mind.
jonathanpoulter 8y ago
Garret Walker, from Bank of America, spoke at PyGotham last year and touched on this topic at the end of his talk: https://2018.pygotham.org/talks/seventeen-million-lines-of-python-later-launching-a-startup-in-an-investment-bank/ https://2018.pygotham.org/talks/seventeen-million-lines-of-p... Disclosure: I work for Bank of America.
omarforgotpwd 8y ago
What a disaster of lost productivity
painful 8y ago
mypy is a joke of a tool. I have actually used it, and 80% of its messages are useless junk or just plain wrong. Granted, the other 20% can be on point. All things considered, it's better to be with a tool like it, than without.
petters 8y ago
> and removed Python 2 from our application binary. This marked the end of our Python version migration! No. :-) Now you need to remove all compatibility code, modernize your syntax and gradually start using new features.

91 points by el_duderino 8y ago

Nice write up. At Thread we did a similar thing. Admittedly our codebase is ~10% as big, but incrementally adding linters for incompatible code, and keeping the CI green, helped loads. We only had about 2-3 days of engineering time to ship the final version, the rest was done in 10% time.

jsilence 8y ago

Why does the desktop client have one million lines of code?

danpalmer 8y ago
Not to be facetious, but why wouldn't it? It has to find and diff files, coordinate a very reliable file upload of many files, as fast as possible. It has to understand enough about the content of those files to be able to do useful things with them. It has to reliably update itself again and again, from possibly very old versions, it has to communicate with an ever changing API. It has to have enough analytics in it to support product development, error reporting, understanding how users use it, how the product needs to evolve over time. It needs to integrate deeply with all major OSes. ...I'm sure I've missed some things. 1 million lines of code sounds like the right ballpark to me though.
- sametmax 8y ago
  Plus the UI, the file manager intagration, the daemon, the authentification, the cache... All that cross platform.
  - saagarjha 8y ago
    None of those should be written in Python, except for perhaps the cache handling code. The rest should be platform-specific.
    - rmdashrfstar 8y ago
      Why?
      - saagarjha 8y ago
        Because they're platform specific. For example, on macOS, the UI and file manager integration should be done in Swift or Objective-C(++), the authentication in some C-compatible language to match the security frameworks, and the daemon in some language that isn't single threaded or garbage collected.
        chrisseaton 8y ago
        You’re talking about that it uses platform specific APIs. You can use these APIs from Python, and then there is all the logic which decides how to use the APIs and that can be in Python as well.
        saagarjha 8y ago
        > You can use these APIs from Python You can do it hand-written assembly too, but usually it's more trouble than it's worth.
        chrisseaton 8y ago
        I think what we've worked out what you mean is just that if you were doing it you'd chose to do it in Objective C etc. That's not quite the same as anyone else 'should' do it.
        sametmax 8y ago
        Not only you can do all that confortably and productively in python, but you should. It's crossplatform, the task has not bottleneck for this vm, and it takes complexity down to one language. One language that as the very specific characteristic of being the best at nothing, but pretty good at everything, which in such an heterogenous software is what you want.
bdcravens 8y ago
I wonder if their code is over 1M, or if they're doing what you see in a lot of marketing, which is count all the lines from included packages.
- rhizome 8y ago
  Who markets on the basis of LOC?
  - chillydawg 8y ago
    Dropbox, right now. These blogs are always with an eye to recruitment.
    - chrisseaton 8y ago
      Why would a massive legacy codebase be a recruitment turn-on?
      - pmoriarty 8y ago
        Because then you can put on your resume that you have experience working on a massive codebase.
        deleted 8y ago
        [deleted]
barkingcat 8y ago
million lines isn't really that much

rhacker 8y ago

Is python finally getting past the 2/3 jump?

dangoor 8y ago
Python 2 EOL on January 1, 2020 is pushing a lot of folks to seriously look at this. This is a big thing we're looking at right now at Khan Academy.
Alex3917 8y ago
It's already been since 2015. Every major Python dependency has been Python3 compatible for several years, and many have already gone Python3 only. If you have a good test suite it's pretty easy to migrate, you can easily do a couple thousand lines per day. There were a few third-party testing tools that used to be fairly popular that are no longer well maintained, so if your tests are written in those then you might be looking at re-writing the app from scratch. But otherwise it's fairly straightforward to port. Even if you did your string encoding just using guess-and-check, which most people did, it still doesn't take all that long to upgrade as long as there are tests.
lincolnq 8y ago
I would say yes. I'm pretty sure most people are writing all new projects exclusively in python3, except if you need them to run on old distros without any dependencies (lots of distributions shipped for years with python2 but not python3 installed by default). The ecosystem of 3 is very mature - I've been running my servers' production code for 5+ years in python3 and had no major issues with that choice. The only thing left is "legacy codebases".
- Jare 8y ago
  IIRC osx still ships python2 only with the OS.
sametmax 8y ago
It has for 2 years now. 3.6 was the tipping point, such a fantastic release. See https://www.jetbrains.com/research/devecosystem-2018/python/ https://www.jetbrains.com/research/devecosystem-2018/python/ For 2018 stats

lincolnq 8y ago

It's worth noting that Guido van Rossum, creator of Python, is working at Dropbox on this project, and that Mypy is funded primarily by Dropbox (I think). I'm glad that they are doing it, because it's contributed hugely to the Python ecosystem -- especially making it easier for other companies to make the switch to Python 3. I highly recommend mypy for new and current python3 users. It's one of the biggest and best reasons to be using python3 (not that you should need any more :) )

xvector 8y ago
I've never used mypy and I used Python a lot until I recently got my first job. It kind of looks like a band-aid. Could you elaborate on why I would use, say, mypy as opposed to Golang if I don't need the benefits of a scripting language? edit: I can understand why mypy would be useful to refactor an existing (large) project to bring forth type safety guarantees as you go forwards, but surely for new projects there are other languages of choice if type safety is what you want?
- joshuamorton 8y ago
  If python offers type safety, and language x offers type safety, what reason do I have to pick language x over python?
  - rantanplan 8y ago
    5x(probably more) reduction on server costs because of less memory/cpu requirements, if you choose Go for example.
    - joshuamorton 8y ago
      Ah, so your reasons to pick a different language are based not on type safety, but on performance. Indeed there may be a plethora of valid reasons to pick another language over python for some project. But that wasn't the question.

nvr219 8y ago

That "learn python the hard way" guy says don't use python 3. As someone who is at the "Hello World" stage, should I use 2 or 3?

qgsrw3fd 8y ago
Whoever says to use 2 today is a retard.
- klardotsh 8y ago
  I can't downvote, so instead I'll call you out publicly. This is neither a productive nor appropriate response. We can discuss the merits of Python 3 over 2 (almost all of which I full-heartedly would agree with!) without resorting to name-calling of "the other camp".
  - qgsrw3fd 8y ago
    You are right. My apologies. Whoever tells someone who is learning Python to use 2, which will be EOL in less than one year is not a retard, but a fucking retard.
- sametmax 8y ago
  After 12 years since v3 came out and the annual python survey showing 84% v3, i think it's really disrespectful from anyone to advice v2. But retard is not a convincing argument, even if i'm tempted to use it too.
carbocation 8y ago
I don't write a lot of Python (use it mostly when deep learning calls for it), but my understanding is that it's time to start with Python 3. I try to write Python 3 whenever I have to write Python. https://pythonclock.org/ https://pythonclock.org/
phowat 8y ago
Python 2 will be EOL'd on the 1st of January 2020. So you should definitely go with python 3.

deleted 8y ago

[deleted]

saltcured 8y ago

At work, we have much smaller codebases that have fractional FTEs allocated to their ongoing maintenance. In spite of the difference in scale, my experience is similar to what was described in the article. Because we had focused on getting the unicode to work right in those old code bases, we had good test coverage for those features as a result. The other common legacy problems to address were: - Other implicit encode/decode behaviors in py2 that need to be explicit in py3 - Old 'print' and 'except' statements not valid in py3, easily rewritten - Implicitly relative 'import' statements not valid in py3, rewritten with a little care - Arithmetic needing a change to the '//' operator for integer division - Waiting for py3 support in all third-party dependencies - Dealing with restructuring in standard lib packages and in upgraded third-party libs w/ py3 support As I reviewed the techniques for straddling py2 and py3, I was displeased with how many seemed to involve a third dialect which was not really idiomatic py2 nor py3, particularly for the unicode/bytes handling. Many third-party libraries and frameworks also made different choices for how they handled this. Trying to integrate those approaches looked to produce even uglier code. Also, some of our code had evolved since the Python 2.2 days and had accumulated cruft to import and wrap multiple generations of older standard lib and add-on packages which we have not cared about for 5+ years. The additional package restructuring in py3 would have made this even more bizarre. I wanted to see the code reset to use standard libs where possible and cull these legacy third-party dependencies. I also wanted the code to become more idiomatically py3, so whoever visited it for future maintenance would not need to work so hard to understand it. So, we chose a clean break where we finally clean up and modernize the code to py3-only without the added burden of supporting py2 deployments from the same code. The declared 2020 deadline helped this decision. We branched our repos and worked on py3 ports and integration testing in parallel while continuing to run py2 in production. We declared a feature freeze on the py2 code, so we would not have a merging nightmare later, and so that we could use that as pressure to prevent procrastination on scheduling the flag day where we merge PRs and convert all our repos to a py3-only worldview.

raverbashing 8y ago

Python 2 to 3 migration is straightforward for the most part but in the end it has some nooks and crannies Trickiest one that we tripped: strings in Python 3 have the __iter__ method

dekhn 8y ago

While adding Python3 support for numpy at a time when I worked for a large Ad-serving company, we managed to uncover a bug in python 2/3's import code that had been latent (no crash) in Python 2 for 15+ years. The problem was unique to people who had made it possible to import the same library from two filesystem locations. In python2, it was silent, in python3 it was a segfault. There was really only one person in the company truly qualified to understand and fix the bug. I'm finally moved over to Python3 but boy, was that an unwanted transition.

ilovecaching 8y ago

I've noticed a trend that migrating from Python 2 to Python 3 includes adding mypy annotations and going async. At that point, why not Go? You're trying to correct for a language that was designed for scripting, not application software. Go already has a type system, coroutines are a simpler model of concurrency than async (and Go can actually use multithreading), you don't have to choose between writing and async or nonasync library code, built in formatter (wheras Black is still experimental), and the code will run 10x faster.

ApolloFortyNine 8y ago
>You're trying to correct for a language that was designed for scripting, not application software. Well, that's not true. Are you basing this solely over a lack of a type system?
- eesmith 8y ago
  As an historical observation, Python 0.9p1's README said (emphasis mine): > This is Python, an extensible interpreted programming language that combines remarkable power with very clear syntax. > This is version 0.9 (the first beta release), patchlevel 1. > Python can be used instead of shell, Awk or Perl scripts, to write prototypes of real applications, or as an extension language of large systems, you name it. Note the emphasis on "prototypes of real applications." That would suggest that version 0.9 was not really designed for "real applications". However, by the 1.0 release this timid statement disappeared. For example, by the 1.0.3 release the README was telling "For a quick summary of what Python can mean for a UNIX/C programmer, read Misc/BLURB.LUTZ." That quotes an email dated Thu, 14 Oct 1993 17:10:37 GMT which said: > It's central goal is to provide the best of both worlds: the dynamic nature of scripting languages like Perl/TCL/REXX, but also support for general programming found in the more traditional languages like Icon, C, Modula,... > As such, it can function as a scripting/extension language, as a rapid prototyping language, and as a serious software development language. Python is suitable for fast development of large programs, but also does well at throw-away shell coding. Thus, in a very narrow and irrelevantly technical sense, ilovecaching is correct - the very earliest versions of Python were not described as being for full-blown application software. As it turns out, despite not being designed for 'real applications', it turned out to be useful for 'serious software development', and the evolution for the last 27 or so years has kept that goal in mind.

jonathanpoulter 8y ago

Garret Walker, from Bank of America, spoke at PyGotham last year and touched on this topic at the end of his talk: https://2018.pygotham.org/talks/seventeen-million-lines-of-python-later-launching-a-startup-in-an-investment-bank/ https://2018.pygotham.org/talks/seventeen-million-lines-of-p... Disclosure: I work for Bank of America.

omarforgotpwd 8y ago

What a disaster of lost productivity

painful 8y ago

mypy is a joke of a tool. I have actually used it, and 80% of its messages are useless junk or just plain wrong. Granted, the other 20% can be on point. All things considered, it's better to be with a tool like it, than without.

petters 8y ago

> and removed Python 2 from our application binary. This marked the end of our Python version migration! No. :-) Now you need to remove all compatibility code, modernize your syntax and gradually start using new features.