4 ms·

Don't they have source to source compilers that do the migration for you? I don't understand why the friction is so strong with python.

gowld 7y ago
The friction is that people don't like changing thngs that work, and a lot of Python 3 is backported from future People migrate when the allure of Python 3 outweighs the hassle of juggling 2 versions. Naturally, people who grew up on Python 2 are more hesitant than people who started later on Python 3.
kccqzy 7y ago
Yes but the source-to-source compilers do a fraction of the work needed to get a code base migrated. I've tried both the builtin 2to3 utility and many proprietary converters, including fancy ones that run type inference on Python 2 code and produce type annotations. None of them, as expected, can deal with the str/bytes split without manual intervention. Fundamentally the issue is that the Python language is too dynamic for static analyzers to fully understand the code and perform conversions. I'm not just talking about type systems, but also monkey patching, reflections, etc. What we really need is a tool that runs your Python 2 code, records types and other dynamic manipulations, and then produce Python 3 code. But I'm not aware of such a tool.
- darksaints 7y ago
  > What we really need is a tool that runs your Python 2 code, records types and other dynamic manipulations, and then produce Python 3 code. But I'm not aware of such a tool. That's a great idea, but it sounds pretty intense to implement. I ended up becoming addicted to ML-family languages pretty early (Scala, F#, OCaml, Rust) for a variety of reasons, but I never fully grasped how much strong/static types help for refactoring until I tried refactoring some Ruby code. I can't imagine trying to write an automated tool for the task.
- takeda 7y ago
  Funny that you mentioned it, because there's such tool and one of it orginates from Dropbox - PyAnnotate[1] There's also MonekyType[2] from instagram, but it only works on Python 3. I will also drop PyType[3] from Google, although this one tries to statically determine the types and when I used it felt a bit buggy. [1] https://github.com/dropbox/pyannotate https://github.com/dropbox/pyannotate [2] https://github.com/instagram/MonkeyType https://github.com/instagram/MonkeyType [3] https://github.com/google/pytype https://github.com/google/pytype
takeda 7y ago
Unfortunately that's not as easy, because python is a dynamic language, and allows you to do some shenanigans. For example in python 3 they actually made import statements more predictable they are now absolute by default, it was a bit ambiguous in python 2. Ignoring the ambiguity (which is a problem by itself) some people were creative and instead doing standard imports like god intended decided to do it dynamically using importlib, that can't be easily fixed if the file you're importing is a variable. Another problem: unicode, actually most python 2 code is broken and it can crap out whenever you use characters outside of ascii. That's because it conflates bytes with text, python 3 makes the distinction strict, so you have to properly address it. Though for 99% of application user meant text, so perhaps this assumption could be made? - division, in python 2 division worked similar to C, if both the parameters were integers you get integer as a result, python 3 changed it to be more what most people would expect, although python 3 also introduced // which had the prior behavior so I guess this could be automated There are probably others too.
nickm12 7y ago
Python is too dynamic to do a source-to-source compilation that works 100% of the time. Yes, you can get 80% there or maybe even 90% there, but the last 10% is a killer. For a small program, you can stop all development, certify the program under Py3 and then go on. Dropbox's server code base is literally millions of lines of Python 2. The "stop all development" approach simply won't work there. The only thing that works is to port the whole code base to "straddling code" that works on both interpreters, then switch interpreters. Automated tooling to prevent regressions is essential.

gowld 7y ago

The friction is that people don't like changing thngs that work, and a lot of Python 3 is backported from future People migrate when the allure of Python 3 outweighs the hassle of juggling 2 versions. Naturally, people who grew up on Python 2 are more hesitant than people who started later on Python 3.

kccqzy 7y ago

Yes but the source-to-source compilers do a fraction of the work needed to get a code base migrated. I've tried both the builtin 2to3 utility and many proprietary converters, including fancy ones that run type inference on Python 2 code and produce type annotations. None of them, as expected, can deal with the str/bytes split without manual intervention. Fundamentally the issue is that the Python language is too dynamic for static analyzers to fully understand the code and perform conversions. I'm not just talking about type systems, but also monkey patching, reflections, etc. What we really need is a tool that runs your Python 2 code, records types and other dynamic manipulations, and then produce Python 3 code. But I'm not aware of such a tool.

darksaints 7y ago
> What we really need is a tool that runs your Python 2 code, records types and other dynamic manipulations, and then produce Python 3 code. But I'm not aware of such a tool. That's a great idea, but it sounds pretty intense to implement. I ended up becoming addicted to ML-family languages pretty early (Scala, F#, OCaml, Rust) for a variety of reasons, but I never fully grasped how much strong/static types help for refactoring until I tried refactoring some Ruby code. I can't imagine trying to write an automated tool for the task.
takeda 7y ago
Funny that you mentioned it, because there's such tool and one of it orginates from Dropbox - PyAnnotate[1] There's also MonekyType[2] from instagram, but it only works on Python 3. I will also drop PyType[3] from Google, although this one tries to statically determine the types and when I used it felt a bit buggy. [1] https://github.com/dropbox/pyannotate https://github.com/dropbox/pyannotate [2] https://github.com/instagram/MonkeyType https://github.com/instagram/MonkeyType [3] https://github.com/google/pytype https://github.com/google/pytype

takeda 7y ago

Unfortunately that's not as easy, because python is a dynamic language, and allows you to do some shenanigans. For example in python 3 they actually made import statements more predictable they are now absolute by default, it was a bit ambiguous in python 2. Ignoring the ambiguity (which is a problem by itself) some people were creative and instead doing standard imports like god intended decided to do it dynamically using importlib, that can't be easily fixed if the file you're importing is a variable. Another problem: unicode, actually most python 2 code is broken and it can crap out whenever you use characters outside of ascii. That's because it conflates bytes with text, python 3 makes the distinction strict, so you have to properly address it. Though for 99% of application user meant text, so perhaps this assumption could be made? - division, in python 2 division worked similar to C, if both the parameters were integers you get integer as a result, python 3 changed it to be more what most people would expect, although python 3 also introduced // which had the prior behavior so I guess this could be automated There are probably others too.

nickm12 7y ago

Python is too dynamic to do a source-to-source compilation that works 100% of the time. Yes, you can get 80% there or maybe even 90% there, but the last 10% is a killer. For a small program, you can stop all development, certify the program under Py3 and then go on. Dropbox's server code base is literally millions of lines of Python 2. The "stop all development" approach simply won't work there. The only thing that works is to port the whole code base to "straddling code" that works on both interpreters, then switch interpreters. Automated tooling to prevent regressions is essential.