4 ms·

You can start porting your live codebase to python 3 right now - no downtime at all. You can have a 2+3 compatible codebase live in a week or two, it'll get yo

by tony 7y ago

You can start porting your live codebase to python 3 right now - no downtime at all. You can have a 2+3 compatible codebase live in a week or two, it'll get you about 98% the way there [1] My projects (e.g. https://tmuxp.git-pull.com https://tmuxp.git-pull.com, https://unihan-etl.git-pull.com https://unihan-etl.git-pull.com) are both python 2/3 compatible. I learned by reading through other projects like Sphinx, Flask, Werkzeug, and SQLAlchemy: - Flask: https://github.com/pallets/flask https://github.com/pallets/flask - Werkzeug: https://github.com/pallets/werkzeug https://github.com/pallets/werkzeug - Sphinx https://github.com/sphinx-doc/sphinx/tree/v1.8.5 https://github.com/sphinx-doc/sphinx/tree/v1.8.5 (see compat: https://github.com/sphinx-doc/sphinx/blob/v1.8.5/sphinx/util/compat.py https://github.com/sphinx-doc/sphinx/blob/v1.8.5/sphinx/util...) - https://github.com/sqlalchemy/sqlalchemy https://github.com/sqlalchemy/sqlalchemy Helpful blog posts: http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python/ http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python..., http://lucumr.pocoo.org/2013/5/21/porting-to-python-3-redux/ http://lucumr.pocoo.org/2013/5/21/porting-to-python-3-redux/ As for automation tools, for 2/3 compatibility, I used futurize and huge codebase to great success: https://python-future.org/futurize.html https://python-future.org/futurize.html. I started with this: futurize --write --stage1 --unicode-literals --nobackups <files> and https://docs.python.org/2/library/2to3.html https://docs.python.org/2/library/2to3.html 2to3-2.7 -w -n <files> Really helped. Also, wire Travis / your CI system to have your tests run on python 2 and 3. If you want to test your python 3 codebase live on a subdomain / same SQL/NoSQL DB, be careful about jobs/tasks! Pickle version mismatches and stuff. Use a separate redis/whatever DB for the deployments. After you're finally on python 3, you can use https://github.com/asottile/pyupgrade https://github.com/asottile/pyupgrade to modern your code. [1] For the other 2%, use conditionals in your requirements file: enum34==1.0.4;python_version<"3" ushlex==0.99.1;python_version<"3"

dangoor 7y ago
I appreciate all of the links and the fact that, for a lot of folks, this is the state of Python 2 to 3 migration today. What you're suggesting is, imho, the best path for most (and I appreciate you mentioning the pickle incompatibility because that is a thing which would trip people up when trying to make the move). If we would have been able to have a 2&3 compatible codebase live in a week or two, we absolutely would have done that. Incompatible changes in some libraries we use, App Engine first gen to second gen changes (which are for the better, but still a big deal), the choice of storing some pickles permanently, plus a need to really verify unicode handling all over the place (especially in a 10 year old codebase), and other factors that aren't coming to mind at the moment mean that this is not a couple week thing for us. Moving to Go is more work than moving to Python 3, but in our particular case it's not as much more work as people might expect.
- tony 7y ago
  Thanks for your reply to my reply! Sure, it may take more than two weeks, and maybe golang is a good choice for your stack. Maybe a better way of saying it is python 2/3 compatibility can be gradually adopted while maintaining. Also, I can't speak for the technical specifics of your project, but I would like to speak a bit from my prior experiences: > Incompatible changes in some libraries we use What are those libraries? There may be python2/3 forks available on pypi. For instance, on Peergrade we went from pyPdf -> pyPdf2, boto -> boto3 (that one is a lot of work). We were still able to stick on the python 2 codebase, but gain python 3 forward compatibility. In the case of very specific packages, we had to do forks. We had to move some patches from a python 2 only library and port them into a python 3 only library, then do one of those version constraint things. > App Engine first gen to second gen changes (which are for the better, but still a big deal) I'm not familiar with app engine so can't speak to it. > the choice of storing some pickles permanently Can you clarify this? I can't speak for it without seeing it, but would it be possible to serialize the data to json, use python import strings (https://devel.tech/tips/n/djms3tTe/how-django-uses-deferred-imports-to-scale/ https://devel.tech/tips/n/djms3tTe/how-django-uses-deferred-...), then write a migration to port the old pickles over, and have something more portable? It's possible, depending on what's being stored, it'd also help you migrate to golang if the data stored in it could be consumed by a go service. > plus a need to really verify unicode handling all over the place (especially in a 10 year old codebase) Yes I had a lot of pitfalls with this one. Areas to look out for are hashing functions. They are very strict in whether they're dealing with bytes or strings. Are you already using unicode_literals? Those can be implemented gradually in a current codebase. How is your test suite? Sometimes having test coverage, even naively, can also act as a smoke test to catch unicode issues. Selenium tests can get a lot of coverage for very cheap to verify behavior at a high level. > Moving to Go is more work than moving to Python 3, but in our particular case it's not as much more work as people might expect. I think the services + golang part is smart, and can't speak for the specifics of your codebase. I will say in hindsight, I feel the effort / hours I put into upgrading a Python 2 codebase -> 2/3 and eventually 3-only made it much easier to breath. Cleaner syntax, no unicode headaches, and no need to have the lingering prospect of a language exodus looming over the head. One negative aspect we haven't talked about those with "gradual" python 2/3 migrating is breakages. There are refactors that end up being done that when pushed out risk breaking - and when they're purely internal code changes. There is a business case to eliminate the tech debt because the value equation: A python 2 codebase, even with warts to it, is meeting an EOL in the next few days (https://pythonclock.org/ https://pythonclock.org/). Even assuming golang + microservices is the final destination. Amortized, on the projects I've been on, moving to python 2/3 (or 3-only) paid back. If there's opportunities where the python codebase could be split into apps / separate wsgi entry points, then have golang services replace them later - that could be an option. Even without golang, the (probably huge?) refactors involved in moving to python 3 and being "(micro?)service ready" has benefits.

by tony 7y ago

dangoor 7y ago

I appreciate all of the links and the fact that, for a lot of folks, this is the state of Python 2 to 3 migration today. What you're suggesting is, imho, the best path for most (and I appreciate you mentioning the pickle incompatibility because that is a thing which would trip people up when trying to make the move). If we would have been able to have a 2&3 compatible codebase live in a week or two, we absolutely would have done that. Incompatible changes in some libraries we use, App Engine first gen to second gen changes (which are for the better, but still a big deal), the choice of storing some pickles permanently, plus a need to really verify unicode handling all over the place (especially in a 10 year old codebase), and other factors that aren't coming to mind at the moment mean that this is not a couple week thing for us. Moving to Go is more work than moving to Python 3, but in our particular case it's not as much more work as people might expect.

tony 7y ago
Thanks for your reply to my reply! Sure, it may take more than two weeks, and maybe golang is a good choice for your stack. Maybe a better way of saying it is python 2/3 compatibility can be gradually adopted while maintaining. Also, I can't speak for the technical specifics of your project, but I would like to speak a bit from my prior experiences: > Incompatible changes in some libraries we use What are those libraries? There may be python2/3 forks available on pypi. For instance, on Peergrade we went from pyPdf -> pyPdf2, boto -> boto3 (that one is a lot of work). We were still able to stick on the python 2 codebase, but gain python 3 forward compatibility. In the case of very specific packages, we had to do forks. We had to move some patches from a python 2 only library and port them into a python 3 only library, then do one of those version constraint things. > App Engine first gen to second gen changes (which are for the better, but still a big deal) I'm not familiar with app engine so can't speak to it. > the choice of storing some pickles permanently Can you clarify this? I can't speak for it without seeing it, but would it be possible to serialize the data to json, use python import strings (https://devel.tech/tips/n/djms3tTe/how-django-uses-deferred-imports-to-scale/ https://devel.tech/tips/n/djms3tTe/how-django-uses-deferred-...), then write a migration to port the old pickles over, and have something more portable? It's possible, depending on what's being stored, it'd also help you migrate to golang if the data stored in it could be consumed by a go service. > plus a need to really verify unicode handling all over the place (especially in a 10 year old codebase) Yes I had a lot of pitfalls with this one. Areas to look out for are hashing functions. They are very strict in whether they're dealing with bytes or strings. Are you already using unicode_literals? Those can be implemented gradually in a current codebase. How is your test suite? Sometimes having test coverage, even naively, can also act as a smoke test to catch unicode issues. Selenium tests can get a lot of coverage for very cheap to verify behavior at a high level. > Moving to Go is more work than moving to Python 3, but in our particular case it's not as much more work as people might expect. I think the services + golang part is smart, and can't speak for the specifics of your codebase. I will say in hindsight, I feel the effort / hours I put into upgrading a Python 2 codebase -> 2/3 and eventually 3-only made it much easier to breath. Cleaner syntax, no unicode headaches, and no need to have the lingering prospect of a language exodus looming over the head. One negative aspect we haven't talked about those with "gradual" python 2/3 migrating is breakages. There are refactors that end up being done that when pushed out risk breaking - and when they're purely internal code changes. There is a business case to eliminate the tech debt because the value equation: A python 2 codebase, even with warts to it, is meeting an EOL in the next few days (https://pythonclock.org/ https://pythonclock.org/). Even assuming golang + microservices is the final destination. Amortized, on the projects I've been on, moving to python 2/3 (or 3-only) paid back. If there's opportunities where the python codebase could be split into apps / separate wsgi entry points, then have golang services replace them later - that could be an option. Even without golang, the (probably huge?) refactors involved in moving to python 3 and being "(micro?)service ready" has benefits.