7 ms·

How to Port from Python 2 to Python 3 (2019)

54 points by noego 7y ago

o_x 7y ago
Isn't it ironic that Sentry is one of the tools mentioned in py2->py3 migration? (Sentry is on py2 and as far as I remember they were not very optimistic about migrating)
- zojirushibottle 7y ago
  that's correct. sentry itself is on python 2.7, not the python client. not to pick on sentry here, but you know, my experience is that people are having a hard time migrating due to them using obscure tricks and features of python 2.7. so their code is breaking because the language evolved. the saying goes write dumb code or something because debugging is twice as hard. if there is anything to learn from all this, it's to write dumb code because maintenance is twice as hard too. that's all that is happening really!
  - ggregoire 7y ago
    Why would you even build a logs collector in python? Especially if it’s your core business and you know you will need scale and reliability. You kinda shoot yourself in the foot.
    - baq 7y ago
      they're using python 2.7. perhaps started on an earlier one. maybe there was nothing except java 1.4 when they started.
LoreleiPenn 7y ago
Hopefully no more people will keep saying "learn Python 2 because 3 has almost no packages". It is so easy for people to just repeat what they heard even if that idea originated a decade ago and was valid a decade ago. And that way we got into a mess of not migrating until pass the time it is no longer supported...
swalsh 7y ago
I literally just got on a phone call to discuss our migration away from 2.7, very timely post.
ryandvm 7y ago
Python 3 and IPv6 are the poster-children of how _not_ to do a major upgrade. I'm not sure what the right way is, but if the short-term advantages of the upgrade do not outweigh the immediate pain, prepare for the matter to drag out for _decades_.
- rb808 7y ago
  At least Python3 brings an improvement. I still can't think of a good reason I should spend any time trying to figure out IPv6. Maybe my external router has to think about it, but that is about it. Edit: If you down vote me please say why I should care about IPv6
  - Fr0styMatt88 7y ago
    The main mental stumbling block I have about IPv6 (and I know it's kind of a silly one, but it's honestly what I feel on introspection) is that I can't remember an IPv6 address off by heart. IPv4 addresses just _feel_ so much more human-consumable than IPv6 addresses. I can't imagine myself using IPv6 addresses on the command line like I do with IPv4 now. There's also the issue that honestly, I have no idea what is using IPv6 and what's using IPv4 right now. On my internal network I only ever deal with IPv4, but I have no intuition as to what is using IPv6; I couldn't tell you off the top of my head if my ISP supports it.
    - deathanatos 7y ago
      > is that I can't remember an IPv6 address off by heart. IPv4 addresses just _feel_ so much more human-consumable than IPv6 addresses. Stop remembering numbers meant for a machine, and use DNS. It will make your life so much easier. I spend ~$15/yr for my personal DNS name & hosting, and I never want to memorize an IPv4 or v6 address ever again. > I have no idea what is using IPv6 and what's using IPv4 right now. Any "end" machine (laptop, tablet, phone, server) is going to be dual stack, supporting both IPv4 and IPv6. If it is able to auto-configure an address, then it will use those. Otherwise, it won't. If a domain is IPv4-only, it'll use the IPv4 address. All of this is automatic. The big issue, for home consumers, is that a lot of ISPs are dragging their feet. They don't need anything from the customers — they just need to get it deployed & turned on. Generally, typing "what is my IP address?" into Google will tell you if you have working IPv6; it will display an IPv6 address if you do. In the cloud… some cloud vendors have been dragging their feet about rolling support out. You need to do some things, like associate an AAAA record to your domain (s.t. it resolves to an IPv6 address), and make sure things like logging can handle the new addresses, or if you implement IP blocking, that you can block those addresses/networks. If you're writing network code, you need to check that you're not assumptions about the socket type. You can also do things like HTTP proxy from an IPv6 connection to an IPv4 VLAN, e.g., I think w/ an ELB. That is, client <-- HTTP/IPv6 --> ELB <-- HTTP/IPv4 --> backend server which allows a partial upgrade. None of it is terribly hard, but typical project management puts upgrading to future tech in the perpetual backlog.
zitterbewegung 7y ago
The big issue of ports like these is not the tutorial but to justify that to your boss. From enterprise to a self run startup you have to see if it’s worth it .
- GrayTzar 7y ago
  Hi, I'm one of the STX Next content crew. We actually have a companion piece to this that goes over the reasons why you should migrate: https://stxnext.com/blog/2019/07/30/why-migrate-from-python-2-to-python-3/ https://stxnext.com/blog/2019/07/30/why-migrate-from-python-... Maybe that would be useful for a conversation with one's boss.
- BeetleB 7y ago
  > The big issue of ports like these is not the tutorial but to justify that to your boss. I think that's a red herring. Convincing your boss is an issue even for minor upgrades. In a previous job, we couldn't convince him to move from Python 2.5 to 2.7.
classified 7y ago
IIRC, the Python used in the macOS vim(1) is still 2.x. So at least on a Mac it won't be possible to just move on to Py3 and forget / uninstall Py2 for the foreseeable future.
- cinnamonheart 7y ago
  I think you can just do brew install vim -- --with-override-system-vi --with-python3 To get a vim with python 3 on mac os. You are correct that it defaults to python 2, though.
- throwaway333444 7y ago
  Additionally, the next version of macOS, out this fall, will remove/hide the system version of python (python2) from users. Look under Scripting Language Runtimes https://developer.apple.com/documentation/macos_release_notes/macos_catalina_10_15_release_notes https://developer.apple.com/documentation/macos_release_note...
blebleble 7y ago
Here's how you can run your python2 code using python3 in one easy command: mv python2 python3
pjc50 7y ago
We built a product with an embedded Jython interpreter. Jython is stuck on Python2 and somewhat abandoned. So that's nice. Re: packages, one of the huge advantages of the C ecosystem has been that compiled packages are usually fine across language transitions, not only between major compiler version numbers but even from C to C++ which are much more different languages than Python2 to 3. How different would the Python transition have been if it were possible to load Python2 packages in a Python3 program?
zdw 7y ago
I've been doing a lot of Python 2 -> 3 lately, and found this to be one of the best actionable guides: https://portingguide.readthedocs.io/en/latest https://portingguide.readthedocs.io/en/latest Also, using tox on the project to run tests against both python 2.7 and multiple versions of 3 and the work goes pretty quickly.
- mixmastamyk 7y ago
  Looks like a lot of the guide assumes you want to run on Py2 and 3 concurrently. The time for that has passed to be honest. A clean port is easier to do.
  - zdw 7y ago
    For larger code bases making direct jump straight to Py3 that breaks Py2 compatibility without a transitional period can be problematic especially if it's a library. The time may have passed, but lots of code is still out there that needs to be updated.
    - mixmastamyk 7y ago
      Maintaining two branches shouldn’t be a problem. If you waited this long to port, velocity on the legacy branch can’t be especially high. Now that I think of it the legacy branch should be eol soon.
grifball 7y ago
sed -i 's/print $"[^"]*"$/print(\1)'
mixmastamyk 7y ago
Porting is a non-event for most non-large projects. In short: - First cut a new major version - Write a few tests if needed, they go a long way here. - Update to 2.7 best practices and logging - Run tests, commit - Add a few future statements, commit - Run pyflakes3 on it, fix, commit - Run under 3.x/fix until clean, commit However, if your project is huge and/or does a lot of string and bit twiddling it's excruciating. Hence the controversy between factions.

54 points by noego 7y ago

o_x 7y ago

Isn't it ironic that Sentry is one of the tools mentioned in py2->py3 migration? (Sentry is on py2 and as far as I remember they were not very optimistic about migrating)

zojirushibottle 7y ago
that's correct. sentry itself is on python 2.7, not the python client. not to pick on sentry here, but you know, my experience is that people are having a hard time migrating due to them using obscure tricks and features of python 2.7. so their code is breaking because the language evolved. the saying goes write dumb code or something because debugging is twice as hard. if there is anything to learn from all this, it's to write dumb code because maintenance is twice as hard too. that's all that is happening really!
- ggregoire 7y ago
  Why would you even build a logs collector in python? Especially if it’s your core business and you know you will need scale and reliability. You kinda shoot yourself in the foot.
  - baq 7y ago
    they're using python 2.7. perhaps started on an earlier one. maybe there was nothing except java 1.4 when they started.

LoreleiPenn 7y ago

Hopefully no more people will keep saying "learn Python 2 because 3 has almost no packages". It is so easy for people to just repeat what they heard even if that idea originated a decade ago and was valid a decade ago. And that way we got into a mess of not migrating until pass the time it is no longer supported...

swalsh 7y ago

I literally just got on a phone call to discuss our migration away from 2.7, very timely post.

ryandvm 7y ago

Python 3 and IPv6 are the poster-children of how _not_ to do a major upgrade. I'm not sure what the right way is, but if the short-term advantages of the upgrade do not outweigh the immediate pain, prepare for the matter to drag out for _decades_.

rb808 7y ago
At least Python3 brings an improvement. I still can't think of a good reason I should spend any time trying to figure out IPv6. Maybe my external router has to think about it, but that is about it. Edit: If you down vote me please say why I should care about IPv6
- Fr0styMatt88 7y ago
  The main mental stumbling block I have about IPv6 (and I know it's kind of a silly one, but it's honestly what I feel on introspection) is that I can't remember an IPv6 address off by heart. IPv4 addresses just _feel_ so much more human-consumable than IPv6 addresses. I can't imagine myself using IPv6 addresses on the command line like I do with IPv4 now. There's also the issue that honestly, I have no idea what is using IPv6 and what's using IPv4 right now. On my internal network I only ever deal with IPv4, but I have no intuition as to what is using IPv6; I couldn't tell you off the top of my head if my ISP supports it.
  - deathanatos 7y ago
    > is that I can't remember an IPv6 address off by heart. IPv4 addresses just _feel_ so much more human-consumable than IPv6 addresses. Stop remembering numbers meant for a machine, and use DNS. It will make your life so much easier. I spend ~$15/yr for my personal DNS name & hosting, and I never want to memorize an IPv4 or v6 address ever again. > I have no idea what is using IPv6 and what's using IPv4 right now. Any "end" machine (laptop, tablet, phone, server) is going to be dual stack, supporting both IPv4 and IPv6. If it is able to auto-configure an address, then it will use those. Otherwise, it won't. If a domain is IPv4-only, it'll use the IPv4 address. All of this is automatic. The big issue, for home consumers, is that a lot of ISPs are dragging their feet. They don't need anything from the customers — they just need to get it deployed & turned on. Generally, typing "what is my IP address?" into Google will tell you if you have working IPv6; it will display an IPv6 address if you do. In the cloud… some cloud vendors have been dragging their feet about rolling support out. You need to do some things, like associate an AAAA record to your domain (s.t. it resolves to an IPv6 address), and make sure things like logging can handle the new addresses, or if you implement IP blocking, that you can block those addresses/networks. If you're writing network code, you need to check that you're not assumptions about the socket type. You can also do things like HTTP proxy from an IPv6 connection to an IPv4 VLAN, e.g., I think w/ an ELB. That is, client <-- HTTP/IPv6 --> ELB <-- HTTP/IPv4 --> backend server which allows a partial upgrade. None of it is terribly hard, but typical project management puts upgrading to future tech in the perpetual backlog.

zitterbewegung 7y ago

The big issue of ports like these is not the tutorial but to justify that to your boss. From enterprise to a self run startup you have to see if it’s worth it .

GrayTzar 7y ago
Hi, I'm one of the STX Next content crew. We actually have a companion piece to this that goes over the reasons why you should migrate: https://stxnext.com/blog/2019/07/30/why-migrate-from-python-2-to-python-3/ https://stxnext.com/blog/2019/07/30/why-migrate-from-python-... Maybe that would be useful for a conversation with one's boss.
BeetleB 7y ago
> The big issue of ports like these is not the tutorial but to justify that to your boss. I think that's a red herring. Convincing your boss is an issue even for minor upgrades. In a previous job, we couldn't convince him to move from Python 2.5 to 2.7.

classified 7y ago

IIRC, the Python used in the macOS vim(1) is still 2.x. So at least on a Mac it won't be possible to just move on to Py3 and forget / uninstall Py2 for the foreseeable future.

cinnamonheart 7y ago
I think you can just do brew install vim -- --with-override-system-vi --with-python3 To get a vim with python 3 on mac os. You are correct that it defaults to python 2, though.
throwaway333444 7y ago
Additionally, the next version of macOS, out this fall, will remove/hide the system version of python (python2) from users. Look under Scripting Language Runtimes https://developer.apple.com/documentation/macos_release_notes/macos_catalina_10_15_release_notes https://developer.apple.com/documentation/macos_release_note...

blebleble 7y ago

Here's how you can run your python2 code using python3 in one easy command: mv python2 python3

pjc50 7y ago

We built a product with an embedded Jython interpreter. Jython is stuck on Python2 and somewhat abandoned. So that's nice. Re: packages, one of the huge advantages of the C ecosystem has been that compiled packages are usually fine across language transitions, not only between major compiler version numbers but even from C to C++ which are much more different languages than Python2 to 3. How different would the Python transition have been if it were possible to load Python2 packages in a Python3 program?

zdw 7y ago

I've been doing a lot of Python 2 -> 3 lately, and found this to be one of the best actionable guides: https://portingguide.readthedocs.io/en/latest https://portingguide.readthedocs.io/en/latest Also, using tox on the project to run tests against both python 2.7 and multiple versions of 3 and the work goes pretty quickly.

mixmastamyk 7y ago
Looks like a lot of the guide assumes you want to run on Py2 and 3 concurrently. The time for that has passed to be honest. A clean port is easier to do.
- zdw 7y ago
  For larger code bases making direct jump straight to Py3 that breaks Py2 compatibility without a transitional period can be problematic especially if it's a library. The time may have passed, but lots of code is still out there that needs to be updated.
  - mixmastamyk 7y ago
    Maintaining two branches shouldn’t be a problem. If you waited this long to port, velocity on the legacy branch can’t be especially high. Now that I think of it the legacy branch should be eol soon.

grifball 7y ago

sed -i 's/print $"[^"]*"$/print(\1)'

mixmastamyk 7y ago

Porting is a non-event for most non-large projects. In short: - First cut a new major version - Write a few tests if needed, they go a long way here. - Update to 2.7 best practices and logging - Run tests, commit - Add a few future statements, commit - Run pyflakes3 on it, fix, commit - Run under 3.x/fix until clean, commit However, if your project is huge and/or does a lot of string and bit twiddling it's excruciating. Hence the controversy between factions.