5 ms·

The breakup between Python 2 and 3 has been very slow and painful. Python devs know that, and that's why they won't break compatibility in such big and drastic

The breakup between Python 2 and 3 has been very slow and painful. Python devs know that, and that's why they won't break compatibility in such big and drastic way ever again. I'm glad that we're starting to see light at the end of the tunnel. I find myself using Python 3 in most of my projects. Some times I still have to resort to Python 2 when some dependency is not ready but those cases are more rare every day. Also it's frustrating to use Python 2 when many cool features are now Python 3 only. It will take some more time, but I'm sure that the transition will eventually be completed.

thearn4 11y ago
I'm happy that we finally updated a lot of our code at work to be Python 3 compatible last year. The conversion was not that bad in the end. However, under Python 3 a number of benchmarks are slower, which we do need to look into some more. Has anyone else experienced this, specifically with scientific computing code migrating from 2 to 3? Ultimately, we test in 2 and 3 via CI, but still primarily develop using 2.
- kmike84 11y ago
  A data point: `scrapy bench` is about 1.5x - 2x slower in Python 3.5 than in Python 2.7. This command starts a simple HTTP server which serves HTML pages linked in arbitrary depth, and crawls it using Scrapy. Profiler shows URL parsing is much slower in Python 3 for some reason, I'm not sure why. I haven't digged further. Python 3 code also have to encode/decode strings more often, but the overhead was not significant for `scrapy bench`.
  - ogrisel 11y ago
    I think you should google the python.org bug tracker with the function names that seems to exhibit a performance regression according to your profiling session. If don't find any existing issue, please open a new one with a minimalist reproduction script that demonstrates the performance regression.
    - kmike84 11y ago
      Yeah, totally agree; I did it for io.BytesIO vs cStringIO performance issue (https://news.ycombinator.com/item?id=11037213 https://news.ycombinator.com/item?id=11037213), and Python core devs fixed it very fast even though the fix was far from trivial. You have to wait a year or so before next release to use the fix, but that's expected. For URL parsing it was just a quick profiling session. Also, stdlib URL parsing was a bottleneck even in Python 2. It takes time to benchmark it properly and submit a meaningful issue to bug tracker or explain the problem in python-dev. It is in my todo list, but you only have so much time :)
- kmike84 11y ago
  For scientific code I've seen slowdowns, but in the end they turn out to be unrelated to Python 3 - e.g. numpy compiled against different BLAS libraries in Python 2.x and 3.x. There also was a gotcha with replacing cStringIO with io.BytesIO: the latter copied the data even for read-only cases (when you want to have a file-like interface for a binary object), so it can be much slower. To make things worse, Python 3.x didn't have cStringIO alternative which can be used to provide file-like interface without overhead, and most porting guides suggest using io.BytesIO instead of cStringIO in Python 3. So libraries like tornado, django and flask are affected, and likely some scientific libraries as well. This problem should be fixed in Python 3.5: io.BytesIO becomes as efficient as cStringIO for read-only cases since Python 3.5.
  - aidos 11y ago
    You sound like a good person to ask. I use numpy and I haven't looked into the BLAS/LAPACK details but I understand there are different ways of setting up numpy for optimal performance. Any advice on the best way to approach this? At the moment I more or less: apt-get install liblapack-dev libopenblas-dev pip install numpy
    - dagw 11y ago
      Honestly it is very dependent on exactly what you are doing. For general purpose computing on a general purpose machine you're not going to do much better on average than an up to date openblas. In some cases, especially in the parallel case, Intel's MKL BLAS is slightly faster (but in some cases it is also slower). There is also scikit.cuda which wraps Nvidia's cuBLAS and which can be very fast in certain cases, but isn't in any way a drop in replacement for openblas. Then there's NumbaPro (a commercial product) from Continuum Analytics which is an LLVM backed JIT that attempts to automatically speed up your numpy coda and can automatically make your code use cuBLAS where it makes sense to do so.
      - aidos 11y ago
        Ok. Think I'll leave well enough alone then!
bootload 11y ago
"The breakup between Python 2 and 3 has been very slow and painful. Python devs know that, and that's why they won't break compatibility in such big and drastic way ever again." Fork, don't break, cf Pillow (PIL fork) ~ http://python-pillow.github.io/ http://python-pillow.github.io/ What are the most commonly used Py2 packages that need to be Py3?
- jacobolus 11y ago
  > What are the most commonly used Py2 packages that need to be Py3? https://python3wos.appspot.com https://python3wos.appspot.com http://py3readiness.org http://py3readiness.org
  - nerdwaller 11y ago
    For most of the ones that aren't compatible I've found an alternative or monkey patched a method to support python 3 (I'm looking at supervisor -> circus and flask-session -> monkey patch because the author seems to have abandoned it).
- Siecje 11y ago
  For me it's supervisor and beanstalkc and fabric but you can run your fabric scripts with Python2 easily enough.
- brownbat 11y ago
  PaiMei was a pretty good toolkit, but it doesn't even work with anything after 2.4, at least not out of the box. So maybe that's not even an example of the 2/3 split, it's just old.

thearn4 11y ago

I'm happy that we finally updated a lot of our code at work to be Python 3 compatible last year. The conversion was not that bad in the end. However, under Python 3 a number of benchmarks are slower, which we do need to look into some more. Has anyone else experienced this, specifically with scientific computing code migrating from 2 to 3? Ultimately, we test in 2 and 3 via CI, but still primarily develop using 2.

kmike84 11y ago
A data point: `scrapy bench` is about 1.5x - 2x slower in Python 3.5 than in Python 2.7. This command starts a simple HTTP server which serves HTML pages linked in arbitrary depth, and crawls it using Scrapy. Profiler shows URL parsing is much slower in Python 3 for some reason, I'm not sure why. I haven't digged further. Python 3 code also have to encode/decode strings more often, but the overhead was not significant for `scrapy bench`.
- ogrisel 11y ago
  I think you should google the python.org bug tracker with the function names that seems to exhibit a performance regression according to your profiling session. If don't find any existing issue, please open a new one with a minimalist reproduction script that demonstrates the performance regression.
  - kmike84 11y ago
    Yeah, totally agree; I did it for io.BytesIO vs cStringIO performance issue (https://news.ycombinator.com/item?id=11037213 https://news.ycombinator.com/item?id=11037213), and Python core devs fixed it very fast even though the fix was far from trivial. You have to wait a year or so before next release to use the fix, but that's expected. For URL parsing it was just a quick profiling session. Also, stdlib URL parsing was a bottleneck even in Python 2. It takes time to benchmark it properly and submit a meaningful issue to bug tracker or explain the problem in python-dev. It is in my todo list, but you only have so much time :)
kmike84 11y ago
For scientific code I've seen slowdowns, but in the end they turn out to be unrelated to Python 3 - e.g. numpy compiled against different BLAS libraries in Python 2.x and 3.x. There also was a gotcha with replacing cStringIO with io.BytesIO: the latter copied the data even for read-only cases (when you want to have a file-like interface for a binary object), so it can be much slower. To make things worse, Python 3.x didn't have cStringIO alternative which can be used to provide file-like interface without overhead, and most porting guides suggest using io.BytesIO instead of cStringIO in Python 3. So libraries like tornado, django and flask are affected, and likely some scientific libraries as well. This problem should be fixed in Python 3.5: io.BytesIO becomes as efficient as cStringIO for read-only cases since Python 3.5.
- aidos 11y ago
  You sound like a good person to ask. I use numpy and I haven't looked into the BLAS/LAPACK details but I understand there are different ways of setting up numpy for optimal performance. Any advice on the best way to approach this? At the moment I more or less: apt-get install liblapack-dev libopenblas-dev pip install numpy
  - dagw 11y ago
    Honestly it is very dependent on exactly what you are doing. For general purpose computing on a general purpose machine you're not going to do much better on average than an up to date openblas. In some cases, especially in the parallel case, Intel's MKL BLAS is slightly faster (but in some cases it is also slower). There is also scikit.cuda which wraps Nvidia's cuBLAS and which can be very fast in certain cases, but isn't in any way a drop in replacement for openblas. Then there's NumbaPro (a commercial product) from Continuum Analytics which is an LLVM backed JIT that attempts to automatically speed up your numpy coda and can automatically make your code use cuBLAS where it makes sense to do so.
    - aidos 11y ago
      Ok. Think I'll leave well enough alone then!

bootload 11y ago

"The breakup between Python 2 and 3 has been very slow and painful. Python devs know that, and that's why they won't break compatibility in such big and drastic way ever again." Fork, don't break, cf Pillow (PIL fork) ~ http://python-pillow.github.io/ http://python-pillow.github.io/ What are the most commonly used Py2 packages that need to be Py3?

jacobolus 11y ago
> What are the most commonly used Py2 packages that need to be Py3? https://python3wos.appspot.com https://python3wos.appspot.com http://py3readiness.org http://py3readiness.org
- nerdwaller 11y ago
  For most of the ones that aren't compatible I've found an alternative or monkey patched a method to support python 3 (I'm looking at supervisor -> circus and flask-session -> monkey patch because the author seems to have abandoned it).
Siecje 11y ago
For me it's supervisor and beanstalkc and fabric but you can run your fabric scripts with Python2 easily enough.
brownbat 11y ago
PaiMei was a pretty good toolkit, but it doesn't even work with anything after 2.4, at least not out of the box. So maybe that's not even an example of the 2/3 split, it's just old.