5 ms·

Disagree; Python's hardest problem is packaging, followed closely by bikeshedding over networking libraries and an ever-growing dichotomy between the Python 3 a

Disagree; Python's hardest problem is packaging, followed closely by bikeshedding over networking libraries and an ever-growing dichotomy between the Python 3 and Python 2 universes of code. You can skip all conversation about the GIL and threads neatly by simply preferring a different concurrency model. There are plenty of ways to do this (see above re: bikesheds and their colors), but being permanently tied to CPython and the threading module is increasingly uncommon for professional Python, and it isn't as unavoidable as things like networking or even which language you're going to use. Edit: I see that the author's in this thread. Nicely written article, but a tad hyperbolic.

leoh 13y ago
I've enjoyed the Python networking libraries I've used. A lot. Could you be more specific? Also, I'm not quite clear on your issue with packaging. Can you be more specific? Compared to C, I find importing existing code to be nearly heavenly. Regarding backwards compatibility, maybe it's an issue, maybe it's not. On the one hand it's frustrating and some people will make it an issue. On the other hand, I can't help but think of cases like Apple, where they were the first to abandon modems, serial ports, diskette drives, and CD drives—but probably made the right decision.
- zanny 13y ago
  Hes not talking about Python imports, hes talking about pulling Python packages through pip or one of the dozen utilities to do so from pypi. Though I do feel pip has been standardized on enough to solve that problem.
- MostAwesomeDude 13y ago
  There are two options for Python networking: Twisted, and things that are not Twisted. Guess which one most people pick. What do you use to package? distutils, setuptools, distribute, distutils2, pkgutils...? How to choose?
  - d0mine 13y ago
    start with distutils which is in stdlib and if it doesn't provide enough features; use setuptools (distribute and setuptools are merged) that extends distutils functionality.
coolsunglasses 13y ago
Packaging is fine unless you're ignoring the current state of tooling. Not sure what you mean by networking unless you're talking about gevent/twisted/tornado/diesel which are really proxies for the problem of concurrency (GIL) rather than expressly about networking.
- scw 13y ago
  Packaging is starting to come together, but still has a ways to go. The Setuptools-Distribute merge was just announced a couple months ago, we still have 6 ways of doing things that are mostly incompatible. Packaging is mostly solved if you're on Linux, sort of solved if you're on OS X, and a horrible mess if you're on Windows. Perhaps if you can live in the world of pure Python it's good, but once you have C-based dependencies, it quickly becomes a mess. Compared to something like CRAN and the R community, it's clear Python could be improved in this area.
parfe 13y ago
The transition was always planned to take several years and I can't think of any major project which decided not to port to python 3. http://py3ksupport.appspot.com/ http://py3ksupport.appspot.com/ and https://python3wos.appspot.com/ https://python3wos.appspot.com/ both list package compatibility. What issues do you see with python2 to 3 transition?
- thezilch 13y ago
  To be fair, the Python 3 transition timer shouldn't have had its tick starting until more recent versions of Py3. This won't be true for all Python communities (eg. Mathematics / Research), but until the more sane implementation of bytes/strings/unicode, the transition for most web frameworks or protocols was a mess. However, from the later list, there are some really damning packages. For one, we're going to have transition off Twistd, which is much more than just an event loop. Py3 might not have enough advantage(s) to compel us to make that transition.
  - parfe 13y ago
    http://twistedmatrix.com/trac/milestone/Python-3.x http://twistedmatrix.com/trac/milestone/Python-3.x So, just wait to migrate until twisted is ready?
    - thezilch 13y ago
      Honestly, I didn't check the dozen-plus packages pending update for us and then someone else cutting their teeth on it for a while, but Twisted was clearly marked as not being updated for Py3. But yes, clearly we're waiting; apparently Guido also wants to take a stab at the async batteries, which could help or hurt, if the Twisted contributors think they will be affected. To the point of the comment you first responded to, there is lot of issues not being "solved" with respect to network libs, their upgrading to Py3, and Py3 looking to step into that arena. In other words, there is no horizon for networking libs updating, and they may feel even less motivated if the Py3 crowd is saying, "don't worry about it, because we are going to standardize our way." Does the Py3 transition-clock only start then?
craigyk 13y ago
Absolutely agree.... in other words the community. To that I would add the ever growing dichotomy and fragmentation regarding improving performance between people solving this (CPython vs. PyPy vs. Continuum). I'd trade a python that was 5-10X faster (and could use numpy/scipy) over one that didn't have a GIL.
- simonster 13y ago
  Right. If you're spending a lot of time in the CPython interpreter, you probably don't care too much about performance anyway. The CPython interpreter is a hell of a lot slower than using Numba or Cython (which can operate without the GIL) or PyPy (which is an completely separate codebase from CPython, although it still has a GIL in its typical incarnation), or many alternative languages. As soon as you're out of the interpreter, the GIL ceases to be an issue (e.g. NumPy releases the GIL while performing array operations).
- pwang 13y ago
  I don't think it's quite accurate to frame the current state of innovation in High-Performance Python as being one in which the principal actors are fundamentally opposed to each other. Nothing that we do at Continuum detracts from the CPython ecosystem; in fact, our tools are designed to specifically provide maximal utility, as quickly as possible, to certain (significant) groups of Python users. We're not tackling anything nearly as ambitious as PyPy's meta-tracing JIT or generic STM. Much of our work builds on tools and lessons learned over the last decade, including work on Cython, Mython, LLVMpy, etc. etc. As for the GIL, the OP is right that it's a CPython implementation problem, not a structural one that has to exist in Python. There is some exciting work that has been done in the direction of fixing this at the implementation level, and we are excited about seeing how far we can take it. Stay tuned...
zanny 13y ago
There isn't a growing dichotomy, there is a migration from 2 to 3. We have passed critical library mass I feel in Python3 space and now it is just waiting for the world to catch up and just use 3 already.
Chris2048 13y ago
> bikeshedding Not just network libraries; How many cli modules are there now?

leoh 13y ago

I've enjoyed the Python networking libraries I've used. A lot. Could you be more specific? Also, I'm not quite clear on your issue with packaging. Can you be more specific? Compared to C, I find importing existing code to be nearly heavenly. Regarding backwards compatibility, maybe it's an issue, maybe it's not. On the one hand it's frustrating and some people will make it an issue. On the other hand, I can't help but think of cases like Apple, where they were the first to abandon modems, serial ports, diskette drives, and CD drives—but probably made the right decision.

zanny 13y ago
Hes not talking about Python imports, hes talking about pulling Python packages through pip or one of the dozen utilities to do so from pypi. Though I do feel pip has been standardized on enough to solve that problem.
MostAwesomeDude 13y ago
There are two options for Python networking: Twisted, and things that are not Twisted. Guess which one most people pick. What do you use to package? distutils, setuptools, distribute, distutils2, pkgutils...? How to choose?
- d0mine 13y ago
  start with distutils which is in stdlib and if it doesn't provide enough features; use setuptools (distribute and setuptools are merged) that extends distutils functionality.

coolsunglasses 13y ago

Packaging is fine unless you're ignoring the current state of tooling. Not sure what you mean by networking unless you're talking about gevent/twisted/tornado/diesel which are really proxies for the problem of concurrency (GIL) rather than expressly about networking.

scw 13y ago
Packaging is starting to come together, but still has a ways to go. The Setuptools-Distribute merge was just announced a couple months ago, we still have 6 ways of doing things that are mostly incompatible. Packaging is mostly solved if you're on Linux, sort of solved if you're on OS X, and a horrible mess if you're on Windows. Perhaps if you can live in the world of pure Python it's good, but once you have C-based dependencies, it quickly becomes a mess. Compared to something like CRAN and the R community, it's clear Python could be improved in this area.

parfe 13y ago

The transition was always planned to take several years and I can't think of any major project which decided not to port to python 3. http://py3ksupport.appspot.com/ http://py3ksupport.appspot.com/ and https://python3wos.appspot.com/ https://python3wos.appspot.com/ both list package compatibility. What issues do you see with python2 to 3 transition?

thezilch 13y ago
To be fair, the Python 3 transition timer shouldn't have had its tick starting until more recent versions of Py3. This won't be true for all Python communities (eg. Mathematics / Research), but until the more sane implementation of bytes/strings/unicode, the transition for most web frameworks or protocols was a mess. However, from the later list, there are some really damning packages. For one, we're going to have transition off Twistd, which is much more than just an event loop. Py3 might not have enough advantage(s) to compel us to make that transition.
- parfe 13y ago
  http://twistedmatrix.com/trac/milestone/Python-3.x http://twistedmatrix.com/trac/milestone/Python-3.x So, just wait to migrate until twisted is ready?
  - thezilch 13y ago
    Honestly, I didn't check the dozen-plus packages pending update for us and then someone else cutting their teeth on it for a while, but Twisted was clearly marked as not being updated for Py3. But yes, clearly we're waiting; apparently Guido also wants to take a stab at the async batteries, which could help or hurt, if the Twisted contributors think they will be affected. To the point of the comment you first responded to, there is lot of issues not being "solved" with respect to network libs, their upgrading to Py3, and Py3 looking to step into that arena. In other words, there is no horizon for networking libs updating, and they may feel even less motivated if the Py3 crowd is saying, "don't worry about it, because we are going to standardize our way." Does the Py3 transition-clock only start then?

craigyk 13y ago

Absolutely agree.... in other words the community. To that I would add the ever growing dichotomy and fragmentation regarding improving performance between people solving this (CPython vs. PyPy vs. Continuum). I'd trade a python that was 5-10X faster (and could use numpy/scipy) over one that didn't have a GIL.

simonster 13y ago
Right. If you're spending a lot of time in the CPython interpreter, you probably don't care too much about performance anyway. The CPython interpreter is a hell of a lot slower than using Numba or Cython (which can operate without the GIL) or PyPy (which is an completely separate codebase from CPython, although it still has a GIL in its typical incarnation), or many alternative languages. As soon as you're out of the interpreter, the GIL ceases to be an issue (e.g. NumPy releases the GIL while performing array operations).
pwang 13y ago
I don't think it's quite accurate to frame the current state of innovation in High-Performance Python as being one in which the principal actors are fundamentally opposed to each other. Nothing that we do at Continuum detracts from the CPython ecosystem; in fact, our tools are designed to specifically provide maximal utility, as quickly as possible, to certain (significant) groups of Python users. We're not tackling anything nearly as ambitious as PyPy's meta-tracing JIT or generic STM. Much of our work builds on tools and lessons learned over the last decade, including work on Cython, Mython, LLVMpy, etc. etc. As for the GIL, the OP is right that it's a CPython implementation problem, not a structural one that has to exist in Python. There is some exciting work that has been done in the direction of fixing this at the implementation level, and we are excited about seeing how far we can take it. Stay tuned...

zanny 13y ago

There isn't a growing dichotomy, there is a migration from 2 to 3. We have passed critical library mass I feel in Python3 space and now it is just waiting for the world to catch up and just use 3 already.

Chris2048 13y ago

> bikeshedding Not just network libraries; How many cli modules are there now?