13 ms·

>However, many are subjective preferences Certainly, it is titled "Problems I Have" for a reason. :-) I do not expect everyone to agree with me, but it is what

by darkf 10y ago

>However, many are subjective preferences Certainly, it is titled "Problems I Have" for a reason. :-) I do not expect everyone to agree with me, but it is what I feel I personally lack when using it quite a lot. > I imagine this statement could offend some of the smart and hard-working people who are working on improving the python language. That was certainly not my intention -- as stated, I do love the language and appreciate all work going into it. I do not intend to undermine their efforts, just point out some of my perceived design flaws. >- the community does not agree with the author's subjective idea of what Python should look like I think we all agree there should be a good solution to concurrency (and "stackless" variants which power eventlet, etc. have been used for ages; as has Twisted, of which asyncio is not a sufficient clone.), parallelism, etc. The standard library in general encourages use of higher-order functions and concepts borrowed primarily from FPLs (see: comprehensions, map/reduce, sort, etc.) I could not imagine seeing them backtracking on this -- it only helps them to go further in that direction. >- the solutions to a problem (I'm thinking GIL) come with a lot of consequences which are not readily acceptable I did not propose a solution because there are many, as you note; there are, however, implementations with decent solutions like AFAIK Jython. >- solving some of the issues would exacerbate backwards compatibility. Such as what?

jsmeaton 10y ago
> with decent solutions like AFAIK Jython. >>- solving some of the issues would exacerbate backwards compatibility. Such as what? Jython can't load native C extensions which should be GIL aware. Most programs, and the python interpreter itself, aren't thread safe so suddenly removing the GIL would break a lot of programs. I agree with you that the concurrency story for python sucks, but claiming solutions could exist without breaking back-compat is just not right.
- al2o3cr 10y ago
  FWIW, the folks working on TruffleRuby have done some amazing things on this front - essentially making an interpreter for Ruby C extensions that JITs to code that's more-or-less identical in performance to the original native version.
- darkf 10y ago
  >but claiming solutions could exist without breaking back-compat is just not right. I mean, you have a good point on C extensions but code relying on them (without a really portable API) is almost never going to be forward compatible anyway. (There are still quite a few C extensions not up to CPython 3 yet.)
- Chris2048 10y ago
  Is Jython good though? Last I used it is had issues keeping pace, i.e. demonstrable memory issue that took a long time to fix, lagged considerably behind python 2/3 versions. It's also worth noting, that as an essentially transcompiled language, you need to have a good appreciation of Java machinery, in which case languages like Groovy provide good competition.
  - vorg 10y ago
    When talking about JVM languages suitable for building systems, Jython isn't generally mentioned for the reasons you give, nor is Apache Groovy. Those two are good for scripting, e.g. testing Java classes, build scripts, glue code. Besides Java, languages like Clojure, Scala, and Kotlin are usually considered as systems languages on the JVM.
    - Chris2048 10y ago
      True, but I believe Groovy and Jython compete for the same space, if not system-building.
acqq 10y ago
If you claim it's "incompetence" (it's not) then publish the patches that don't break anything but significantly speed up it, for example. Because you complain "it' slow." It's not what you think it is. It's Python, not a toy language used by nobody. Just first try yourself to "fix" Python and keep its existing users happy by not breaking anything for them, then write about it.
- darkf 10y ago
  Cool, I think you completely dodged the point. I never said it was slow because they were incompetent.
  - BlackFingolfin 10y ago
    Your post quite strongly alludes to it being either due to incompetence, or politics, or both. So I think grandparent has a very valid point, and you might want to change the tone of your post a bit; then it'll produce fewer knee-jerk reactions, and might be taken more seriously.
    - darkf 10y ago
      Nah, just people connecting that sentiment with other statements. It should be cleared up since it's causing some mass confusion. It's funny because I preface it by saying "Remember that it's a matter of opinion" (and, well, the title alone) and people come out of the woodwork completely disregarding this, or outright misinterpreting sections of it. I maintain that a large reader base here does not actually... read.
      - kemayo 10y ago
        > I maintain that a large reader base here does not actually... read. I would disagree. Although there's often a fair number of commenters who clearly read the title of an article, and then just start commenting on that, in this case we can see that people have read (at a minimum) your opening statement and whichever list item they're taking issue with. I believe that if a large set of people are misinterpreting what I've written, it's a sign that I probably wrote it poorly. Not in the sense of arguing for the wrong thing, but in the sense that I'm not conveying my argument well enough. This is easy to do, because when I'm writing something, I know what I mean. This sounds obvious, but it's hard to read what I've written tabula rasa, without bringing that "of course I mean X" view to it. So, the feedback you're getting here is that your opening statement colors the rest of your piece. That "Incompetence? Politics? Both?" aside obviously makes a large subset of readers assume that you're saying "all of these complaints in my list must be unaddressed due to incompetence or politics". That's at odds with the later "just an opinion" statement, and people are sticking with the more-inflammatory initial claim.
dev360 10y ago
I was taken back by this rather harsh treatment of Python. Is it really realistic to 'have it all'? I'm fully aware that I'd have to go to crazier languages if I want parallelism or speed. For what Python is, it offers me reasonable tradeoffs (mostly slanted towards productivity).. Regarding the FP comments, since it lacks TCO, my take away has always been that Python can only ever become a quasi-functional language. Its hard to be more than that in its current state. Anyways. These questions made me want to ask you - what languages do you think are better in comparison?
- xapata 10y ago
  The lack of tail-call optimization to make the CPython interpreter simpler and debugging easier by preserving the call stack. It was a choice, not an oversight.
  - KingMob 10y ago
    Why not just make it a dev/production flag, then?
    - sabauma 10y ago
      Probably because many tail-recursive functions _rely_ on tail-call elimination working reliably. Without also having an unbounded call stack, disabling tail-call elimination will likely just cause your programs to crash.
  - pinjo 10y ago
    From a debugging viewpoint, this does not make sense, there is usually no interesting information in the in between frames. TCE can also make debugging easier, how useful is a stack trace of 1000 lines consisting of ... File "bla.py", line 4, in fib return fib(n - 1) + fib(n - 2) File "bla.py", line 4, in fib return fib(n - 1) + fib(n - 2) File "bla.py", line 4, in fib return fib(n - 1) + fib(n - 2) ... Not so much I think.
    - sabauma 10y ago
ProblemFactory 10y ago
> I did not propose a solution because there are many, as you note; there are, however, implementations with decent solutions like AFAIK Jython. There are no solutions that satisfy everyone that I am aware of yet. Guido has said in the past that he'd be happy to get rid of the GIL, and would merge a patch that solves it, as long as: * It does not reduce the performance of single-threaded Python code. * It stays compatible with all existing pure Python code and C extensions. But in practice, GIL is not that much of an issue for many types of applications where Python is popular. * It's not an issue for web apps, because these are typically served from multiple physical servers each running multiple python processes. These do not share a GIL anyway, and "thread safety" is pushed to database transactions. * It is not an issue for apps which spend most of the time doing I/O. Most IO libraries release the GIL, and other threads can run while you're waiting for results from the database. * It is not an issue for data science doing heavy number crunching with numpy and everything built on top of numpy. Numpy releases the GIL while doing large computations in C. * It is not an issue for small scripts, as a "better than Bash". The GIL is only an issue for apps that do heavy computation in pure Python code, and need parallelism within a single process (socket servers? text data processing?). As a result, many Python users just don't find it a big enough problem to be worth solving, if the solution comes with downsides for their use cases.
- xapata 10y ago
  The GIL is only a problem because there's "no free lunch" -- no single strategy that is best in all cases.
- diek 10y ago
  > It is not an issue for apps which spend most of the time doing I/O. This is a common misconception that doesn't seem to be backed up by any data. Dave Beazley did a number of performance tests with profiling, looking at GIL contention in a multi-core scenario: http://www.dabeaz.com/python/GIL.pdf http://www.dabeaz.com/python/GIL.pdf The results were that even IO-bound workloads still suffered because of the poor implementation of the GIL (details on slide 35 or so). This was an issue up until Python 3.2 (!) when a new GIL implementation was added, which he also profiled: http://www.dabeaz.com/python/NewGIL.pdf http://www.dabeaz.com/python/NewGIL.pdf
KirinDave 10y ago
Once, I was on mailing lists with GVR and other language contributors, and have seem him go off deeply into functional programming. Some of the stuff he wrote went right over my head. For someone who declares he hates functional programming even at the most basic levels of data stream manipulation, he knows it quite well. I've always been frustrated with this disconnect, even more acutely than you have, because I know GVR is being disingenuous when he says, "I don't get it." He absolutely does. He thinks other people won't.
takeda 10y ago
>>- the solutions to a problem (I'm thinking GIL) come with a lot of consequences which are not readily acceptable > I did not propose a solution because there are many, as you note; there are, however, implementations with decent solutions like AFAIK Jython. >>- solving some of the issues would exacerbate backwards compatibility. > Such as what? Let's use the GIL for example. The reason that is still present is not that it is hard to remove it. It already was removed in the past. The problem is that when GIL is replaced with smaller locks, the python becomes much slower, because of some features and behavior that people got used to. One could make python faster by changing behavior, but then it would break existing code and C extensions. There's no easy way to do this without sacrificing something else. Larry Hastings work on removing GIL and has interesting talk about it[1] [1] https://www.youtube.com/watch?v=fgWUwQVoLHo https://www.youtube.com/watch?v=fgWUwQVoLHo
- ehsankia 10y ago
  That talk was a great overview of the issue. Most people who are ignorant about the subject always assume the GIL is stupid and useless, but the GIL allows Python to be extremely fast in single threaded scenarios, and any attempt to remove it introduces at least 20% slow downs. And C extension support is also a huge factor as you mention. All of Python's scientific modules would be lost if they broke that. The author assumes these are simple problems that can be solved but aren't because of politics or incompetence, but some of the smartest mind have attempted and failed. I'd like the author to try and come up to solutions or at least draft ideas for how each of his points can be fixed. A lot of those are easy to state but difficult to solve without breaking more stuff.
pvg 10y ago
The standard library in general encourages use of higher-order functions and concepts borrowed primarily from FPLs (see: comprehensions, map/reduce, sort, etc.) I could not imagine seeing them backtracking on this -- it only helps them to go further in that direction. I don't think the standard library really 'encourages' this in a way that's different from most languages that support first class (rather than 'higher order') functions. If you consider python's evolution, its support for many common programming paradigms was somewhat haphazard and weak and developed over time, mostly pragmatically. OO has become stronger, the 80% use case of common functional idioms is covered by comprehensions, etc. Ill thought out features (e.g. terrible lambdas) have become de-emphasised. The choices are extensively document, even if not everyone cup of tea so it seems both glib and inaccurate to say (re: FP) 'it only helps them to go further in that direction'. How does it help them?

by darkf 10y ago

jsmeaton 10y ago

> with decent solutions like AFAIK Jython. >>- solving some of the issues would exacerbate backwards compatibility. Such as what? Jython can't load native C extensions which should be GIL aware. Most programs, and the python interpreter itself, aren't thread safe so suddenly removing the GIL would break a lot of programs. I agree with you that the concurrency story for python sucks, but claiming solutions could exist without breaking back-compat is just not right.

al2o3cr 10y ago
FWIW, the folks working on TruffleRuby have done some amazing things on this front - essentially making an interpreter for Ruby C extensions that JITs to code that's more-or-less identical in performance to the original native version.
darkf 10y ago
>but claiming solutions could exist without breaking back-compat is just not right. I mean, you have a good point on C extensions but code relying on them (without a really portable API) is almost never going to be forward compatible anyway. (There are still quite a few C extensions not up to CPython 3 yet.)
Chris2048 10y ago
Is Jython good though? Last I used it is had issues keeping pace, i.e. demonstrable memory issue that took a long time to fix, lagged considerably behind python 2/3 versions. It's also worth noting, that as an essentially transcompiled language, you need to have a good appreciation of Java machinery, in which case languages like Groovy provide good competition.
- vorg 10y ago
  When talking about JVM languages suitable for building systems, Jython isn't generally mentioned for the reasons you give, nor is Apache Groovy. Those two are good for scripting, e.g. testing Java classes, build scripts, glue code. Besides Java, languages like Clojure, Scala, and Kotlin are usually considered as systems languages on the JVM.
  - Chris2048 10y ago
    True, but I believe Groovy and Jython compete for the same space, if not system-building.

acqq 10y ago

If you claim it's "incompetence" (it's not) then publish the patches that don't break anything but significantly speed up it, for example. Because you complain "it' slow." It's not what you think it is. It's Python, not a toy language used by nobody. Just first try yourself to "fix" Python and keep its existing users happy by not breaking anything for them, then write about it.

darkf 10y ago
Cool, I think you completely dodged the point. I never said it was slow because they were incompetent.
- BlackFingolfin 10y ago
  Your post quite strongly alludes to it being either due to incompetence, or politics, or both. So I think grandparent has a very valid point, and you might want to change the tone of your post a bit; then it'll produce fewer knee-jerk reactions, and might be taken more seriously.
  - darkf 10y ago
    Nah, just people connecting that sentiment with other statements. It should be cleared up since it's causing some mass confusion. It's funny because I preface it by saying "Remember that it's a matter of opinion" (and, well, the title alone) and people come out of the woodwork completely disregarding this, or outright misinterpreting sections of it. I maintain that a large reader base here does not actually... read.
    - kemayo 10y ago
      > I maintain that a large reader base here does not actually... read. I would disagree. Although there's often a fair number of commenters who clearly read the title of an article, and then just start commenting on that, in this case we can see that people have read (at a minimum) your opening statement and whichever list item they're taking issue with. I believe that if a large set of people are misinterpreting what I've written, it's a sign that I probably wrote it poorly. Not in the sense of arguing for the wrong thing, but in the sense that I'm not conveying my argument well enough. This is easy to do, because when I'm writing something, I know what I mean. This sounds obvious, but it's hard to read what I've written tabula rasa, without bringing that "of course I mean X" view to it. So, the feedback you're getting here is that your opening statement colors the rest of your piece. That "Incompetence? Politics? Both?" aside obviously makes a large subset of readers assume that you're saying "all of these complaints in my list must be unaddressed due to incompetence or politics". That's at odds with the later "just an opinion" statement, and people are sticking with the more-inflammatory initial claim.

dev360 10y ago

I was taken back by this rather harsh treatment of Python. Is it really realistic to 'have it all'? I'm fully aware that I'd have to go to crazier languages if I want parallelism or speed. For what Python is, it offers me reasonable tradeoffs (mostly slanted towards productivity).. Regarding the FP comments, since it lacks TCO, my take away has always been that Python can only ever become a quasi-functional language. Its hard to be more than that in its current state. Anyways. These questions made me want to ask you - what languages do you think are better in comparison?

xapata 10y ago
The lack of tail-call optimization to make the CPython interpreter simpler and debugging easier by preserving the call stack. It was a choice, not an oversight.
- KingMob 10y ago
  Why not just make it a dev/production flag, then?
  - sabauma 10y ago
    Probably because many tail-recursive functions _rely_ on tail-call elimination working reliably. Without also having an unbounded call stack, disabling tail-call elimination will likely just cause your programs to crash.
- pinjo 10y ago
  From a debugging viewpoint, this does not make sense, there is usually no interesting information in the in between frames. TCE can also make debugging easier, how useful is a stack trace of 1000 lines consisting of ... File "bla.py", line 4, in fib return fib(n - 1) + fib(n - 2) File "bla.py", line 4, in fib return fib(n - 1) + fib(n - 2) File "bla.py", line 4, in fib return fib(n - 1) + fib(n - 2) ... Not so much I think.
  - sabauma 10y ago

ProblemFactory 10y ago

> I did not propose a solution because there are many, as you note; there are, however, implementations with decent solutions like AFAIK Jython. There are no solutions that satisfy everyone that I am aware of yet. Guido has said in the past that he'd be happy to get rid of the GIL, and would merge a patch that solves it, as long as: * It does not reduce the performance of single-threaded Python code. * It stays compatible with all existing pure Python code and C extensions. But in practice, GIL is not that much of an issue for many types of applications where Python is popular. * It's not an issue for web apps, because these are typically served from multiple physical servers each running multiple python processes. These do not share a GIL anyway, and "thread safety" is pushed to database transactions. * It is not an issue for apps which spend most of the time doing I/O. Most IO libraries release the GIL, and other threads can run while you're waiting for results from the database. * It is not an issue for data science doing heavy number crunching with numpy and everything built on top of numpy. Numpy releases the GIL while doing large computations in C. * It is not an issue for small scripts, as a "better than Bash". The GIL is only an issue for apps that do heavy computation in pure Python code, and need parallelism within a single process (socket servers? text data processing?). As a result, many Python users just don't find it a big enough problem to be worth solving, if the solution comes with downsides for their use cases.

xapata 10y ago
The GIL is only a problem because there's "no free lunch" -- no single strategy that is best in all cases.
diek 10y ago
> It is not an issue for apps which spend most of the time doing I/O. This is a common misconception that doesn't seem to be backed up by any data. Dave Beazley did a number of performance tests with profiling, looking at GIL contention in a multi-core scenario: http://www.dabeaz.com/python/GIL.pdf http://www.dabeaz.com/python/GIL.pdf The results were that even IO-bound workloads still suffered because of the poor implementation of the GIL (details on slide 35 or so). This was an issue up until Python 3.2 (!) when a new GIL implementation was added, which he also profiled: http://www.dabeaz.com/python/NewGIL.pdf http://www.dabeaz.com/python/NewGIL.pdf

KirinDave 10y ago

Once, I was on mailing lists with GVR and other language contributors, and have seem him go off deeply into functional programming. Some of the stuff he wrote went right over my head. For someone who declares he hates functional programming even at the most basic levels of data stream manipulation, he knows it quite well. I've always been frustrated with this disconnect, even more acutely than you have, because I know GVR is being disingenuous when he says, "I don't get it." He absolutely does. He thinks other people won't.

takeda 10y ago

>>- the solutions to a problem (I'm thinking GIL) come with a lot of consequences which are not readily acceptable > I did not propose a solution because there are many, as you note; there are, however, implementations with decent solutions like AFAIK Jython. >>- solving some of the issues would exacerbate backwards compatibility. > Such as what? Let's use the GIL for example. The reason that is still present is not that it is hard to remove it. It already was removed in the past. The problem is that when GIL is replaced with smaller locks, the python becomes much slower, because of some features and behavior that people got used to. One could make python faster by changing behavior, but then it would break existing code and C extensions. There's no easy way to do this without sacrificing something else. Larry Hastings work on removing GIL and has interesting talk about it[1] [1] https://www.youtube.com/watch?v=fgWUwQVoLHo https://www.youtube.com/watch?v=fgWUwQVoLHo

ehsankia 10y ago
That talk was a great overview of the issue. Most people who are ignorant about the subject always assume the GIL is stupid and useless, but the GIL allows Python to be extremely fast in single threaded scenarios, and any attempt to remove it introduces at least 20% slow downs. And C extension support is also a huge factor as you mention. All of Python's scientific modules would be lost if they broke that. The author assumes these are simple problems that can be solved but aren't because of politics or incompetence, but some of the smartest mind have attempted and failed. I'd like the author to try and come up to solutions or at least draft ideas for how each of his points can be fixed. A lot of those are easy to state but difficult to solve without breaking more stuff.

pvg 10y ago

The standard library in general encourages use of higher-order functions and concepts borrowed primarily from FPLs (see: comprehensions, map/reduce, sort, etc.) I could not imagine seeing them backtracking on this -- it only helps them to go further in that direction. I don't think the standard library really 'encourages' this in a way that's different from most languages that support first class (rather than 'higher order') functions. If you consider python's evolution, its support for many common programming paradigms was somewhat haphazard and weak and developed over time, mostly pragmatically. OO has become stronger, the 80% use case of common functional idioms is covered by comprehensions, etc. Ill thought out features (e.g. terrible lambdas) have become de-emphasised. The choices are extensively document, even if not everyone cup of tea so it seems both glib and inaccurate to say (re: FP) 'it only helps them to go further in that direction'. How does it help them?