7 ms·

You are making excuses. It should require neither new hardware nor Herculean effort to realize acceptable multicore performance on CPU-bound tasks. And your bad

You are making excuses. It should require neither new hardware nor Herculean effort to realize acceptable multicore performance on CPU-bound tasks. And your badgering and down voting people here who disagree with you position doesn't make your argument any more persuasive.

sho_hn 14y ago
Just to come to your aid and make your last sentence not a baseless accusation: Yes, I've been downvoting you. However, it's not because I'm disagreeing with your position, it's because I consider the way you conduct yourself in the debate, well, crappy, because: - First and foremost, your frequent use of ridicule and hyperbole is escalating the tone of the debate. - You prefer to move goal posts instead of addressing arguments (e.g. you still haven't actually gotten around to explaining how the 3.x transition has supposedly set back PyPy by years). - You're not beneath skewing things to fit your narrative, even if it doesn't really make any sense (e.g. pointing out that PyPy's STM tinkering doesn't yet support Python 3.x yet - yeah, duh, given that both are WIP branches that aren't merged into PyPy proper yet, and nevermind that the STM work doesn't really impact the same area of the codebase that adding Python 3.x support does, so there is no cause to anticipate problems with eventually integrating both). - You make bold claims and leave them unsubstantiated (e.g. "It should require neither new hardware nor Herculean effort to realize acceptable multicore performance on CPU-bound tasks." just now - if you've got it all figured out, where's your proposal to eliminate the GIL?). - Those bold claims even miss their mark in the context of the debate (I pointed out that "it's slower" refers to single-threaded workloads; when it comes to multicore performance the STM proof of concept is already showing some benefits). And so on and so on. Simply put, your posts are low quality, that's why I'm hitting that downward arrow.
- dmbaggett 14y ago
  I still don't see that you are actually challenging the original claims, which were simply that splitting the language in two was a very bad idea, and that the split damaged the Python ecosystem. I stand by my claim that any reasonable general-purpose programming language should be able to run simple CPU-bound threaded tasks properly. I didn't claim arbitrary parallel programming should be easy; just that the GIL causes multi-threaded programs written in standard style in Python to not work properly in CPU-bound contexts, because the GIL prevents these programs from using all the cores available, and, furthermore, because even in single-core contexts it is prone to performance failure modes such as convoying and starvation that induce worse performance when there are more cores available. (cf. very extensive discussions in bugs.python.org on GIL problems, along with David Beazley's, Antoine Pitrou's, and others' attempts to address them). It is a standard, uncontroversial expectation of general purpose languages that offer threading that the threads run in parallel, using the available cores. I don't believe this needs substantiation any more than the claim that general purpose programming languages should run serial programs deterministically. I believe the better course for Python post 2.X would have been to fix the core problems with the language's reference implementation, while maintaining backward compatibility with the very large existing code base, rather than adding features of dubious value, doing nothing to address the GIL problem, and breaking compatibility with the vast majority of 2.X code. While this is certainly my opinion, and yours obviously differs, you can't badger away the limitations of the GIL, and you can't just browbeat people into accepting the 3.X series as a good thing. Look, I don't hate the language -- I use it every day and like it a lot. But don't pretend that there aren't real problems with Python when there clearly are -- the discussions on bugs.python.org are mainly driven by strong proponents of the language and are hardly politicized -- and don't pretend that Python 3 wasn't problematic; it's created years -- yes, years -- of problems and confusion with 2 vs 3 incompatibility, has expanded the footprint of the language and changed its semantics in substantive ways (cf. the many, many accepted PEPs post 2.X) that make implementing the language harder. Every language bump (whether 2.5 to 2.6 or 2.7 to 3.0) has meant additional work for the developers of alternative implementations; this is obvious on the face of it; you need only read these projects' blogs and look at the elapsed calendar time.
  - hynek 14y ago
    You seem to presume that if there weren’t Python 3, we’d be busy removing the GIL. That is not the case, removing the GIL has no priority for reasons already too often discussed.
    - dmbaggett 14y ago
      No, I neither believe nor tried to assert that. I said I believe that effort should have been invested in fixing the GIL and other core language problems rather than adding more stuff to the language, fixing a few warts (and, admittedly, one big problem, chr vs unicode vs byte). I also claimed that the additional features added to Python 3.X have meant that those who have already fixed the GIL problem (Jython/IronPython) and those who might still (PyPy) have been forced by the 3.X change to chase the taillights of CPython, making it essentially impossible for these implementations to replace CPython as the canonical version. (I do take for granted that the canonical implementation of Python should have multithreading primitives that can take advantage of all the cores in machine; that has been debated above already so I won't repeat it here.)
      - hynek 14y ago
        Ironically, Py3s purpose _is_ to fix Python’s warts. You just seem to be lucky enough to not have encountered them yet. :) Python 3 was about _losing_ baggage so we can start adding features again. The fact that Python 3.3 has a lot of more features than Python 2.7 is is due to the fact that 2.7 was the last release that got Python 3 features back ported (two years ago) and is in bug-fixing mode since then. (and people in these comments still complain about Python moving too fast) There also have been efforts to ease the pain for PyPy et al by requiring new modules to have also a Python based implementation and by – put simply – giving them money from PSF funds. Jython’s problem was rather lack of interest and developing funds as they were stuck at 2.5 forever until one of their developers got funding to bring it to 2.7. It’s not all as bleak as it seems to you and chances are that your porting effort will be negligible as soon as the dependencies are met. And the GIL…just leave it be. There are really no plans or efforts to remove it for many, many reasons I won’t reiterate here.
zurn 14y ago
Huh? It's a pretty widely accepted fact that correct & performant parallel programming is inherently a hard problem (barring invention of a silver bullet...)
- dmbaggett 14y ago
  Indeed, and I am not claiming otherwise. However, it is well understood (see bugs.python.org and search for "GIL") that the GIL causes performance problems in otherwise correct and performant Python code. For example, such programs can run more slowly if there are more cores available than if there is only a single core available. Various workarounds to the GIL implementation in CPython have been proposed to address these problems, and it's better in Python 3.X, but the problems still exist. And one still cannot write true multithreaded programs in Python using the language's standard thread library. To use all the cores, you have to resort to the processing module, which relies on processes rather than threads; this vastly increases the memory consumption, and makes the resulting programs less idiomatic as well. (You have to pass only pickle-able stuff to your child processes, for example.)
  - zurn 14y ago
    I was just taking issue at your more general claim about parallel programming. As for the GIL, it's such a dead horse. Threads are a very problematic parallel programming model and parallel threads would bring in an enormous amount of implementation complexity as well, if it even could be pulled off.

sho_hn 14y ago

Just to come to your aid and make your last sentence not a baseless accusation: Yes, I've been downvoting you. However, it's not because I'm disagreeing with your position, it's because I consider the way you conduct yourself in the debate, well, crappy, because: - First and foremost, your frequent use of ridicule and hyperbole is escalating the tone of the debate. - You prefer to move goal posts instead of addressing arguments (e.g. you still haven't actually gotten around to explaining how the 3.x transition has supposedly set back PyPy by years). - You're not beneath skewing things to fit your narrative, even if it doesn't really make any sense (e.g. pointing out that PyPy's STM tinkering doesn't yet support Python 3.x yet - yeah, duh, given that both are WIP branches that aren't merged into PyPy proper yet, and nevermind that the STM work doesn't really impact the same area of the codebase that adding Python 3.x support does, so there is no cause to anticipate problems with eventually integrating both). - You make bold claims and leave them unsubstantiated (e.g. "It should require neither new hardware nor Herculean effort to realize acceptable multicore performance on CPU-bound tasks." just now - if you've got it all figured out, where's your proposal to eliminate the GIL?). - Those bold claims even miss their mark in the context of the debate (I pointed out that "it's slower" refers to single-threaded workloads; when it comes to multicore performance the STM proof of concept is already showing some benefits). And so on and so on. Simply put, your posts are low quality, that's why I'm hitting that downward arrow.

dmbaggett 14y ago
I still don't see that you are actually challenging the original claims, which were simply that splitting the language in two was a very bad idea, and that the split damaged the Python ecosystem. I stand by my claim that any reasonable general-purpose programming language should be able to run simple CPU-bound threaded tasks properly. I didn't claim arbitrary parallel programming should be easy; just that the GIL causes multi-threaded programs written in standard style in Python to not work properly in CPU-bound contexts, because the GIL prevents these programs from using all the cores available, and, furthermore, because even in single-core contexts it is prone to performance failure modes such as convoying and starvation that induce worse performance when there are more cores available. (cf. very extensive discussions in bugs.python.org on GIL problems, along with David Beazley's, Antoine Pitrou's, and others' attempts to address them). It is a standard, uncontroversial expectation of general purpose languages that offer threading that the threads run in parallel, using the available cores. I don't believe this needs substantiation any more than the claim that general purpose programming languages should run serial programs deterministically. I believe the better course for Python post 2.X would have been to fix the core problems with the language's reference implementation, while maintaining backward compatibility with the very large existing code base, rather than adding features of dubious value, doing nothing to address the GIL problem, and breaking compatibility with the vast majority of 2.X code. While this is certainly my opinion, and yours obviously differs, you can't badger away the limitations of the GIL, and you can't just browbeat people into accepting the 3.X series as a good thing. Look, I don't hate the language -- I use it every day and like it a lot. But don't pretend that there aren't real problems with Python when there clearly are -- the discussions on bugs.python.org are mainly driven by strong proponents of the language and are hardly politicized -- and don't pretend that Python 3 wasn't problematic; it's created years -- yes, years -- of problems and confusion with 2 vs 3 incompatibility, has expanded the footprint of the language and changed its semantics in substantive ways (cf. the many, many accepted PEPs post 2.X) that make implementing the language harder. Every language bump (whether 2.5 to 2.6 or 2.7 to 3.0) has meant additional work for the developers of alternative implementations; this is obvious on the face of it; you need only read these projects' blogs and look at the elapsed calendar time.
- hynek 14y ago
  You seem to presume that if there weren’t Python 3, we’d be busy removing the GIL. That is not the case, removing the GIL has no priority for reasons already too often discussed.
  - dmbaggett 14y ago
    No, I neither believe nor tried to assert that. I said I believe that effort should have been invested in fixing the GIL and other core language problems rather than adding more stuff to the language, fixing a few warts (and, admittedly, one big problem, chr vs unicode vs byte). I also claimed that the additional features added to Python 3.X have meant that those who have already fixed the GIL problem (Jython/IronPython) and those who might still (PyPy) have been forced by the 3.X change to chase the taillights of CPython, making it essentially impossible for these implementations to replace CPython as the canonical version. (I do take for granted that the canonical implementation of Python should have multithreading primitives that can take advantage of all the cores in machine; that has been debated above already so I won't repeat it here.)
    - hynek 14y ago
      Ironically, Py3s purpose _is_ to fix Python’s warts. You just seem to be lucky enough to not have encountered them yet. :) Python 3 was about _losing_ baggage so we can start adding features again. The fact that Python 3.3 has a lot of more features than Python 2.7 is is due to the fact that 2.7 was the last release that got Python 3 features back ported (two years ago) and is in bug-fixing mode since then. (and people in these comments still complain about Python moving too fast) There also have been efforts to ease the pain for PyPy et al by requiring new modules to have also a Python based implementation and by – put simply – giving them money from PSF funds. Jython’s problem was rather lack of interest and developing funds as they were stuck at 2.5 forever until one of their developers got funding to bring it to 2.7. It’s not all as bleak as it seems to you and chances are that your porting effort will be negligible as soon as the dependencies are met. And the GIL…just leave it be. There are really no plans or efforts to remove it for many, many reasons I won’t reiterate here.

zurn 14y ago

Huh? It's a pretty widely accepted fact that correct & performant parallel programming is inherently a hard problem (barring invention of a silver bullet...)

dmbaggett 14y ago
Indeed, and I am not claiming otherwise. However, it is well understood (see bugs.python.org and search for "GIL") that the GIL causes performance problems in otherwise correct and performant Python code. For example, such programs can run more slowly if there are more cores available than if there is only a single core available. Various workarounds to the GIL implementation in CPython have been proposed to address these problems, and it's better in Python 3.X, but the problems still exist. And one still cannot write true multithreaded programs in Python using the language's standard thread library. To use all the cores, you have to resort to the processing module, which relies on processes rather than threads; this vastly increases the memory consumption, and makes the resulting programs less idiomatic as well. (You have to pass only pickle-able stuff to your child processes, for example.)
- zurn 14y ago
  I was just taking issue at your more general claim about parallel programming. As for the GIL, it's such a dead horse. Threads are a very problematic parallel programming model and parallel threads would bring in an enormous amount of implementation complexity as well, if it even could be pulled off.