3 ms·

You seem to presume that if there weren’t Python 3, we’d be busy removing the GIL. That is not the case, removing the GIL has no priority for reasons already to

by hynek 14y ago

You seem to presume that if there weren’t Python 3, we’d be busy removing the GIL. That is not the case, removing the GIL has no priority for reasons already too often discussed.

dmbaggett 14y ago
No, I neither believe nor tried to assert that. I said I believe that effort should have been invested in fixing the GIL and other core language problems rather than adding more stuff to the language, fixing a few warts (and, admittedly, one big problem, chr vs unicode vs byte). I also claimed that the additional features added to Python 3.X have meant that those who have already fixed the GIL problem (Jython/IronPython) and those who might still (PyPy) have been forced by the 3.X change to chase the taillights of CPython, making it essentially impossible for these implementations to replace CPython as the canonical version. (I do take for granted that the canonical implementation of Python should have multithreading primitives that can take advantage of all the cores in machine; that has been debated above already so I won't repeat it here.)
- hynek 14y ago
  Ironically, Py3s purpose _is_ to fix Python’s warts. You just seem to be lucky enough to not have encountered them yet. :) Python 3 was about _losing_ baggage so we can start adding features again. The fact that Python 3.3 has a lot of more features than Python 2.7 is is due to the fact that 2.7 was the last release that got Python 3 features back ported (two years ago) and is in bug-fixing mode since then. (and people in these comments still complain about Python moving too fast) There also have been efforts to ease the pain for PyPy et al by requiring new modules to have also a Python based implementation and by – put simply – giving them money from PSF funds. Jython’s problem was rather lack of interest and developing funds as they were stuck at 2.5 forever until one of their developers got funding to bring it to 2.7. It’s not all as bleak as it seems to you and chances are that your porting effort will be negligible as soon as the dependencies are met. And the GIL…just leave it be. There are really no plans or efforts to remove it for many, many reasons I won’t reiterate here.
  - dmbaggett 14y ago
    OK, it's now off-topic, but nobody is probably reading this thread anymore anyway and I can't resist posting some sort of epitaph here. You're saying it's not bleak, but then immediately say the GIL is a permanent fixture of the language. The GIL not being fixed is in fact what makes it so bleak (to me, at least). Python isn't just for scripts anymore; we have 150k lines of Python code doing nontrivial stuff and relying on many third party libraries -- ultimately probably a million lines of code or so if you trace the dependencies all the way down. This is in most ways really great: the Python code is very clear and concise, and comes along with all the documentation and unit testing conventions established by the community over the years. All those high-quality third party libraries mean we don't even have to write lots of code in the first place. It's easily a 10x productivity improvement over C/C++ for us. But to make this code base take proper advantage of multiple cores, we must run under Jython or IronPython. (And at least Jython has its own problems; start up times are long because the Java class loader is slow.) We can't move to 3.X even if we wanted to, because no GIL-free implementation supports 3.X. We can't rely on the processing module, because we can't afford the overhead of an entire process of memory usage per "thread" at scale. It just sounds like the Python devs have decided that fixing the GIL is too hard, so people should just live without using multiple cores. Do they view our use case as exotic or somehow unsupported? Do they not see that it's important for a general purpose programming to support real threading? I'm not trying to be obnoxious; it's a serious question. We are likely to ultimately abandon the language because of this core issue, which is tragic given the other great things we get from using it, and the lack of a genuinely equivalent alternative. "So what, no one cares about you -- you are nobody," you may say. For the sake of argument, I will stipulate that. But I don't imagine our constraints are unique, and this bodes ill for the Python ecosystem. Our abandoning it is one thing; increasingly many developers abandoning it as multicore use becomes ever more important is another. Cores are getting a little bit faster with each tick and tock at Intel. But the rate at which we're getting additional cores looks to far outpace the per-core improvements for quite some time. This makes real threading increasingly important as time goes on. I know STM is the solution du jour, and it's an admirable piece of work. If that solves the problem in the long run, great, but it's far from obvious that it's going to win without dedicated hardware support, and in any case isn't part of the reference implementation of the language (which seems to always be ahead of the other implementations). I know you're saying it's not a big deal, don't worry about it, etc. but it just doesn't seem that way to me. :(

by hynek 14y ago

You seem to presume that if there weren’t Python 3, we’d be busy removing the GIL. That is not the case, removing the GIL has no priority for reasons already too often discussed.

dmbaggett 14y ago

No, I neither believe nor tried to assert that. I said I believe that effort should have been invested in fixing the GIL and other core language problems rather than adding more stuff to the language, fixing a few warts (and, admittedly, one big problem, chr vs unicode vs byte). I also claimed that the additional features added to Python 3.X have meant that those who have already fixed the GIL problem (Jython/IronPython) and those who might still (PyPy) have been forced by the 3.X change to chase the taillights of CPython, making it essentially impossible for these implementations to replace CPython as the canonical version. (I do take for granted that the canonical implementation of Python should have multithreading primitives that can take advantage of all the cores in machine; that has been debated above already so I won't repeat it here.)

hynek 14y ago
Ironically, Py3s purpose _is_ to fix Python’s warts. You just seem to be lucky enough to not have encountered them yet. :) Python 3 was about _losing_ baggage so we can start adding features again. The fact that Python 3.3 has a lot of more features than Python 2.7 is is due to the fact that 2.7 was the last release that got Python 3 features back ported (two years ago) and is in bug-fixing mode since then. (and people in these comments still complain about Python moving too fast) There also have been efforts to ease the pain for PyPy et al by requiring new modules to have also a Python based implementation and by – put simply – giving them money from PSF funds. Jython’s problem was rather lack of interest and developing funds as they were stuck at 2.5 forever until one of their developers got funding to bring it to 2.7. It’s not all as bleak as it seems to you and chances are that your porting effort will be negligible as soon as the dependencies are met. And the GIL…just leave it be. There are really no plans or efforts to remove it for many, many reasons I won’t reiterate here.
- dmbaggett 14y ago
  OK, it's now off-topic, but nobody is probably reading this thread anymore anyway and I can't resist posting some sort of epitaph here. You're saying it's not bleak, but then immediately say the GIL is a permanent fixture of the language. The GIL not being fixed is in fact what makes it so bleak (to me, at least). Python isn't just for scripts anymore; we have 150k lines of Python code doing nontrivial stuff and relying on many third party libraries -- ultimately probably a million lines of code or so if you trace the dependencies all the way down. This is in most ways really great: the Python code is very clear and concise, and comes along with all the documentation and unit testing conventions established by the community over the years. All those high-quality third party libraries mean we don't even have to write lots of code in the first place. It's easily a 10x productivity improvement over C/C++ for us. But to make this code base take proper advantage of multiple cores, we must run under Jython or IronPython. (And at least Jython has its own problems; start up times are long because the Java class loader is slow.) We can't move to 3.X even if we wanted to, because no GIL-free implementation supports 3.X. We can't rely on the processing module, because we can't afford the overhead of an entire process of memory usage per "thread" at scale. It just sounds like the Python devs have decided that fixing the GIL is too hard, so people should just live without using multiple cores. Do they view our use case as exotic or somehow unsupported? Do they not see that it's important for a general purpose programming to support real threading? I'm not trying to be obnoxious; it's a serious question. We are likely to ultimately abandon the language because of this core issue, which is tragic given the other great things we get from using it, and the lack of a genuinely equivalent alternative. "So what, no one cares about you -- you are nobody," you may say. For the sake of argument, I will stipulate that. But I don't imagine our constraints are unique, and this bodes ill for the Python ecosystem. Our abandoning it is one thing; increasingly many developers abandoning it as multicore use becomes ever more important is another. Cores are getting a little bit faster with each tick and tock at Intel. But the rate at which we're getting additional cores looks to far outpace the per-core improvements for quite some time. This makes real threading increasingly important as time goes on. I know STM is the solution du jour, and it's an admirable piece of work. If that solves the problem in the long run, great, but it's far from obvious that it's going to win without dedicated hardware support, and in any case isn't part of the reference implementation of the language (which seems to always be ahead of the other implementations). I know you're saying it's not a big deal, don't worry about it, etc. but it just doesn't seem that way to me. :(