3 ms·

Disgusting. Could you at least make an effort to convince the developer that they personally would benefit from the massive amount of work you are demanding of

Disgusting. Could you at least make an effort to convince the developer that they personally would benefit from the massive amount of work you are demanding of them, instead of just trying to bully them into it? Come on, it's 2010. Multi-core is here to stay. It's disgusting that a general-purpose language like Python has such a horrible default properties on multi-core.

shadowsun7 16y ago
nkurz isn't arguing about multi-core (and various benefits thereof). He's arguing about convincing the developer that there are benefits of adding such support, instead of 'just complaining'. On a personal note, I must say here that I've never considered the differences between the two, and I'm taking this to heart.
rbanffy 16y ago
Care to explain what real-world problems you had with the GIL? I am usually the first guy to say the GIL must go, but, quite frankly, I have learned to make Zope+Plone and Django stacks scale without messing with it. If you have a 32-core server and you want to serve, say, a Zope site, just use ZEO (ZEORaid, perhaps) and 31+ ZEO clients behind your Varnish cache. The GIL must go, but the fact the very clever people who have tried to make threading nicer have tried to get rid of it and failed indicates this is not an easy problem. It's not about getting rid of the GIL. It's about getting rid of it without breaking compatibility with the mountain of modules written in C there are - and that experience little or no GIL problems by themselves. And, mind you, we are talking about CPython, the canonical implementation. IIRC, neither Jython nor PyPy have the same kind of multi-core performance problem. I am not sure, but if you write your performance-critical module in RPython and then compile it into C, you won't have many GIL problems either. While the GIL is the biggest problem I can see in CPython, for most usage scenarios, it's not a big one.
- fauigerzigerk 16y ago
  Yes, the question is whether Python intends to support the growing number of usage scenarios for which the GIL is a problem. You could write something like Cassandra or Lucene using a combination of Python and C, but the GIL prevents that.
  - rbanffy 16y ago
    You could write something like Cassandra or Lucene in Python. Python is a language (much like Java, BTW). You could then run it under Jython or PyPy. The GIL is unique do CPython, AFAIK.
    - fauigerzigerk 16y ago
      PyPy and Jython lag far behind the current version of Python, which is defined by the implementation of CPython. Also, as I said, I would like to use a combination of Python and C. That's for a reason. The JVM does not have structured value types (structs in C). All structured types in Java have to be referenced, which is the reason why Java software eats so much more memory than software implemented in C, C++, C# or Go. Maybe PyPy can work some day, if they can catch up. That day is definately not today.
      - rbanffy 16y ago
        PyPy and Jython may lag behind, but that's not a huge problem. I would love to use dict comprehensions and {} sets like I can in CPython 2.7, but I know I can't always deploy against the latest version. And, more than that, I know I can work around those features. The code may look uglier, but it's usually as fast as it would be with the newest features. And you are right - CPython is the canonical implementation of the language. The fact we can use a reference implementation in production environments is great. You probably wouldn't be able to write something like Cassandra in Python 2.7 and expect it to run unmodified under Jython or IronPython, but you could write it in Python 2.5 and run it under both 2.7 CPython (with the expected multi-core annoyances) or under current Jython. And you can't have painless C modules in Java either.
    - j_baker 16y ago
      Actually, I believe PyPy has a GIL as well (unless they've removed it recently).
      - rbanffy 16y ago
        Oops... May have gotten that one wrong, but the point stands. Even if Jython is the only GIL-free implementation of Python, you can run your thread-heavy code on it. There is no such thing as "You wouldn't be able to do [cool-multi-core-product] in Python because of the GIL"
- JabavuAdams 16y ago
  At one point, I started writing a 3D game engine in Python, with the understanding that I'd profile and if necessary re-write the necessary parts. I've also used Panda3D, which has a lot of Python-wrapped C++ code. Now, I don't have specific benchmarks to point to, but it's important not "prematurely pessimize", just as it's important not to optimize prematurely. In high-performance 3D games, you'd switch compilers for say a 10% gain in optimized code performance. Ever since I've become aware of how deep the GIL problems go, I can't really justify starting to write an engine in Python, if I'm just going to have to re-write critical and error-prone parts in C++ anyway. There's a more general principle here that applies to languages and runtimes, as well as library design. You want to be able to get started with a new component or language / runtime easily, trading ease of use for performance, BUT as the project progresses you want a way to be sure that you could beat-down or recover from any problems in a way that doesn't essentially require a re-write. These GIL issues remove that confidence in Python, for me. It's not a win if I can only write the easy parts of my code in Python. I specifically want to use Python to write the hard parts of my code. I agree that there doesn't seem to be an easy, or even moderately difficult solution to this problem. That makes it even worse from the point of view of advocating Python use on a project. To summarize, it's reasonable to say "don't optimize until you know there's a problem", but you can come at it the other way from a risk-management perspective. "What is the risk that there's some dead-end due to this language / runtime that we just won't be able to overcome (because the smartest dudes who know the language inside-out, can't fix it either)."
  - JabavuAdams 16y ago
    It's sad, because I'm a huge fan of higher-level languages, but even dealing with other people's broken or retarded bindings to C or C++ libraries is a huge time sink. At a certain point it just becomes more productive to write in C++, never mind the FUD about programmer productivity. That said, for my current project I'm using Unity, and writing script code in Javascript and C#, but this is more about having a toolset that Just Works out of the box. It's not clear that I would choose that approach if I had to write it from scratch.

shadowsun7 16y ago

nkurz isn't arguing about multi-core (and various benefits thereof). He's arguing about convincing the developer that there are benefits of adding such support, instead of 'just complaining'. On a personal note, I must say here that I've never considered the differences between the two, and I'm taking this to heart.

rbanffy 16y ago

Care to explain what real-world problems you had with the GIL? I am usually the first guy to say the GIL must go, but, quite frankly, I have learned to make Zope+Plone and Django stacks scale without messing with it. If you have a 32-core server and you want to serve, say, a Zope site, just use ZEO (ZEORaid, perhaps) and 31+ ZEO clients behind your Varnish cache. The GIL must go, but the fact the very clever people who have tried to make threading nicer have tried to get rid of it and failed indicates this is not an easy problem. It's not about getting rid of the GIL. It's about getting rid of it without breaking compatibility with the mountain of modules written in C there are - and that experience little or no GIL problems by themselves. And, mind you, we are talking about CPython, the canonical implementation. IIRC, neither Jython nor PyPy have the same kind of multi-core performance problem. I am not sure, but if you write your performance-critical module in RPython and then compile it into C, you won't have many GIL problems either. While the GIL is the biggest problem I can see in CPython, for most usage scenarios, it's not a big one.

fauigerzigerk 16y ago
Yes, the question is whether Python intends to support the growing number of usage scenarios for which the GIL is a problem. You could write something like Cassandra or Lucene using a combination of Python and C, but the GIL prevents that.
- rbanffy 16y ago
  You could write something like Cassandra or Lucene in Python. Python is a language (much like Java, BTW). You could then run it under Jython or PyPy. The GIL is unique do CPython, AFAIK.
  - fauigerzigerk 16y ago
    PyPy and Jython lag far behind the current version of Python, which is defined by the implementation of CPython. Also, as I said, I would like to use a combination of Python and C. That's for a reason. The JVM does not have structured value types (structs in C). All structured types in Java have to be referenced, which is the reason why Java software eats so much more memory than software implemented in C, C++, C# or Go. Maybe PyPy can work some day, if they can catch up. That day is definately not today.
    - rbanffy 16y ago
      PyPy and Jython may lag behind, but that's not a huge problem. I would love to use dict comprehensions and {} sets like I can in CPython 2.7, but I know I can't always deploy against the latest version. And, more than that, I know I can work around those features. The code may look uglier, but it's usually as fast as it would be with the newest features. And you are right - CPython is the canonical implementation of the language. The fact we can use a reference implementation in production environments is great. You probably wouldn't be able to write something like Cassandra in Python 2.7 and expect it to run unmodified under Jython or IronPython, but you could write it in Python 2.5 and run it under both 2.7 CPython (with the expected multi-core annoyances) or under current Jython. And you can't have painless C modules in Java either.
  - j_baker 16y ago
    Actually, I believe PyPy has a GIL as well (unless they've removed it recently).
    - rbanffy 16y ago
      Oops... May have gotten that one wrong, but the point stands. Even if Jython is the only GIL-free implementation of Python, you can run your thread-heavy code on it. There is no such thing as "You wouldn't be able to do [cool-multi-core-product] in Python because of the GIL"
JabavuAdams 16y ago
At one point, I started writing a 3D game engine in Python, with the understanding that I'd profile and if necessary re-write the necessary parts. I've also used Panda3D, which has a lot of Python-wrapped C++ code. Now, I don't have specific benchmarks to point to, but it's important not "prematurely pessimize", just as it's important not to optimize prematurely. In high-performance 3D games, you'd switch compilers for say a 10% gain in optimized code performance. Ever since I've become aware of how deep the GIL problems go, I can't really justify starting to write an engine in Python, if I'm just going to have to re-write critical and error-prone parts in C++ anyway. There's a more general principle here that applies to languages and runtimes, as well as library design. You want to be able to get started with a new component or language / runtime easily, trading ease of use for performance, BUT as the project progresses you want a way to be sure that you could beat-down or recover from any problems in a way that doesn't essentially require a re-write. These GIL issues remove that confidence in Python, for me. It's not a win if I can only write the easy parts of my code in Python. I specifically want to use Python to write the hard parts of my code. I agree that there doesn't seem to be an easy, or even moderately difficult solution to this problem. That makes it even worse from the point of view of advocating Python use on a project. To summarize, it's reasonable to say "don't optimize until you know there's a problem", but you can come at it the other way from a risk-management perspective. "What is the risk that there's some dead-end due to this language / runtime that we just won't be able to overcome (because the smartest dudes who know the language inside-out, can't fix it either)."
- JabavuAdams 16y ago
  It's sad, because I'm a huge fan of higher-level languages, but even dealing with other people's broken or retarded bindings to C or C++ libraries is a huge time sink. At a certain point it just becomes more productive to write in C++, never mind the FUD about programmer productivity. That said, for my current project I'm using Unity, and writing script code in Javascript and C#, but this is more about having a toolset that Just Works out of the box. It's not clear that I would choose that approach if I had to write it from scratch.