6 ms·

Progress on the Gilectomy

86 points by etrevino 9y ago

deleted 9y ago
[deleted]
deleted 9y ago
[deleted]
deleted 9y ago
[deleted]
krallja 9y ago
Thanks for posting this summary. The video of the talk was posted in May, but I never got around to watching it, because video always feels like it needs a much larger block of uninterrupted time than reading.
Animats 9y ago
Multithreading in Python is all uphill. In Python, any thread can store into anything at any time. By name, even, using "setattr". There's no thread-local data. This implies excessive locking activity. That's why this guy is working so hard to do things that ought to be simple. A few restrictions on the language would make it easier, but when this is suggested, Guido has a cow.
- nine_k 9y ago
  Is there a link explaining the suggestions succinctly?
  - Animats 9y ago
    - Multiple interpreters with mostly non-shared data in the same process.[1]. - VecPy, for number-crunching.[2] - Newthreading, which I wrote in 2010.[3] [1] https://lwn.net/Articles/650489/ https://lwn.net/Articles/650489/ [2] https://www.andrew.cmu.edu/user/dfarrow/proposal.html https://www.andrew.cmu.edu/user/dfarrow/proposal.html [3] http://www.animats.com/papers/languages/newthreadingintro.html http://www.animats.com/papers/languages/newthreadingintro.ht...
    - simcop2387 9y ago
      To be fair at least towards #1, that's a really difficult one to get right. Perl has had that kind of threading for a long time and it's almost always considered a bad decision to try to use them. I'm sure it could be done better, but perl did it via a complete copy on write setup in order to create the mostly non-shared data. But you end up with a lot of locking and other contention for the shared data because there's no good way to expose the fact that it's shared and changing it requires such restrictions. Instead the method usually used is to take advantage of most OS's built in COW when forking a new process and communicating via sockets, or some other method instead that forces you to deal with the limitations (copying data, or if you used shared memory you have to do explicit sharing and locking/serialization yourself).
- benhoyt 9y ago
  "Guido has a cow". Hmmm, or maybe he knows this would be mean huge, backwards-incompatible changes to the language's dynamic nature, and especially after the experience with the changes (I think much lesser ones compared to this) from Python 2 to Python 3, he's very wary of that. Still, if you were going to propose "a few restrictions on the language" to make it less dynamic and easier to do multi-threading / optimization, what would they look like?
- VectorLock 9y ago
  From seeing the "Status of the Gilectomy" talk at PyCon it sounded more like the big blocker is not breaking the C API too bad. Now if somebody suggested a Python 4 with a totally backwards incompatible but thread-safe C API now that I could see him having a cow over (even though I like that idea)
- omginternets 9y ago
  >when this is suggested, Guido has a cow. Like what? My first reaction is to think "and he's [probably] right!" Python's added value is it's very rich and flexible object model. Imposing restrictions on the object model is robbing Peter to pay Paul in an environment where Peter is more important, albeit for historical reasons.
deleted 9y ago
[deleted]
stickfigure 9y ago
I don't really understand the problem with the GIL. Sure, people doing exotic high performance stuff probably hate it, but Python probably isn't their go-to language anyways. For boring old web development and business processing, the GIL seems to be a very efficient way of utilizing a CPU+caches without resorting to the horrible event-driven programming model of Node. Want twice the throughput? Run twice the processes. I kinda wish Java would bring back the option of green threads, it was a perfect programming model for web development.
- saghul 9y ago
  The problem is with CPU intensive tasks. Being as expressive as Python is, it would be nice if CPU intensive algorithms could be written in it, without blocking all other Python threads while the algorithm runs. For example.
  - crdoconnor 9y ago
    >it would be nice if CPU intensive algorithms could be written in it It would be nicer if instead of trying to turn python into something it is not (a language for writing CPU intensive tasks), it was made better at what it is (glue code). It would be especially nice if there were tighter, more seamless integration between python and rust so that hot paths in python can easily be migrated to rust.
    - mjburgess 9y ago
      No to threading, but yes to tighter integration with rust. Ah yes, let's not focus on something python is not. Let's appeal to the majority use case. What minor planet are people here from?
    - Walkman 9y ago
      Python is far more than glue code.
    - sevensor 9y ago
      I'm not a Rust guy, so take this with a grain of salt, but I was under the impression that Rust did a great job with C interoperability, which is all you need for writing Python extensions. Some hasty googling brings up https://github.com/PyO3/setuptools-rust https://github.com/PyO3/setuptools-rust, which appears to be exactly the thing you're asking for. In other words, it's not heavily advertised, but it appears to be entirely feasible to build Python extensions in Rust, and not spectacularly complicated either.
akx 9y ago
Previous thread (5 days ago): https://news.ycombinator.com/item?id=14432854 https://news.ycombinator.com/item?id=14432854
absoluute 9y ago
Getting rid of the GIL is easy - just move everything into the interpreter instance struct and add an interpreter pointer as the first parameter in every C API call except create. This should have been done in Python 3 since they broke the API anyway. Now everyone will have to go through another painful transition when they come to their senses. The current thread local storage hacks and PyThreadState/PyEval_SaveThread/PyEval_RestoreThread/PyThreadState_Swap are ridiculous.

86 points by etrevino 9y ago

deleted 9y ago

[deleted]

deleted 9y ago

[deleted]

deleted 9y ago

[deleted]

krallja 9y ago

Thanks for posting this summary. The video of the talk was posted in May, but I never got around to watching it, because video always feels like it needs a much larger block of uninterrupted time than reading.

Animats 9y ago

Multithreading in Python is all uphill. In Python, any thread can store into anything at any time. By name, even, using "setattr". There's no thread-local data. This implies excessive locking activity. That's why this guy is working so hard to do things that ought to be simple. A few restrictions on the language would make it easier, but when this is suggested, Guido has a cow.

nine_k 9y ago
Is there a link explaining the suggestions succinctly?
- Animats 9y ago
  - Multiple interpreters with mostly non-shared data in the same process.[1]. - VecPy, for number-crunching.[2] - Newthreading, which I wrote in 2010.[3] [1] https://lwn.net/Articles/650489/ https://lwn.net/Articles/650489/ [2] https://www.andrew.cmu.edu/user/dfarrow/proposal.html https://www.andrew.cmu.edu/user/dfarrow/proposal.html [3] http://www.animats.com/papers/languages/newthreadingintro.html http://www.animats.com/papers/languages/newthreadingintro.ht...
  - simcop2387 9y ago
    To be fair at least towards #1, that's a really difficult one to get right. Perl has had that kind of threading for a long time and it's almost always considered a bad decision to try to use them. I'm sure it could be done better, but perl did it via a complete copy on write setup in order to create the mostly non-shared data. But you end up with a lot of locking and other contention for the shared data because there's no good way to expose the fact that it's shared and changing it requires such restrictions. Instead the method usually used is to take advantage of most OS's built in COW when forking a new process and communicating via sockets, or some other method instead that forces you to deal with the limitations (copying data, or if you used shared memory you have to do explicit sharing and locking/serialization yourself).
benhoyt 9y ago
"Guido has a cow". Hmmm, or maybe he knows this would be mean huge, backwards-incompatible changes to the language's dynamic nature, and especially after the experience with the changes (I think much lesser ones compared to this) from Python 2 to Python 3, he's very wary of that. Still, if you were going to propose "a few restrictions on the language" to make it less dynamic and easier to do multi-threading / optimization, what would they look like?
VectorLock 9y ago
From seeing the "Status of the Gilectomy" talk at PyCon it sounded more like the big blocker is not breaking the C API too bad. Now if somebody suggested a Python 4 with a totally backwards incompatible but thread-safe C API now that I could see him having a cow over (even though I like that idea)
omginternets 9y ago
>when this is suggested, Guido has a cow. Like what? My first reaction is to think "and he's [probably] right!" Python's added value is it's very rich and flexible object model. Imposing restrictions on the object model is robbing Peter to pay Paul in an environment where Peter is more important, albeit for historical reasons.

deleted 9y ago

[deleted]

stickfigure 9y ago

I don't really understand the problem with the GIL. Sure, people doing exotic high performance stuff probably hate it, but Python probably isn't their go-to language anyways. For boring old web development and business processing, the GIL seems to be a very efficient way of utilizing a CPU+caches without resorting to the horrible event-driven programming model of Node. Want twice the throughput? Run twice the processes. I kinda wish Java would bring back the option of green threads, it was a perfect programming model for web development.

saghul 9y ago
The problem is with CPU intensive tasks. Being as expressive as Python is, it would be nice if CPU intensive algorithms could be written in it, without blocking all other Python threads while the algorithm runs. For example.
- crdoconnor 9y ago
  >it would be nice if CPU intensive algorithms could be written in it It would be nicer if instead of trying to turn python into something it is not (a language for writing CPU intensive tasks), it was made better at what it is (glue code). It would be especially nice if there were tighter, more seamless integration between python and rust so that hot paths in python can easily be migrated to rust.
  - mjburgess 9y ago
    No to threading, but yes to tighter integration with rust. Ah yes, let's not focus on something python is not. Let's appeal to the majority use case. What minor planet are people here from?
  - Walkman 9y ago
    Python is far more than glue code.
  - sevensor 9y ago
    I'm not a Rust guy, so take this with a grain of salt, but I was under the impression that Rust did a great job with C interoperability, which is all you need for writing Python extensions. Some hasty googling brings up https://github.com/PyO3/setuptools-rust https://github.com/PyO3/setuptools-rust, which appears to be exactly the thing you're asking for. In other words, it's not heavily advertised, but it appears to be entirely feasible to build Python extensions in Rust, and not spectacularly complicated either.

akx 9y ago

Previous thread (5 days ago): https://news.ycombinator.com/item?id=14432854 https://news.ycombinator.com/item?id=14432854

absoluute 9y ago

Getting rid of the GIL is easy - just move everything into the interpreter instance struct and add an interpreter pointer as the first parameter in every C API call except create. This should have been done in Python 3 since they broke the API anyway. Now everyone will have to go through another painful transition when they come to their senses. The current thread local storage hacks and PyThreadState/PyEval_SaveThread/PyEval_RestoreThread/PyThreadState_Swap are ridiculous.