15 ms·

Subinterpreters for Python

157 points by lukastyrychtr 6y ago

bsder 6y ago
Oh how I wish Python had just sucked it up at 3.0 and eaten the performance hit for removing the GIL. By now, everybody would have optimized it back to normal (or better!).
- loeg 6y ago
  There was no possible way to do that given the things python lets you get away with "atomically". Requiring explicit synchronization to avoid data races would have been an even bigger language break than the other breaking changes in 3. Python is not a language for people who want to do fine-grained explicit concurrency. And doing it implicitly would require stupidly fine-grained locks on all objects, destroying any performance gains; there's no way to "optimize" that to GIL performance.
  - Doxin 6y ago
    There's still plenty of programs where you'd want to spawn a thread that basically doesn't touch any of the data of the parent thread. It'd be nice if there was a way around the GIL for those cases. As an example take the multiprocessing.pool.ThreadPool.map function. Most of the use-cases of that function only read from the parent threads memory once to pass the function arguments. After that the thread may very well spend a lot of time only reading memory it reserved itself. It's rather wasteful to have that thread wait on the GIL. Of course I haven't got an obvious solution either but it seems to me that for at least a subset of the uses of threading you could work around the GIL without breaking python too badly. Perhaps split the GIL into a per-thread Thread-interpreter-lock. That way each object can simply be annotated with which lock it belongs to. That way you still get proper atomicity like you're used to in python, but a thread might also actually run concurrently a lot of the time if it only touches objects it created itself.
    - loeg 6y ago
      > There's still plenty of programs where you'd want to spawn a thread that basically doesn't touch any of the data of the parent thread. It'd be nice if there was a way around the GIL for those cases. Subinterpreters gets you that (eventually; not in v1 apparently). > Perhaps split the GIL into a per-thread Thread-interpreter-lock. That way each object can simply be annotated with which lock it belongs to. That way you still get proper atomicity like you're used to in python, but a thread might also actually run concurrently a lot of the time if it only touches objects it created itself. I don't think this scheme would work especially well (or not better than subinterpreters). Message passing still requires copying, or some way to quiesce other threads and recursively change lock ownership of an object graph. To avoid deadlocks, at least one thread will need to drop its own lock to acquire the thread lock of objects owned by a 2nd thread. I'm not sure you could do that and preserve the legacy global atomicity behavior, and it would by definition be slower and bloatier on any individual thread than the existing GIL behavior.
  - jashmatthews 6y ago
    Does the CPython GIL give you a stronger effective guarantee than the one in CRuby? Because the one in CRuby will absolutely not save you from data races.
- shockinglytrue 6y ago
  There really was no hope of this without ditching reference counting, and consequently a much more destablizing change to the C API which was all but entirely unharmed during the 3.x transition. There have been several attempts to bring new science to bear on solving the refcounting problem, I don't think any of them ever got far enough to be taken seriously A significant chunk of Python performance is derived from optimizations made possible by the GIL - for example, so long as it is held, no additional locks are necessary to allocate a small object, unless that entails asking the system allocator for more memory
  - __d 6y ago
    The big problem with the 2->3 transition was that the cost exceeded the perceived benefit. Removing the GIL would have increased the cost, but also the benefit. I think the core team over-estimated the benefits of the change, hence people's reluctance to pay the cost. I suspect that "threading like Go" might have been a benefit more worth paying the cost.
    - CogitoCogito 6y ago
      > The big problem with the 2->3 transition was that the cost exceeded the perceived benefit. I believe it's generally agreed that the cost of the transition was surprisingly high. Hindsight is 20/20. > Removing the GIL would have increased the cost, but also the benefit. The only attempts I'm aware of resulted in a significant (~50%) slowdown in single-threaded code execution due to the requirement of adding in so many more locks elsewhere (removing the GIL doesn't remove the need for the locks around the various critical sections). Sure maybe it would result in better performance for some programs, but I'm not even sure it would result in something better than a cleaner design. In my experience, the portions of code that could make use of the threading could just be moved to a C/C++-extension and make use of it there (though in 99% of cases I wouldn't go that far and just stick to e.g. numpy). The examples of the code that really should stay in python, but also should make use of concurrency seem to usually sit quite nicely in the pypy framework. That's not to say that removing the GIL wouldn't be nice, but I think the need for it is often overstated.
- klyrs 6y ago
  Oh how I wish Python 3 was named Monty or something. Kill the GIL! Ruthlessly abandon backwards compatibility in the C API! It's a new language, you can do anything! Folks who delayed adoption until this year had a long and glorious period of stability, and are now waking up to a shitshow of incompatibility and half-adopted features.
  - acdha 6y ago
    > a shitshow of incompatibility and half-adopted features. To the extent that this isn’t wrong, it’s because they’re years behind on testing and upgrading from unsupported libraries. The switch isn’t hard on a relatively clean project.
    - klyrs 6y ago
      I did preface that I was talking about people who delayed adoption until the very last minute. The stability was seductive, and IMO the forced transition was long overdue. In hindsight, I wish that had happened sometime around Py36 -- but at this point, I'd rather a new language unfettered by the old C API. But what I said about half-adopted features is due to my experience with rather large bodies of code -- they're not "relatively clean" in any sense, and I often find myself doing long-overdue modernization that hinders my productivity
      - acdha 6y ago
        Again, is the problem the fraction of a Python 3 upgrade which can’t be done automatically or needing to update old third-party libraries or not having sufficient test coverage to be able to make any changes with confidence?
        klyrs 6y ago
        I'm not sure what the point of your questions is. Breaking backwards-compatibility makes headaches for maintainers for all of the reasons you list and more. I'd love to see Monty take an even more radical approach, with a "thin enough" compatibility layer. It'd be a great language with all sorts of improvements and mostly-familiar syntax, and old, stable Python libraries would stay stable -- even through a compatibility layer.
- BiteCode_dev 6y ago
  There are plenty of things that could have been down, but the Python core dev team is small and very resource limited.
shanemhansen 6y ago
I don't see the benefit of a subinterpreter compared to a subprocess. I wish python didn't have a GIL, but I don't see what problems this would solve for me that multiprocessing doesn't.
- deleted 6y ago
  [deleted]
- cheez 6y ago
  Yeah... super confusing.
- joelthelion 6y ago
  Perfomance,I suppose? Spawning a process is expensive.
  - chrstphrhrt 6y ago
    This makes sense. Combine it with asyncio and you have a full Node-like runtime without relying on Uvicorn and the like (even though clustering is still good for scaling).
    - pdonis 6y ago
      If you're using asyncio, I don't see what having a subinterpreter gains you. Asyncio by itself is already a Node-like runtime without relying on third-party libraries.
      - chrstphrhrt 6y ago
        True true, my comment was not clear. With asyncio it's still a single threaded event loop (that can be clustered like Node or any Python processes as has always been done). With sub-interpreters, I wonder how different it is from normal multithreading like in JVM. Need to read up on this idea. I can't see how it wouldn't require the same locking and mutexes or message passing if multiple interpreters are to work on logically related data. So my immediate thoughts were about leveraging it as a replacement for multithreading and event loop clustering by treating the interpreters as lightweight processes and having them communicate over some kind of protocol. Like how the BEAM does built-in process supervision. The "supervisor" process itself is async in the way it coordinates the tree.
coldtea 6y ago
>In particular, giving each subinterpreter its own global interpreter lock (GIL) is not (yet) on the table. What's the use then?
- NegativeLatency 6y ago
  https://news.ycombinator.com/item?id=23172884 https://news.ycombinator.com/item?id=23172884
  - coldtea 6y ago
    So, basically, no real reason besides "preparation for per-interpreter GIL"... I, for one, wouldn't be encouraged to use it in this state, where it doesn't really bring any benefits yet!
- AlphaSite 6y ago
  Next release iirc.
davidw 6y ago
Tcl has had those for...20 years or so? They're pretty handy for some things.
- bch 6y ago
  Absolute sandboxing, w the option to engage “safe” interp that also (tuneably) segregates it from OS facilities like the network and filesystem. /s Will be looking forward to Pythonistas to be extolling the virtues of multiple interps as much as hearing node folk go on about event-oriented programming ;) Nb: Much love to both node and python. Looking forward to seeing python exercise this.
- blacksqr 6y ago
  Yes, and have you heard about the new hotness in Python, f-strings? Special strings where you can actually embed variables and executable commands WITHIN the string!!!
VWWHFSfQ 6y ago
I feel like Lua would have gotten more traction in the early days, and maybe eclipsed Python, if it wasn't so hyper-focused on being a tiny embed-able language/runtime. It was so well-designed from the very beginning. Every new release of Python seems like it gets bigger and bigger and more and more incomprehensible. The Python 2 -> 3 transition was a (necessary?) disaster. And now we're trying, in earnest, to figure out how to get rid of the GIL. The async/await syntax is a whole other fiasco. Now we have colored functions all over the place. Python code doesn't even work with Python code. It's just an absolute mess.
- gunshai 6y ago
  Sorry can you EILI(have 3ish years of python experience but rarely get this far into the weeds.) What is an example of being more incomprehensible? >Python code doesn't even work with Python code Is this statement a function of the 2 > 3 transition.
  - brianjlogan 6y ago
    Probably additions like the Walrus operator that "complicate" the syntax and negate the "Zen" of Python by having one right way of doing something. Migrating away from the pseudocodeness of Python. Just a guess though I have about the same experience as you.
  - pansa2 6y ago
    > What is an example of being more incomprehensible? Python's type hints, in particular, have added a huge amount of complexity to the last few versions of the language.
    - deleted 6y ago
      [deleted]
    - comfydragon 6y ago
      Type annotations are completely optional though. It's possible to write Python that targets 3.5 and 3.8 at the same time.
geofft 6y ago
From the PEP: https://www.python.org/dev/peps/pep-0554/ https://www.python.org/dev/peps/pep-0554/ > A common misconception is that this PEP also includes a promise that subinterpreters will no longer share the GIL. When that is clarified, the next question is "what is the point?". This is already answered at length in this PEP. Just to be clear, the value lies in: * increase exposure of the existing feature, which helps improve the code health of the entire CPython runtime * expose the (mostly) isolated execution of subinterpreters * preparation for per-interpreter GIL * encourage experimentation I think I'll ask the followup question - what is the point of those? Why should we increase exposure of an existing feature we know is not fully baked and we know will cause problems with NumPy/SciPy? How will the exposure improve the code health of CPython and who will do the improvement? What is the advantage in exposing isolated execution of subinterpreters? In what way does exposing this feature help prepare for a per-interpreter GIL? What experiments are being encouraged specifically?
- mehrdadn 6y ago
  I think the Rationale section tries to explain the benefits? https://www.python.org/dev/peps/pep-0554/#rationale https://www.python.org/dev/peps/pep-0554/#rationale What I don't get is how they expect C extensions to adapt. I don't expect widespread support for this for like... a decade?
  - geofft 6y ago
    The Nick Coghlan quote seems a bit out of context - sure, there are all these things it does that multiprocessing doesn't do, but multiprocessing allows concurrency at all and this doesn't. The comparison makes sense if it's talking about no-shared-state concurrency, but this PEP is quite clear that it's only proposing no-shared-state ... single-threaded operation. The rest of the section is sparse on what you actually do with it. It says that it "has the potential to be a powerful tool" but not what you do with the power. It says it's about "enabling the fundamental capability of multiple isolated interpreters" but it's not clear what the capability brings you. The only thing with some details is the first sentence - "Running code in multiple interpreters provides a useful level of isolation within the same process." But what is the level? It sounds like it gives you isolation for most things but not all, which is about as useful as a face mask with a breathing hole cut out in it. It'd help to see a concrete answer of something you can build with this that you can't build without it (or perhaps not as easily/performantly/reliably/etc.). The Ceph PR, where they do something similar themselves (https://github.com/ceph/ceph/pull/14971 https://github.com/ceph/ceph/pull/14971), gives a good answer: "Notably, with this change, it's possible to have more than one mgr module use cherrypy." (But it's still not clear to me why the mgr can't just use multiple Python interpreters... ceph-mgr is kind of an amalgam of various useful services for your Ceph cluster, and it's never been clear to me why it has to be "the" manager with some modules instead of various independent services you can turn off/on as you need.)
jonathanpoulter 6y ago
This seems like such a significant problem for the language and has been around since the beginning of time. Considering the amount of value that Python adds to companies and individuals around the world, is there a reason that institutions or someone with the means hasn't funded a project to "solve" the GIL problem?
- snissn 6y ago
  I think it's just not that big of a problem / can be worked around. I had a multi terabyte of RAM across 1000s of cores python product and we just used one process per core
- davvid 6y ago
  Google threw money at the problem and that didn't help. Lookup the history behind the "Unladen Swallow" project.
  - pansa2 6y ago
    > Google threw money at the problem Not really, especially compared to the amount of money they threw at JavaScript. IIRC Unladen Swallow was just a couple of interns.
  - jashmatthews 6y ago
    Unladen Swallow wasn’t actually a serious attempt like V8 or anything and a large part of why it failed was that LLVM wasn’t ready and even when it was, it was ill suited to compiling a highly dynamic language like Python.
- acdha 6y ago
  It's far more common to have Python programs be I/O bound, and when they are CPU bound it's often not due to GIL contention (remember that C extensions can drop the GIL before doing something lengthy). It would be nice if the GIL was gone but a fair fraction of Python developers would not notice much difference.
  - 6y ago
eatonphil 6y ago
Is this like V8 Isolates for Python?
jamestimmins 6y ago
I really enjoy this style of article. Looks like the author based most of this off of mailing-list interactions. There's clearly fascinating information within those threads, despite how difficult they are to follow retroactively.
- roelschroeven 6y ago
  Absolutely. LWN has quite a lot of those; it's one of the main reasons I have a subscription there.
gtrubetskoy 6y ago
Subinterpreters existed from the very early days in the C API and were key to the implementation of mod_python (which I wrote). So if you used mod_python, you used subinterpeters without realizing it. http://modpython.org/live/current/doc-html/pythonapi.html#multiple-interpreters http://modpython.org/live/current/doc-html/pythonapi.html#mu... EDIT: And it looks like I had subinterpreters in the first released version in May 2000, so the initial git (formerly SVN) commit already had them https://github.com/grisha/mod_python/blob/9b211b7e8a65f1af4b5facd29797703f8e76d97b/src/mod_python.c#L1874 https://github.com/grisha/mod_python/blob/9b211b7e8a65f1af4b... EDIT2: Just noticed this comment: * Nov 1998 - support for multiple interpreters introduced.
- BiteCode_dev 6y ago
  How did you deal with C-extensions, since apparently most don't support it at all (which is a shame, apparently we messed up culturally here).
  - gtrubetskoy 6y ago
    I didn't :)
dilandau 6y ago
I suspect at this point it will require a new interpreter to fully address the deeply-embedded assumptions about memory access and bytecode execution in CPython. This has been tried many times, though, with only PyPy really seeing much traction. CPython being the de-facto interpreter, and the huge ecosystem of widely-used C extensions, makes any change likely to be slow... and the upside would need to be very compelling. I think what we see instead is people moving to different runtimes altogether -- golang, jvm, v8, whatever. I would suggest that Python has gone off the rails trying to mimic other languages too much recently. At some point the identity crisis hopefully ends and Python will return to it's strengths -- which are best expressed by the Zen of Python. Until then, get ready for more features.
- BiteCode_dev 6y ago
  See HPy for another solution to that: https://speakerdeck.com/antocuni/hpy-a-future-proof-way-of-extending-python?slide=1 https://speakerdeck.com/antocuni/hpy-a-future-proof-way-of-e...
aasasd 6y ago
As usual, when someone makes a move towards the Coveted Feature, people promptly insist that gradual progress is impossible and the only way is with every problem being solved at once in the hundred million modules—each somehow taking a year to undo the use of global variables. I.e. same as it was for the past twenty years when gradual progress could've been made. Deference is again made towards Numpy, the de-facto implementation of Python. No trace of choice is offered to users who would perhaps use a handful of modules from those that happen to be converted at a point in time, or use Python in ways that aren't practical now. The actual problem, of course, is that multi-threading Python would still be slow, and those who see it as just a shell to run C modules, do have a point.
trentnelson 6y ago
> NumPy core developer Sebastian Berg chimed in as well. He suggested that it could take up to a solid year of work to support subinterpreters in NumPy. Whoa, that's a shame. I actually found it really easy to patch NumPy to make it PyParallel-compatible -- simply had to tweak the memory allocator stuff: https://github.com/pyparallel/numpy/commit/046311ac1d66cec789fa8fd79b1b582a3dea26a8 https://github.com/pyparallel/numpy/commit/046311ac1d66cec78... Example that loaded a 12GB NumPy array and serviced requests in parallel: https://github.com/pyparallel/pyparallel/blob/branches/3.3-px/examples/wiki/wiki.py https://github.com/pyparallel/pyparallel/blob/branches/3.3-p.... I really wish the PyParallel approach gained more traction. Having the solution Windows-only reaaaaally didn't help with having other core developers experiment with the approach. [*]: https://pyparallel.org https://pyparallel.org
- anentropic 6y ago
  This seems great!! Obviously, being Windows-only is a big drawback - is there any hope it could be implemented for other platforms some day? (I mean in terms of technical feasibility, rather than effort, money etc) I know literally nothing about the OS side of things relevant to this topic, but googling for "async io linux" turns up io_uring which seems to date from 2019ish and is maybe addressing some of what's lacking there?
  - trentnelson 6y ago
    If you're looking for a bit more info on the Windows-only aspect, there is a lot of detail here: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores https://speakerdeck.com/trent/pyparallel-how-we-removed-the-.... I think the TL;DR though is that PyParallel was a successful proof-of-concept (but seems to have failed at moving the bar anywhere) because the threadpool and async I/O primitives on Windows are so much more sophisticated than what's available on any other platform. On Linux/Mac, I'd have to write so much scaffolding to get the same baseline functionality offered by Windows, and many parts wouldn't even be possible to replicate. It'd be a huge engineering effort that would take a team of people. (Just like all the Vista+ threadpool stuff took a team of kernel engineers working for years at Microsoft.) That being said, I've been looking at stuff like the Chromium cross-platform threadpool stuff recently and that could potentially be used as a substitute (I believe it maps 1:1 with native threadpool APIs on Windows, and mimics the best it can on Linux/Mac). But that's an unwieldy 3rd party package for Python to suddenly depend on. I also disagree with the sentiment that the GIL, parallel computing and async I/O are all separate, orthogonal pieces. The reason PyParallel was so performant was the fact that I treated all three as very intertwined concepts that had to be addressed all at once.
pinopinopino 6y ago
This is pretty common for languages right? TCL and the JVM come to mind. Nice for code running in a sandboxed environment, especially if you have full control. I think python is getting more interesting lately, always found it a bit of a boring language.
smabie 6y ago
Why does Python move at such a glacially slow pace? The language itself is unfortunate, the implementation even worse. Why can none of these problems be fixed? Though, maybe it doesn't matter. Julia is better in every way and supports easy Python interop. It's already a questionable decision to start a new Python project today and it probably will only become more questionable in the future. The language no longer serves any niche nor has any purpose. Let's just all move on.
- nurettin 6y ago
  Within the past few years python got a lot of typing improvements, nanosecond precision time, async/await, syntax that is very similar to ES6, lots of speedups, string templating and assignment expressions. How is that slow?
- BiteCode_dev 6y ago
  Read the other comments and people says it moves too fast. People are never happy.
  - nurettin 6y ago
    And beazley wouldn't care, right?
hhas01 6y ago
I recall digging into Python subinterpreters a decade ago. Abandoned it because the real killer wasn’t shared GIL, it was shared modules. If one subinterpreter imports a module and modifies that module’s state, then every other subinterpreter that uses that module is impacted too. Article says nothing about that, which makes me very cautious. The GIL only impacts performance; module sharing wrecks robustness. Even if each subinterpreter does now keep its own module cache, there’s still the challenge of working safely with common C-level resources such as file handles. (Other than telling users “don’t do that”—and GLWT.) I’ve used Python now for 17 years and it’s been a very productive tool for me. But it definitely has its baked-in limitations and fighting those is an exercise in rapidly diminishing returns. Something as fundamental as parallelism can’t just be slopped on top of a language as an afterthought; it needs to be designed for it from the start. So rather than trying to retrofit bad parallelism to Python, perhaps it’d be sense to bring the positive parts of Python over to something that already does parallelism right, such as Erlang, and build from there.
anentropic 6y ago
My first thought was, like many others, "if they still share a GIL then why bother?" But if this is a first step along the road to that, then it's all good and why not. Another thought is: could this be a "thread-safe alternative to gevent" in some use cases? I'm thinking particularly of web app hosting situation where you have something like uWSGI+gevent+your WSGI app In that case, apart from the required gevent.monkeypatch (which presumably would not be needed with subinterpreters) your web app code does not explicitly do any stuff with gevent. But it is running in a gevent thread, so your web app code now has a non-obvious requirement to be thread-safe. This has bitten me with subtle bugs in the past. I would be interested to explore the characteristics of a uWSGI+subinterpreter+WSGI app set up. It should avoid the thread-safety issue. Like gevent threads, the subinterpreters would share a GIL. I guess it might use more memory?
boublepop 6y ago
My 2 cent would be to make the necessary changes in the standard library such that an external library (on pypi) can enable the feature. That lets us start working with them, but puts no pressure on c extension developers to support it because subintepreters “aren’t standandard” but just yet another pypi library which might fail in combination with others. Then in a couple of years if the ecosystem adopt and supports them, it can be moved to standard.

157 points by lukastyrychtr 6y ago

Oh how I wish Python had just sucked it up at 3.0 and eaten the performance hit for removing the GIL. By now, everybody would have optimized it back to normal (or better!).

loeg 6y ago
There was no possible way to do that given the things python lets you get away with "atomically". Requiring explicit synchronization to avoid data races would have been an even bigger language break than the other breaking changes in 3. Python is not a language for people who want to do fine-grained explicit concurrency. And doing it implicitly would require stupidly fine-grained locks on all objects, destroying any performance gains; there's no way to "optimize" that to GIL performance.
- Doxin 6y ago
  There's still plenty of programs where you'd want to spawn a thread that basically doesn't touch any of the data of the parent thread. It'd be nice if there was a way around the GIL for those cases. As an example take the multiprocessing.pool.ThreadPool.map function. Most of the use-cases of that function only read from the parent threads memory once to pass the function arguments. After that the thread may very well spend a lot of time only reading memory it reserved itself. It's rather wasteful to have that thread wait on the GIL. Of course I haven't got an obvious solution either but it seems to me that for at least a subset of the uses of threading you could work around the GIL without breaking python too badly. Perhaps split the GIL into a per-thread Thread-interpreter-lock. That way each object can simply be annotated with which lock it belongs to. That way you still get proper atomicity like you're used to in python, but a thread might also actually run concurrently a lot of the time if it only touches objects it created itself.
  - loeg 6y ago
    > There's still plenty of programs where you'd want to spawn a thread that basically doesn't touch any of the data of the parent thread. It'd be nice if there was a way around the GIL for those cases. Subinterpreters gets you that (eventually; not in v1 apparently). > Perhaps split the GIL into a per-thread Thread-interpreter-lock. That way each object can simply be annotated with which lock it belongs to. That way you still get proper atomicity like you're used to in python, but a thread might also actually run concurrently a lot of the time if it only touches objects it created itself. I don't think this scheme would work especially well (or not better than subinterpreters). Message passing still requires copying, or some way to quiesce other threads and recursively change lock ownership of an object graph. To avoid deadlocks, at least one thread will need to drop its own lock to acquire the thread lock of objects owned by a 2nd thread. I'm not sure you could do that and preserve the legacy global atomicity behavior, and it would by definition be slower and bloatier on any individual thread than the existing GIL behavior.
- jashmatthews 6y ago
  Does the CPython GIL give you a stronger effective guarantee than the one in CRuby? Because the one in CRuby will absolutely not save you from data races.
shockinglytrue 6y ago
There really was no hope of this without ditching reference counting, and consequently a much more destablizing change to the C API which was all but entirely unharmed during the 3.x transition. There have been several attempts to bring new science to bear on solving the refcounting problem, I don't think any of them ever got far enough to be taken seriously A significant chunk of Python performance is derived from optimizations made possible by the GIL - for example, so long as it is held, no additional locks are necessary to allocate a small object, unless that entails asking the system allocator for more memory
- __d 6y ago
  The big problem with the 2->3 transition was that the cost exceeded the perceived benefit. Removing the GIL would have increased the cost, but also the benefit. I think the core team over-estimated the benefits of the change, hence people's reluctance to pay the cost. I suspect that "threading like Go" might have been a benefit more worth paying the cost.
  - CogitoCogito 6y ago
    > The big problem with the 2->3 transition was that the cost exceeded the perceived benefit. I believe it's generally agreed that the cost of the transition was surprisingly high. Hindsight is 20/20. > Removing the GIL would have increased the cost, but also the benefit. The only attempts I'm aware of resulted in a significant (~50%) slowdown in single-threaded code execution due to the requirement of adding in so many more locks elsewhere (removing the GIL doesn't remove the need for the locks around the various critical sections). Sure maybe it would result in better performance for some programs, but I'm not even sure it would result in something better than a cleaner design. In my experience, the portions of code that could make use of the threading could just be moved to a C/C++-extension and make use of it there (though in 99% of cases I wouldn't go that far and just stick to e.g. numpy). The examples of the code that really should stay in python, but also should make use of concurrency seem to usually sit quite nicely in the pypy framework. That's not to say that removing the GIL wouldn't be nice, but I think the need for it is often overstated.
klyrs 6y ago
Oh how I wish Python 3 was named Monty or something. Kill the GIL! Ruthlessly abandon backwards compatibility in the C API! It's a new language, you can do anything! Folks who delayed adoption until this year had a long and glorious period of stability, and are now waking up to a shitshow of incompatibility and half-adopted features.
- acdha 6y ago
  > a shitshow of incompatibility and half-adopted features. To the extent that this isn’t wrong, it’s because they’re years behind on testing and upgrading from unsupported libraries. The switch isn’t hard on a relatively clean project.
  - klyrs 6y ago
    I did preface that I was talking about people who delayed adoption until the very last minute. The stability was seductive, and IMO the forced transition was long overdue. In hindsight, I wish that had happened sometime around Py36 -- but at this point, I'd rather a new language unfettered by the old C API. But what I said about half-adopted features is due to my experience with rather large bodies of code -- they're not "relatively clean" in any sense, and I often find myself doing long-overdue modernization that hinders my productivity
    - acdha 6y ago
      Again, is the problem the fraction of a Python 3 upgrade which can’t be done automatically or needing to update old third-party libraries or not having sufficient test coverage to be able to make any changes with confidence?
      - klyrs 6y ago
        I'm not sure what the point of your questions is. Breaking backwards-compatibility makes headaches for maintainers for all of the reasons you list and more. I'd love to see Monty take an even more radical approach, with a "thin enough" compatibility layer. It'd be a great language with all sorts of improvements and mostly-familiar syntax, and old, stable Python libraries would stay stable -- even through a compatibility layer.
BiteCode_dev 6y ago
There are plenty of things that could have been down, but the Python core dev team is small and very resource limited.

shanemhansen 6y ago

I don't see the benefit of a subinterpreter compared to a subprocess. I wish python didn't have a GIL, but I don't see what problems this would solve for me that multiprocessing doesn't.

deleted 6y ago
[deleted]
cheez 6y ago
Yeah... super confusing.
joelthelion 6y ago
Perfomance,I suppose? Spawning a process is expensive.
- chrstphrhrt 6y ago
  This makes sense. Combine it with asyncio and you have a full Node-like runtime without relying on Uvicorn and the like (even though clustering is still good for scaling).
  - pdonis 6y ago
    If you're using asyncio, I don't see what having a subinterpreter gains you. Asyncio by itself is already a Node-like runtime without relying on third-party libraries.
    - chrstphrhrt 6y ago
      True true, my comment was not clear. With asyncio it's still a single threaded event loop (that can be clustered like Node or any Python processes as has always been done). With sub-interpreters, I wonder how different it is from normal multithreading like in JVM. Need to read up on this idea. I can't see how it wouldn't require the same locking and mutexes or message passing if multiple interpreters are to work on logically related data. So my immediate thoughts were about leveraging it as a replacement for multithreading and event loop clustering by treating the interpreters as lightweight processes and having them communicate over some kind of protocol. Like how the BEAM does built-in process supervision. The "supervisor" process itself is async in the way it coordinates the tree.

coldtea 6y ago

>In particular, giving each subinterpreter its own global interpreter lock (GIL) is not (yet) on the table. What's the use then?

NegativeLatency 6y ago
https://news.ycombinator.com/item?id=23172884 https://news.ycombinator.com/item?id=23172884
- coldtea 6y ago
  So, basically, no real reason besides "preparation for per-interpreter GIL"... I, for one, wouldn't be encouraged to use it in this state, where it doesn't really bring any benefits yet!
AlphaSite 6y ago
Next release iirc.

davidw 6y ago

Tcl has had those for...20 years or so? They're pretty handy for some things.

bch 6y ago
Absolute sandboxing, w the option to engage “safe” interp that also (tuneably) segregates it from OS facilities like the network and filesystem. /s Will be looking forward to Pythonistas to be extolling the virtues of multiple interps as much as hearing node folk go on about event-oriented programming ;) Nb: Much love to both node and python. Looking forward to seeing python exercise this.
blacksqr 6y ago
Yes, and have you heard about the new hotness in Python, f-strings? Special strings where you can actually embed variables and executable commands WITHIN the string!!!

VWWHFSfQ 6y ago

I feel like Lua would have gotten more traction in the early days, and maybe eclipsed Python, if it wasn't so hyper-focused on being a tiny embed-able language/runtime. It was so well-designed from the very beginning. Every new release of Python seems like it gets bigger and bigger and more and more incomprehensible. The Python 2 -> 3 transition was a (necessary?) disaster. And now we're trying, in earnest, to figure out how to get rid of the GIL. The async/await syntax is a whole other fiasco. Now we have colored functions all over the place. Python code doesn't even work with Python code. It's just an absolute mess.

gunshai 6y ago
Sorry can you EILI(have 3ish years of python experience but rarely get this far into the weeds.) What is an example of being more incomprehensible? >Python code doesn't even work with Python code Is this statement a function of the 2 > 3 transition.
- brianjlogan 6y ago
  Probably additions like the Walrus operator that "complicate" the syntax and negate the "Zen" of Python by having one right way of doing something. Migrating away from the pseudocodeness of Python. Just a guess though I have about the same experience as you.
- pansa2 6y ago
  > What is an example of being more incomprehensible? Python's type hints, in particular, have added a huge amount of complexity to the last few versions of the language.
  - deleted 6y ago
    [deleted]
  - comfydragon 6y ago
    Type annotations are completely optional though. It's possible to write Python that targets 3.5 and 3.8 at the same time.

geofft 6y ago

From the PEP: https://www.python.org/dev/peps/pep-0554/ https://www.python.org/dev/peps/pep-0554/ > A common misconception is that this PEP also includes a promise that subinterpreters will no longer share the GIL. When that is clarified, the next question is "what is the point?". This is already answered at length in this PEP. Just to be clear, the value lies in: * increase exposure of the existing feature, which helps improve the code health of the entire CPython runtime * expose the (mostly) isolated execution of subinterpreters * preparation for per-interpreter GIL * encourage experimentation I think I'll ask the followup question - what is the point of those? Why should we increase exposure of an existing feature we know is not fully baked and we know will cause problems with NumPy/SciPy? How will the exposure improve the code health of CPython and who will do the improvement? What is the advantage in exposing isolated execution of subinterpreters? In what way does exposing this feature help prepare for a per-interpreter GIL? What experiments are being encouraged specifically?

mehrdadn 6y ago
I think the Rationale section tries to explain the benefits? https://www.python.org/dev/peps/pep-0554/#rationale https://www.python.org/dev/peps/pep-0554/#rationale What I don't get is how they expect C extensions to adapt. I don't expect widespread support for this for like... a decade?
- geofft 6y ago
  The Nick Coghlan quote seems a bit out of context - sure, there are all these things it does that multiprocessing doesn't do, but multiprocessing allows concurrency at all and this doesn't. The comparison makes sense if it's talking about no-shared-state concurrency, but this PEP is quite clear that it's only proposing no-shared-state ... single-threaded operation. The rest of the section is sparse on what you actually do with it. It says that it "has the potential to be a powerful tool" but not what you do with the power. It says it's about "enabling the fundamental capability of multiple isolated interpreters" but it's not clear what the capability brings you. The only thing with some details is the first sentence - "Running code in multiple interpreters provides a useful level of isolation within the same process." But what is the level? It sounds like it gives you isolation for most things but not all, which is about as useful as a face mask with a breathing hole cut out in it. It'd help to see a concrete answer of something you can build with this that you can't build without it (or perhaps not as easily/performantly/reliably/etc.). The Ceph PR, where they do something similar themselves (https://github.com/ceph/ceph/pull/14971 https://github.com/ceph/ceph/pull/14971), gives a good answer: "Notably, with this change, it's possible to have more than one mgr module use cherrypy." (But it's still not clear to me why the mgr can't just use multiple Python interpreters... ceph-mgr is kind of an amalgam of various useful services for your Ceph cluster, and it's never been clear to me why it has to be "the" manager with some modules instead of various independent services you can turn off/on as you need.)

jonathanpoulter 6y ago

This seems like such a significant problem for the language and has been around since the beginning of time. Considering the amount of value that Python adds to companies and individuals around the world, is there a reason that institutions or someone with the means hasn't funded a project to "solve" the GIL problem?

snissn 6y ago
I think it's just not that big of a problem / can be worked around. I had a multi terabyte of RAM across 1000s of cores python product and we just used one process per core
davvid 6y ago
Google threw money at the problem and that didn't help. Lookup the history behind the "Unladen Swallow" project.
- pansa2 6y ago
  > Google threw money at the problem Not really, especially compared to the amount of money they threw at JavaScript. IIRC Unladen Swallow was just a couple of interns.
- jashmatthews 6y ago
  Unladen Swallow wasn’t actually a serious attempt like V8 or anything and a large part of why it failed was that LLVM wasn’t ready and even when it was, it was ill suited to compiling a highly dynamic language like Python.
acdha 6y ago
It's far more common to have Python programs be I/O bound, and when they are CPU bound it's often not due to GIL contention (remember that C extensions can drop the GIL before doing something lengthy). It would be nice if the GIL was gone but a fair fraction of Python developers would not notice much difference.
- 6y ago

eatonphil 6y ago

Is this like V8 Isolates for Python?

jamestimmins 6y ago

I really enjoy this style of article. Looks like the author based most of this off of mailing-list interactions. There's clearly fascinating information within those threads, despite how difficult they are to follow retroactively.

roelschroeven 6y ago
Absolutely. LWN has quite a lot of those; it's one of the main reasons I have a subscription there.

gtrubetskoy 6y ago

Subinterpreters existed from the very early days in the C API and were key to the implementation of mod_python (which I wrote). So if you used mod_python, you used subinterpeters without realizing it. http://modpython.org/live/current/doc-html/pythonapi.html#multiple-interpreters http://modpython.org/live/current/doc-html/pythonapi.html#mu... EDIT: And it looks like I had subinterpreters in the first released version in May 2000, so the initial git (formerly SVN) commit already had them https://github.com/grisha/mod_python/blob/9b211b7e8a65f1af4b5facd29797703f8e76d97b/src/mod_python.c#L1874 https://github.com/grisha/mod_python/blob/9b211b7e8a65f1af4b... EDIT2: Just noticed this comment: * Nov 1998 - support for multiple interpreters introduced.

BiteCode_dev 6y ago
How did you deal with C-extensions, since apparently most don't support it at all (which is a shame, apparently we messed up culturally here).
- gtrubetskoy 6y ago
  I didn't :)

dilandau 6y ago

I suspect at this point it will require a new interpreter to fully address the deeply-embedded assumptions about memory access and bytecode execution in CPython. This has been tried many times, though, with only PyPy really seeing much traction. CPython being the de-facto interpreter, and the huge ecosystem of widely-used C extensions, makes any change likely to be slow... and the upside would need to be very compelling. I think what we see instead is people moving to different runtimes altogether -- golang, jvm, v8, whatever. I would suggest that Python has gone off the rails trying to mimic other languages too much recently. At some point the identity crisis hopefully ends and Python will return to it's strengths -- which are best expressed by the Zen of Python. Until then, get ready for more features.

BiteCode_dev 6y ago
See HPy for another solution to that: https://speakerdeck.com/antocuni/hpy-a-future-proof-way-of-extending-python?slide=1 https://speakerdeck.com/antocuni/hpy-a-future-proof-way-of-e...

aasasd 6y ago

As usual, when someone makes a move towards the Coveted Feature, people promptly insist that gradual progress is impossible and the only way is with every problem being solved at once in the hundred million modules—each somehow taking a year to undo the use of global variables. I.e. same as it was for the past twenty years when gradual progress could've been made. Deference is again made towards Numpy, the de-facto implementation of Python. No trace of choice is offered to users who would perhaps use a handful of modules from those that happen to be converted at a point in time, or use Python in ways that aren't practical now. The actual problem, of course, is that multi-threading Python would still be slow, and those who see it as just a shell to run C modules, do have a point.

trentnelson 6y ago

> NumPy core developer Sebastian Berg chimed in as well. He suggested that it could take up to a solid year of work to support subinterpreters in NumPy. Whoa, that's a shame. I actually found it really easy to patch NumPy to make it PyParallel-compatible -- simply had to tweak the memory allocator stuff: https://github.com/pyparallel/numpy/commit/046311ac1d66cec789fa8fd79b1b582a3dea26a8 https://github.com/pyparallel/numpy/commit/046311ac1d66cec78... Example that loaded a 12GB NumPy array and serviced requests in parallel: https://github.com/pyparallel/pyparallel/blob/branches/3.3-px/examples/wiki/wiki.py https://github.com/pyparallel/pyparallel/blob/branches/3.3-p.... I really wish the PyParallel approach gained more traction. Having the solution Windows-only reaaaaally didn't help with having other core developers experiment with the approach. [*]: https://pyparallel.org https://pyparallel.org

anentropic 6y ago
This seems great!! Obviously, being Windows-only is a big drawback - is there any hope it could be implemented for other platforms some day? (I mean in terms of technical feasibility, rather than effort, money etc) I know literally nothing about the OS side of things relevant to this topic, but googling for "async io linux" turns up io_uring which seems to date from 2019ish and is maybe addressing some of what's lacking there?
- trentnelson 6y ago
  If you're looking for a bit more info on the Windows-only aspect, there is a lot of detail here: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores https://speakerdeck.com/trent/pyparallel-how-we-removed-the-.... I think the TL;DR though is that PyParallel was a successful proof-of-concept (but seems to have failed at moving the bar anywhere) because the threadpool and async I/O primitives on Windows are so much more sophisticated than what's available on any other platform. On Linux/Mac, I'd have to write so much scaffolding to get the same baseline functionality offered by Windows, and many parts wouldn't even be possible to replicate. It'd be a huge engineering effort that would take a team of people. (Just like all the Vista+ threadpool stuff took a team of kernel engineers working for years at Microsoft.) That being said, I've been looking at stuff like the Chromium cross-platform threadpool stuff recently and that could potentially be used as a substitute (I believe it maps 1:1 with native threadpool APIs on Windows, and mimics the best it can on Linux/Mac). But that's an unwieldy 3rd party package for Python to suddenly depend on. I also disagree with the sentiment that the GIL, parallel computing and async I/O are all separate, orthogonal pieces. The reason PyParallel was so performant was the fact that I treated all three as very intertwined concepts that had to be addressed all at once.

pinopinopino 6y ago

This is pretty common for languages right? TCL and the JVM come to mind. Nice for code running in a sandboxed environment, especially if you have full control. I think python is getting more interesting lately, always found it a bit of a boring language.

smabie 6y ago

Why does Python move at such a glacially slow pace? The language itself is unfortunate, the implementation even worse. Why can none of these problems be fixed? Though, maybe it doesn't matter. Julia is better in every way and supports easy Python interop. It's already a questionable decision to start a new Python project today and it probably will only become more questionable in the future. The language no longer serves any niche nor has any purpose. Let's just all move on.

nurettin 6y ago
Within the past few years python got a lot of typing improvements, nanosecond precision time, async/await, syntax that is very similar to ES6, lots of speedups, string templating and assignment expressions. How is that slow?
BiteCode_dev 6y ago
Read the other comments and people says it moves too fast. People are never happy.
- nurettin 6y ago
  And beazley wouldn't care, right?

hhas01 6y ago

I recall digging into Python subinterpreters a decade ago. Abandoned it because the real killer wasn’t shared GIL, it was shared modules. If one subinterpreter imports a module and modifies that module’s state, then every other subinterpreter that uses that module is impacted too. Article says nothing about that, which makes me very cautious. The GIL only impacts performance; module sharing wrecks robustness. Even if each subinterpreter does now keep its own module cache, there’s still the challenge of working safely with common C-level resources such as file handles. (Other than telling users “don’t do that”—and GLWT.) I’ve used Python now for 17 years and it’s been a very productive tool for me. But it definitely has its baked-in limitations and fighting those is an exercise in rapidly diminishing returns. Something as fundamental as parallelism can’t just be slopped on top of a language as an afterthought; it needs to be designed for it from the start. So rather than trying to retrofit bad parallelism to Python, perhaps it’d be sense to bring the positive parts of Python over to something that already does parallelism right, such as Erlang, and build from there.

anentropic 6y ago

My first thought was, like many others, "if they still share a GIL then why bother?" But if this is a first step along the road to that, then it's all good and why not. Another thought is: could this be a "thread-safe alternative to gevent" in some use cases? I'm thinking particularly of web app hosting situation where you have something like uWSGI+gevent+your WSGI app In that case, apart from the required gevent.monkeypatch (which presumably would not be needed with subinterpreters) your web app code does not explicitly do any stuff with gevent. But it is running in a gevent thread, so your web app code now has a non-obvious requirement to be thread-safe. This has bitten me with subtle bugs in the past. I would be interested to explore the characteristics of a uWSGI+subinterpreter+WSGI app set up. It should avoid the thread-safety issue. Like gevent threads, the subinterpreters would share a GIL. I guess it might use more memory?

boublepop 6y ago

My 2 cent would be to make the necessary changes in the standard library such that an external library (on pypi) can enable the feature. That lets us start working with them, but puts no pressure on c extension developers to support it because subintepreters “aren’t standandard” but just yet another pypi library which might fail in combination with others. Then in a couple of years if the ecosystem adopt and supports them, it can be moved to standard.