3 ms·

Ruby seems to have got past its GIL problems with Ractor[1]. What's stopping Python from doing something similar? [1] https://scoutapm.com/blog/ruby-ractor htt

Ruby seems to have got past its GIL problems with Ractor[1]. What's stopping Python from doing something similar? [1] https://scoutapm.com/blog/ruby-ractor https://scoutapm.com/blog/ruby-ractor

r4victor 5y ago
As far as I understand, Ruby's Ractors are what Python is now trying to do with subinterpreters (https://lwn.net/Articles/820424/ https://lwn.net/Articles/820424/). I mention them in the post.
- danachow 5y ago
  I just skimmed through your link. Subinterpreters have been in the C api for a long time and never got around the GIL, and reading over the post that you linked — nothing has really changed.
  - r4victor 5y ago
    This is a WIP. Here's how it's going on: https://pythondev.readthedocs.io/subinterpreters.html https://pythondev.readthedocs.io/subinterpreters.html
georgyo 5y ago
Python has multiprocessing which also bypasses the GIL. It also works with message passing.
- helpfulmandrill 5y ago
  Much more heavyweight than Ractors though, right?
- mzs 5y ago
  Which was broken on macOS until 3.8 and is fragile in the face of threading and freezing.
jrochkind1 5y ago
I wouldn't say that ruby has "got past its GIL problems with Ractor" -- you can run very little actually pre-existing libraries with ractors, because it requires no references to global state, and most non-trivial existing code ends up referencing global state (for configuration, caching, etc). It requires a different approach to writing code than has historically been done in ruby, to scrupulously identify all global state and make it ractor-safe. The ruby ractor implementation doesn't allow existing code to "just work". Some of these patterns and mitigations and best approaches for cost/benefit are still being worked out. At this point ractor is early in the experimental stage. It's not like existing applications can currently just "switch to ractor", and problem solved. There is little if any real-world production code running on ractors at present. But yes, this article's description of the work on "subinterprters" does sound very much like ruby ractors: > The idea is to have multiple interpreters within the same process. Threads within one interpreter still share the GIL, but multiple interpreters can run parallel. No GIL is needed to synchronize interpreters because they have no common global state and do not share Python objects. All global state is made per-interpreter, and interpreters communicate via message passing only Yep, that's how ruby ractors work... except that I'm not sure I'd say "all global state is made per-interpreter", ractors still have global state... they just error if you try to access it in (traditional) unsafe ways! I am curious if python's implementation has differnet semantics. In general, I'd be really excited to see someone write up a comparison of python and ruby approaches here. They are very similar languages in the end, and have a lot to learn from each other, it would be interesting to see how differences in implementation/semantic choices here lead to different practical results. But it seems like few people are "expert" in both languages, or otherwise have the interest, to notice and write the comparison!
- helpfulmandrill 5y ago
  Ok, not "solved" in a "everything just works and nobody notices" way, but they have a way forward. The "shared-nothing Actors" model is well demonstrated to work well in Erlang, Elixir etc., so it seems like a reasonable way forward for a language like Ruby or Python.
  - jrochkind1 5y ago
dragonwriter 5y ago
> Ruby seems to have got past its GIL problems with Ractor Ractor doesn’t eliminate the GVL, it just introduces a scope wider than that covered by the GVL and smaller than a process (but bigger than a thread.) It enables a model between simple multithreading and multiple OS-level processes. It is heavier parallelism than a thread-safe core language would allow, but should have less compatibility impact on legacy code and performance impact on single-threaded code (plus, its a lot easier to get to starting with an implementation with a GIL/GVL.)

Ruby seems to have got past its GIL problems with Ractor[1]. What's stopping Python from doing something similar? [1] https://scoutapm.com/blog/ruby-ractor https://scoutapm.com/blog/ruby-ractor

r4victor 5y ago

As far as I understand, Ruby's Ractors are what Python is now trying to do with subinterpreters (https://lwn.net/Articles/820424/ https://lwn.net/Articles/820424/). I mention them in the post.

danachow 5y ago
I just skimmed through your link. Subinterpreters have been in the C api for a long time and never got around the GIL, and reading over the post that you linked — nothing has really changed.
- r4victor 5y ago
  This is a WIP. Here's how it's going on: https://pythondev.readthedocs.io/subinterpreters.html https://pythondev.readthedocs.io/subinterpreters.html

georgyo 5y ago

Python has multiprocessing which also bypasses the GIL. It also works with message passing.

helpfulmandrill 5y ago
Much more heavyweight than Ractors though, right?
mzs 5y ago
Which was broken on macOS until 3.8 and is fragile in the face of threading and freezing.

jrochkind1 5y ago

I wouldn't say that ruby has "got past its GIL problems with Ractor" -- you can run very little actually pre-existing libraries with ractors, because it requires no references to global state, and most non-trivial existing code ends up referencing global state (for configuration, caching, etc). It requires a different approach to writing code than has historically been done in ruby, to scrupulously identify all global state and make it ractor-safe. The ruby ractor implementation doesn't allow existing code to "just work". Some of these patterns and mitigations and best approaches for cost/benefit are still being worked out. At this point ractor is early in the experimental stage. It's not like existing applications can currently just "switch to ractor", and problem solved. There is little if any real-world production code running on ractors at present. But yes, this article's description of the work on "subinterprters" does sound very much like ruby ractors: > The idea is to have multiple interpreters within the same process. Threads within one interpreter still share the GIL, but multiple interpreters can run parallel. No GIL is needed to synchronize interpreters because they have no common global state and do not share Python objects. All global state is made per-interpreter, and interpreters communicate via message passing only Yep, that's how ruby ractors work... except that I'm not sure I'd say "all global state is made per-interpreter", ractors still have global state... they just error if you try to access it in (traditional) unsafe ways! I am curious if python's implementation has differnet semantics. In general, I'd be really excited to see someone write up a comparison of python and ruby approaches here. They are very similar languages in the end, and have a lot to learn from each other, it would be interesting to see how differences in implementation/semantic choices here lead to different practical results. But it seems like few people are "expert" in both languages, or otherwise have the interest, to notice and write the comparison!

helpfulmandrill 5y ago
Ok, not "solved" in a "everything just works and nobody notices" way, but they have a way forward. The "shared-nothing Actors" model is well demonstrated to work well in Erlang, Elixir etc., so it seems like a reasonable way forward for a language like Ruby or Python.
- jrochkind1 5y ago

dragonwriter 5y ago

> Ruby seems to have got past its GIL problems with Ractor Ractor doesn’t eliminate the GVL, it just introduces a scope wider than that covered by the GVL and smaller than a process (but bigger than a thread.) It enables a model between simple multithreading and multiple OS-level processes. It is heavier parallelism than a thread-safe core language would allow, but should have less compatibility impact on legacy code and performance impact on single-threaded code (plus, its a lot easier to get to starting with an implementation with a GIL/GVL.)