4 ms·

Without a GIL it would be incredibly difficult, would it not? AFAIK C extensions to MRI are not thread safe. This could, of course, expose a different C API th

Without a GIL it would be incredibly difficult, would it not? AFAIK C extensions to MRI are not thread safe. This could, of course, expose a different C API that allows for thread safe extensions, but I think the original comment is complaining about lack of compatibility with all of the important gems that depend on the existing MRI C API. This has always been an issue that held back JRuby.

XMPPwocky 7y ago
Could one add, effectively, a RWLock GIL - one where any number of Ruby interpreter threads can run at once (since they know how to not step on each other's toes), but any C code which tried to "grab the GIL" would run exclusively w.r.t. interpreter threads (and other C code)?
- riffraff 7y ago
  I believe the "guild" proposal makes it such that the global lock becomes guild-specific, which seems similar to what you propose.
pjmlp 7y ago
JRuby does not have a GIL.
- y4mi 7y ago
  My personal experience with Ruby is extremely shallow, but a casual glance at the search results wrt the GIL in Ruby seems to say it actually does have one. I could be wrong, as I'm not an experienced Ruby developer, however. I.e. https://thoughtbot.com/blog/untangling-ruby-threads https://thoughtbot.com/blog/untangling-ruby-threads
  - pjmlp 7y ago
    Ruby has multiple implementations, JRuby uses Java Threads without any sort of GIL. Plenty of talks done on this subject by Charles Nutter.
  - yorwba 7y ago
    From your link: "Note that this is not the case for JRuby or Rubinius which do not have a GIL and offer true multi-threading."
- msbarnett 7y ago
  I’m aware. It also lacks compatibility with a lot of important C gems (not only or even predominantly because of the lack of a GIL, of course), which I’m pointing out has always impacted its adoption.
  - pjmlp 7y ago
    Then you should also be aware that it runs most of them via Sulong.
    - eregon 7y ago
      JRuby has no support for C extensions. TruffleRuby supports C extensions, and has no GIL for Ruby code.
    - msbarnett 7y ago
      That’s Truffle, not JRuby. There was a plan several years ago to move Sulong support into JRuby based on trialing it in Truffle; it appears to have made no progress since 2016.
ROFISH 7y ago
The proposed API, MRB_API, is used in mruby. With enough popular support for that C API, it could force mainline ruby to modernize it's C API to that one. Assuming MRB_API thread-safe-able. mruby is not thread safe at all, being a single-threaded script runner.
- p4lindromica 7y ago
  Hi, I'm the author of Artichoke. My expectation is that MRB_API can be implemented in a thread-safe way. The API manipulates the interpreter state (in the case of mruby, an `mrb_state`). The interpreter state itself and the implementation of the API can be thread safe or they can not be. I don't think mruby even assumes the presence of threads on target systems, so it is completely reasonable that the implementation is not thread safe.
p4lindromica 7y ago
Hi, I'm the author of Artichoke. I don't have a good answer for whether or not implementing the MRI API would be difficult. I _suspected_ it would be, which is why I listed it as a non-goal. There is a lot of other work to do before we get to the point of C API compatibility. However, it may be possible to use `rubysys` in rutie as a base for an MRI compatible API: https://github.com/artichoke/artichoke/issues/88 https://github.com/artichoke/artichoke/issues/88
- pohl 7y ago
  You might want to clarify in the readme that the red X means non-goal. Until I read this comment I thought it was a goal that had not yet been implemented.
  - p4lindromica 7y ago
    in the 14 hours since I wrote the comment you replied to, I moved an MRI C API from non-goal to goal [0]. :) The red X means the goal is not yet achieved. [0] https://github.com/artichoke/artichoke/pull/108 https://github.com/artichoke/artichoke/pull/108
jashmatthews 7y ago
Something similar to Racket places or V8 isolates can be implemented and keep compatibility with C extensions expecting GIL. It just needs somebody to actually do the work of swapping out hundreds of global variable accesses in MRI using something like CIL.
- eregon 7y ago
  Do you mean every thread would have its own place/isolate for executing C extensions? I think one difficulty there is some global state in C extensions might expect to be truly process-global. Also, how would you isolate global variables in the C extension? If the C extension is a dynamic library, it's typically only loaded once per process. Maybe something like dlmopen() to load multiple copies of a native library would help, but it's not portable.
  - jashmatthews 7y ago
    Yeah, this is a real problem. The PG gem, for example, uses some global variables. IMHO the only way to do it is to add incremental parallelism which leaves the GIL in place. Racket has already shown a solid path here. Guilds would have a major performance problem: can't allocate objects without GIL. It's also a tricky mental model and requires invasive changes to existing Ruby code to handle frozen objects. Places don't share a heap so they don't need the GIL to allocate objects and have independent GC rather than global GC. It's a model which fits better with existing Ruby code. The GVL can be relaxed while executing Ruby and grabbed by native methods. WDYT?
    - eregon 7y ago
      > Guilds would have a major performance problem: can't allocate objects without GIL. Why not? It'd be possible to have TLABs, isn't it? But yes, GC would still be for all Guilds at once. Racket places don't allow to share objects, only arrays of primitive types, which seems very restrictive. And Racket futures are even more restricted. I'm not sure about Guilds since it's not there yet, but it already sounds closer to Ruby multithreading and more flexible to me from an usage point of view.

XMPPwocky 7y ago

Could one add, effectively, a RWLock GIL - one where any number of Ruby interpreter threads can run at once (since they know how to not step on each other's toes), but any C code which tried to "grab the GIL" would run exclusively w.r.t. interpreter threads (and other C code)?

riffraff 7y ago
I believe the "guild" proposal makes it such that the global lock becomes guild-specific, which seems similar to what you propose.

pjmlp 7y ago

JRuby does not have a GIL.

y4mi 7y ago
My personal experience with Ruby is extremely shallow, but a casual glance at the search results wrt the GIL in Ruby seems to say it actually does have one. I could be wrong, as I'm not an experienced Ruby developer, however. I.e. https://thoughtbot.com/blog/untangling-ruby-threads https://thoughtbot.com/blog/untangling-ruby-threads
- pjmlp 7y ago
  Ruby has multiple implementations, JRuby uses Java Threads without any sort of GIL. Plenty of talks done on this subject by Charles Nutter.
- yorwba 7y ago
  From your link: "Note that this is not the case for JRuby or Rubinius which do not have a GIL and offer true multi-threading."
msbarnett 7y ago
I’m aware. It also lacks compatibility with a lot of important C gems (not only or even predominantly because of the lack of a GIL, of course), which I’m pointing out has always impacted its adoption.
- pjmlp 7y ago
  Then you should also be aware that it runs most of them via Sulong.
  - eregon 7y ago
    JRuby has no support for C extensions. TruffleRuby supports C extensions, and has no GIL for Ruby code.
  - msbarnett 7y ago
    That’s Truffle, not JRuby. There was a plan several years ago to move Sulong support into JRuby based on trialing it in Truffle; it appears to have made no progress since 2016.

ROFISH 7y ago

The proposed API, MRB_API, is used in mruby. With enough popular support for that C API, it could force mainline ruby to modernize it's C API to that one. Assuming MRB_API thread-safe-able. mruby is not thread safe at all, being a single-threaded script runner.

p4lindromica 7y ago
Hi, I'm the author of Artichoke. My expectation is that MRB_API can be implemented in a thread-safe way. The API manipulates the interpreter state (in the case of mruby, an `mrb_state`). The interpreter state itself and the implementation of the API can be thread safe or they can not be. I don't think mruby even assumes the presence of threads on target systems, so it is completely reasonable that the implementation is not thread safe.

p4lindromica 7y ago

Hi, I'm the author of Artichoke. I don't have a good answer for whether or not implementing the MRI API would be difficult. I _suspected_ it would be, which is why I listed it as a non-goal. There is a lot of other work to do before we get to the point of C API compatibility. However, it may be possible to use `rubysys` in rutie as a base for an MRI compatible API: https://github.com/artichoke/artichoke/issues/88 https://github.com/artichoke/artichoke/issues/88

pohl 7y ago
You might want to clarify in the readme that the red X means non-goal. Until I read this comment I thought it was a goal that had not yet been implemented.
- p4lindromica 7y ago
  in the 14 hours since I wrote the comment you replied to, I moved an MRI C API from non-goal to goal [0]. :) The red X means the goal is not yet achieved. [0] https://github.com/artichoke/artichoke/pull/108 https://github.com/artichoke/artichoke/pull/108

jashmatthews 7y ago

Something similar to Racket places or V8 isolates can be implemented and keep compatibility with C extensions expecting GIL. It just needs somebody to actually do the work of swapping out hundreds of global variable accesses in MRI using something like CIL.

eregon 7y ago
Do you mean every thread would have its own place/isolate for executing C extensions? I think one difficulty there is some global state in C extensions might expect to be truly process-global. Also, how would you isolate global variables in the C extension? If the C extension is a dynamic library, it's typically only loaded once per process. Maybe something like dlmopen() to load multiple copies of a native library would help, but it's not portable.
- jashmatthews 7y ago
  Yeah, this is a real problem. The PG gem, for example, uses some global variables. IMHO the only way to do it is to add incremental parallelism which leaves the GIL in place. Racket has already shown a solid path here. Guilds would have a major performance problem: can't allocate objects without GIL. It's also a tricky mental model and requires invasive changes to existing Ruby code to handle frozen objects. Places don't share a heap so they don't need the GIL to allocate objects and have independent GC rather than global GC. It's a model which fits better with existing Ruby code. The GVL can be relaxed while executing Ruby and grabbed by native methods. WDYT?
  - eregon 7y ago
    > Guilds would have a major performance problem: can't allocate objects without GIL. Why not? It'd be possible to have TLABs, isn't it? But yes, GC would still be for all Guilds at once. Racket places don't allow to share objects, only arrays of primitive types, which seems very restrictive. And Racket futures are even more restricted. I'm not sure about Guilds since it's not there yet, but it already sounds closer to Ruby multithreading and more flexible to me from an usage point of view.