12 ms·

About concurrency and the GIL

82 points by telemachos 15y ago

jbert 15y ago
With most web architectures, your state is outside the process (RDBMS, NoSQL, filesystem, whatever persistent store you're using). So the main benefit using threads (easy sharing of in-process state) goes away. For web architectures, concurrent request handling seems to me to be best achieved by multi-process, rather than multi-thread. There are some minor benefits to threading (e.g. an in-memory cache of global state can be shared amongst multiple threads, rather than being replicated among multiple processes) but: - there are out-of-process cacheing technologies (memcached, redis etc) - threading doesn't come "for free", in that you need to pay a performance cost in terms of locking in your interpreter and/or a code complexity cost in terms of access to your shared data structures.
- wladimir 15y ago
  Also, don't forget that threads don't scale beyond one machine; so when you want expand beyond one machine you need to take care of two levels of scaling, threads and processes. This can be avoided when using multiple processes that communicate or share state some other way in the first place. Even with one CPU, with many cores the internal synchronization that happens to "simulate" a shared memory space can be expensive, if you're not careful your cores get clogged ping-ponging memory pages between each other. It might sound more convenient to use threads instead of processes, but in the end I'm not sure that all the work to remove the GIL (and introduce shitloads of finer-grained synchronization primitives) is worth it.
  - bad_user 15y ago
    Threads are important not because you want to scale horizontally, but because you want to also scale vertically, as in having an optimum ratio of performance per Watts or CPU cores or RAM MB. Threads are much, much more efficient than processes - threads consume less memory (even with COW, VMs with garbage-collectors basically prevent COW from being effective), threads have faster context-switch, and threads achieve better cache-locality. And shared memory, which is considered the plague of the software industry along with pointers to memory, is actually a mirror of our current hardware architecture. The fact is that hard-disks and SSDs are much slower than RAM, RAM memory is much slower than L2 cache, which is slower than L1 cache, which is slower than when registers get accessed. You achieve good performance characteristics when you keep your data in one place, keeping your most accessed items in the CPU's cache, or at least prevent those items from being stored in swap. Food for thought: the NoSQL solutions so loved by Rails or Node.js developers are mostly written in C/C++ and rely on Posix threads, with some Erlang here and there. might sound more convenient to use threads instead of processes That's because it IS more convenient to use threads instead of processes in most cases. You can mostly get away with it only when the processes don't need to synchronize (i.e. your workload can be processed fully in parallel), but when processes do need to synchronize lots of bad shit can happen, as processes are more unreliable than threads. Of course, I'm referring to real kernel-managed POSIX threads and processes, not what Erlang does, but then again, the Erlang's light-weight processes are just an abstraction over POSIX threads, not processes as that would have been dumb.
    - wladimir 15y ago
      Threads are much, much more efficient than processes - threads consume less memory (even with COW, VMs with garbage-collectors basically prevent COW from being effective), threads have faster context-switch, and threads achieve better cache-locality. There is hardly any "context switch" if you run one process per core (compared to one thread per core, it makes no difference). And yes, you'll save memory if you do everything in one process (even with COW), but don't forget garbage collection with multiple threads is a much more complex beast than single-threaded, resulting in extra overhead. I'm not denying that threads have some advantages, for example, when parallelizing heavy number-crunching work, but the point I was trying to make is that "they are much more efficient" is not that clear-cut and universally true. That's because it IS more convenient to use threads instead of processes in most cases Not necessarily true either. When using threads there is a much larger chance of introducing race conditions and deadlocks and other synchronization fails. It might look more convenient when writing the code, but debugging and supporting it certainly isn't.
      - buff-a 15y ago
        There is hardly any "context switch" if you run one process per core There is hardly any context switch if I don't run any software at all on my server. Zero bugs either. Back to the real world, and the point of this discussion, we run multiple Ruby processes per CPU so that when one process is waiting for the db or memcache, another can be doing useful work on the same CPU. When using threads there is a much larger chance of introducing race conditions and deadlocks and other synchronization fails As opposed to dropping the problem into memcache and getting it wrong: your code "works", you get no crashes, but customers lose data, lose posts, or lose money. If you're advocating using memcache because multithreading is too hard for your programmer, then your customers are fucked.
        wladimir 15y ago
        We run multiple Ruby processes per CPU so that when one process is waiting for the db or memcache, another can be doing useful work on the same CPU That's the trivial case. When waiting for the db or memcache (I/O in general), the GIL is lifted, so it doesn't get in the way. It only gets in the way if multiple threads are inside the ruby VM doing work.
- buff-a 15y ago
  threading doesn't come "for free", in that you need to pay a performance cost in terms of locking in your interpreter and/or a code complexity cost in terms of access to your shared data structures vs there are out-of-process cacheing technologies (memcached, redis etc) If you subject "out-of-process cacheing technologies" to the same measurements (performance cost, code complexity), as threading solutions, what do you find?
  - simonw 15y ago
    That they're massively easier to work with?
  - jbert 15y ago
    > If you subject "out-of-process cacheing technologies" to the same measurements (performance cost, code complexity), as threading solutions, what do you find? Yes, I think you find that shared state is difficult. So it's the old process (private by default, shared as the exception) versus threads (shared by default) split. imho, "less shared state" == "simpler". And imho, "private by default" == "less shared state". And you have an easier time of it the shared state helps you with concurrent access. This is one benefit of RDBMSs (transactions) and also systems like redis. The fact that your state is external means it is more likely to have been designed to be conncurrency-safe.
    - antrix 15y ago
      I don't get this argument. If I replace my memcache logic with an in-memory thread-safe hashmap, it will always be faster. I can't see how it can be slow. As for 'simpler', as long as you aren't actually implementing the thread safe map, it is simpler than using memcache too. So as long as you use well written shared state implementations, (e.g. Java's concurrent collections), shared state isn't as hard as you make it out to be.
      - jbert 15y ago
        > If I replace my memcache logic with an in-memory thread-safe hashmap, it will always be faster. I can't see how it can be slow. Because in a threaded application context, you have to either: - have locking/synchronization on all data structures (performance cost) OR - manage which of your data structures are shared and which are not (complexity cost/race bugs) With external, shared state (RDBMs, memcached, etc) you get fast in-process access to your (private) data structures (no performance cost due to locking) and a (mostly) concurrency-safe, explicit datastore. And as pointed out elsewhere, to scale beyond a single process you need the external state anyway. There are other approaches to this, software transactional memory, functional programming, clojure paradigm etc. But it's a hard problem and it's not clear these are the right solution at scale.
- jerf 15y ago
  "With most web architectures, your state is outside the process (RDBMS, NoSQL, filesystem, whatever persistent store you're using)." I submit the fact that is nearly 100% is effect, not cause. If web technologies weren't so bad at handling multiple cores, you'd be more likely to move your web apps up to where ever the "real" computation is taking place, so as to remove a link in the chain. But since that is almost entirely not an option, nobody does it. It doesn't prove it's the "right" way to do it if it's the only way to do it right now. Using Erlang and Haskell tech, I've actually had some great fun writing applications that are heavily multithreaded and happen to include a web server too, built in. It isn't the solution to every problem, but damn it's fun and nice not to spend so much time marshalling things over a database gap for what is basically no reason. The database goes back to just being persistence and querying, instead of communication as well.
  - jbert 15y ago
    > I submit the fact that is nearly 100% is effect, not cause. Well, there's also the point that your shared state needs to be process-external as soon as you want to scale beyond a single process. (e.g. need to run on more than one server). Yes - most sites/apps don't need that, but given that a lot of the web site framework and tooling is set up to make this relatively easy to do, it's nice to not have to rework your app fairly fundamentally once you hit that scaling wall.
    - gcampbell 15y ago
      Depending on the application, there are some options for process-internal multi-machine shared state - Erlang's mnesia distributed database, for instance.
      - jbert 15y ago
        That's interesting, thanks.
nupark2 15y ago
I'm really tired of these disingenuous justifications and arguments to 'just use processes!' A GIL is an unnecessary limitation that precludes a huge swath of architecture optimizations. It's there because it's difficult to remove once you've made the incorrect choice to rely on it, not because a GIL is a good idea.
- cbs 15y ago
  This is very true. It reminds me of the BKL, yeah we could technically live with it, but it would be lazy and bad practice for a platform. Others in this thread are saying that GILs are OK because they're working on web applications. Not everyone shares your narrow requirements when using a general purpose language.
  - jbert 15y ago
    > Not everyone shares your narrow requirements when using a general purpose language. IMHO, if the resource difference between processes and threads is significant for your workload then moving away from an interpreted language is probably a step to take before going threaded. i.e. remove the overhead of the interpreter, before you worry about overhead of a GIL. Also note that a GIL gives better performance on a single-threaded workload, since you avoid the overhead of fine-grained locking. So, asking for GIL removal is asking all users of the language to pay a 'threading tax', when it isn't at all clear that there would be any real benificiaries (since the above argument can be made that those workloads would be better in a different language). I'm not rabidly anti-thread, but I think it is important to note that there are good reasons (not just laziness) why you might not want the GIL removed - i.e. I don't think it's as simple as being bad practice.
    - cbs 15y ago
      >remove the overhead of the interpreter, before you worry about overhead of a GIL. The overhead of the GIL killing parallelism is nontrivial, even next to the interpreter overhead >(since the above argument can be made that those workloads would be better in a different language). Thats a pretty self defeating argument. Every single possible ruby workload would perform better in another language. >So, asking for GIL removal is asking all users of the language to pay a 'threading tax', First, the existence of the multithreading support already has a threading tax, the GIL. Second, sprinkling in the use of some more mutexes wouldn't make a significant performance impact, especially not in a interpreted language that we already agree is not good for performance. Of course, if the runtime knew it was only running one thread it could ignore much the synchronization code altogether, but we don't want to pay the tax of a single branch decision, do we? >IMHO, if the resource difference between processes and threads is significant [...] What about all situations where you want to use threads, but not due to resource constraints? I would assume your answer would be "don't use threads" but you say you're not rabidly anti-thread.
drnicwilliams 15y ago
Three quotes, hopefully not out of useful context, that seem incongruous: "Rubinius is about the join JRuby and MacRuby in the realm of GIL-less Ruby implementations" "I spend my free time working on an alternative Ruby implementation which doesn’t use a GIL (MacRuby)" "I respect Matz’ decision to keep the GIL even though, I would personally prefer to push the data safety responsibility to the developers. However, I do know that many Ruby developers would end up shooting themselves in the foot" So developers using GIL-less MacRuby, JRuby and Rubinius are prone to foot shooting? I wish they'd blog about this more, I've never once heard a MacRuby or JRuby developer blog saying "I went back to MRI because I needed my Ruby code to be run more safely".
- jballanc 15y ago
  I can't speak for JRuby or Rubinius, but in MacRuby one of the main reasons that lack-of-GIL is not a more serious problem is because MacRuby has the Dispatch library (based on libdispatch, a.k.a. Grand Central Dispatch) which makes working with multiple threads safe again. If you were to invoke Ruby's Thread library directly in MacRuby, you would find that things get crashy rather quickly!
- regularfry 15y ago
  We run heavily-threaded code in jruby in production, but we also test on MRI because it detects and explicitly fails on deadlocks rather than just locking up.

82 points by telemachos 15y ago

jbert 15y ago

With most web architectures, your state is outside the process (RDBMS, NoSQL, filesystem, whatever persistent store you're using). So the main benefit using threads (easy sharing of in-process state) goes away. For web architectures, concurrent request handling seems to me to be best achieved by multi-process, rather than multi-thread. There are some minor benefits to threading (e.g. an in-memory cache of global state can be shared amongst multiple threads, rather than being replicated among multiple processes) but: - there are out-of-process cacheing technologies (memcached, redis etc) - threading doesn't come "for free", in that you need to pay a performance cost in terms of locking in your interpreter and/or a code complexity cost in terms of access to your shared data structures.

wladimir 15y ago
Also, don't forget that threads don't scale beyond one machine; so when you want expand beyond one machine you need to take care of two levels of scaling, threads and processes. This can be avoided when using multiple processes that communicate or share state some other way in the first place. Even with one CPU, with many cores the internal synchronization that happens to "simulate" a shared memory space can be expensive, if you're not careful your cores get clogged ping-ponging memory pages between each other. It might sound more convenient to use threads instead of processes, but in the end I'm not sure that all the work to remove the GIL (and introduce shitloads of finer-grained synchronization primitives) is worth it.
- bad_user 15y ago
  Threads are important not because you want to scale horizontally, but because you want to also scale vertically, as in having an optimum ratio of performance per Watts or CPU cores or RAM MB. Threads are much, much more efficient than processes - threads consume less memory (even with COW, VMs with garbage-collectors basically prevent COW from being effective), threads have faster context-switch, and threads achieve better cache-locality. And shared memory, which is considered the plague of the software industry along with pointers to memory, is actually a mirror of our current hardware architecture. The fact is that hard-disks and SSDs are much slower than RAM, RAM memory is much slower than L2 cache, which is slower than L1 cache, which is slower than when registers get accessed. You achieve good performance characteristics when you keep your data in one place, keeping your most accessed items in the CPU's cache, or at least prevent those items from being stored in swap. Food for thought: the NoSQL solutions so loved by Rails or Node.js developers are mostly written in C/C++ and rely on Posix threads, with some Erlang here and there. might sound more convenient to use threads instead of processes That's because it IS more convenient to use threads instead of processes in most cases. You can mostly get away with it only when the processes don't need to synchronize (i.e. your workload can be processed fully in parallel), but when processes do need to synchronize lots of bad shit can happen, as processes are more unreliable than threads. Of course, I'm referring to real kernel-managed POSIX threads and processes, not what Erlang does, but then again, the Erlang's light-weight processes are just an abstraction over POSIX threads, not processes as that would have been dumb.
  - wladimir 15y ago
    Threads are much, much more efficient than processes - threads consume less memory (even with COW, VMs with garbage-collectors basically prevent COW from being effective), threads have faster context-switch, and threads achieve better cache-locality. There is hardly any "context switch" if you run one process per core (compared to one thread per core, it makes no difference). And yes, you'll save memory if you do everything in one process (even with COW), but don't forget garbage collection with multiple threads is a much more complex beast than single-threaded, resulting in extra overhead. I'm not denying that threads have some advantages, for example, when parallelizing heavy number-crunching work, but the point I was trying to make is that "they are much more efficient" is not that clear-cut and universally true. That's because it IS more convenient to use threads instead of processes in most cases Not necessarily true either. When using threads there is a much larger chance of introducing race conditions and deadlocks and other synchronization fails. It might look more convenient when writing the code, but debugging and supporting it certainly isn't.
    - buff-a 15y ago
      There is hardly any "context switch" if you run one process per core There is hardly any context switch if I don't run any software at all on my server. Zero bugs either. Back to the real world, and the point of this discussion, we run multiple Ruby processes per CPU so that when one process is waiting for the db or memcache, another can be doing useful work on the same CPU. When using threads there is a much larger chance of introducing race conditions and deadlocks and other synchronization fails As opposed to dropping the problem into memcache and getting it wrong: your code "works", you get no crashes, but customers lose data, lose posts, or lose money. If you're advocating using memcache because multithreading is too hard for your programmer, then your customers are fucked.
      - wladimir 15y ago
        We run multiple Ruby processes per CPU so that when one process is waiting for the db or memcache, another can be doing useful work on the same CPU That's the trivial case. When waiting for the db or memcache (I/O in general), the GIL is lifted, so it doesn't get in the way. It only gets in the way if multiple threads are inside the ruby VM doing work.
buff-a 15y ago
threading doesn't come "for free", in that you need to pay a performance cost in terms of locking in your interpreter and/or a code complexity cost in terms of access to your shared data structures vs there are out-of-process cacheing technologies (memcached, redis etc) If you subject "out-of-process cacheing technologies" to the same measurements (performance cost, code complexity), as threading solutions, what do you find?
- simonw 15y ago
  That they're massively easier to work with?
- jbert 15y ago
  > If you subject "out-of-process cacheing technologies" to the same measurements (performance cost, code complexity), as threading solutions, what do you find? Yes, I think you find that shared state is difficult. So it's the old process (private by default, shared as the exception) versus threads (shared by default) split. imho, "less shared state" == "simpler". And imho, "private by default" == "less shared state". And you have an easier time of it the shared state helps you with concurrent access. This is one benefit of RDBMSs (transactions) and also systems like redis. The fact that your state is external means it is more likely to have been designed to be conncurrency-safe.
  - antrix 15y ago
    I don't get this argument. If I replace my memcache logic with an in-memory thread-safe hashmap, it will always be faster. I can't see how it can be slow. As for 'simpler', as long as you aren't actually implementing the thread safe map, it is simpler than using memcache too. So as long as you use well written shared state implementations, (e.g. Java's concurrent collections), shared state isn't as hard as you make it out to be.
    - jbert 15y ago
      > If I replace my memcache logic with an in-memory thread-safe hashmap, it will always be faster. I can't see how it can be slow. Because in a threaded application context, you have to either: - have locking/synchronization on all data structures (performance cost) OR - manage which of your data structures are shared and which are not (complexity cost/race bugs) With external, shared state (RDBMs, memcached, etc) you get fast in-process access to your (private) data structures (no performance cost due to locking) and a (mostly) concurrency-safe, explicit datastore. And as pointed out elsewhere, to scale beyond a single process you need the external state anyway. There are other approaches to this, software transactional memory, functional programming, clojure paradigm etc. But it's a hard problem and it's not clear these are the right solution at scale.
jerf 15y ago
"With most web architectures, your state is outside the process (RDBMS, NoSQL, filesystem, whatever persistent store you're using)." I submit the fact that is nearly 100% is effect, not cause. If web technologies weren't so bad at handling multiple cores, you'd be more likely to move your web apps up to where ever the "real" computation is taking place, so as to remove a link in the chain. But since that is almost entirely not an option, nobody does it. It doesn't prove it's the "right" way to do it if it's the only way to do it right now. Using Erlang and Haskell tech, I've actually had some great fun writing applications that are heavily multithreaded and happen to include a web server too, built in. It isn't the solution to every problem, but damn it's fun and nice not to spend so much time marshalling things over a database gap for what is basically no reason. The database goes back to just being persistence and querying, instead of communication as well.
- jbert 15y ago
  > I submit the fact that is nearly 100% is effect, not cause. Well, there's also the point that your shared state needs to be process-external as soon as you want to scale beyond a single process. (e.g. need to run on more than one server). Yes - most sites/apps don't need that, but given that a lot of the web site framework and tooling is set up to make this relatively easy to do, it's nice to not have to rework your app fairly fundamentally once you hit that scaling wall.
  - gcampbell 15y ago
    Depending on the application, there are some options for process-internal multi-machine shared state - Erlang's mnesia distributed database, for instance.
    - jbert 15y ago
      That's interesting, thanks.

nupark2 15y ago

I'm really tired of these disingenuous justifications and arguments to 'just use processes!' A GIL is an unnecessary limitation that precludes a huge swath of architecture optimizations. It's there because it's difficult to remove once you've made the incorrect choice to rely on it, not because a GIL is a good idea.

cbs 15y ago
This is very true. It reminds me of the BKL, yeah we could technically live with it, but it would be lazy and bad practice for a platform. Others in this thread are saying that GILs are OK because they're working on web applications. Not everyone shares your narrow requirements when using a general purpose language.
- jbert 15y ago
  > Not everyone shares your narrow requirements when using a general purpose language. IMHO, if the resource difference between processes and threads is significant for your workload then moving away from an interpreted language is probably a step to take before going threaded. i.e. remove the overhead of the interpreter, before you worry about overhead of a GIL. Also note that a GIL gives better performance on a single-threaded workload, since you avoid the overhead of fine-grained locking. So, asking for GIL removal is asking all users of the language to pay a 'threading tax', when it isn't at all clear that there would be any real benificiaries (since the above argument can be made that those workloads would be better in a different language). I'm not rabidly anti-thread, but I think it is important to note that there are good reasons (not just laziness) why you might not want the GIL removed - i.e. I don't think it's as simple as being bad practice.
  - cbs 15y ago
    >remove the overhead of the interpreter, before you worry about overhead of a GIL. The overhead of the GIL killing parallelism is nontrivial, even next to the interpreter overhead >(since the above argument can be made that those workloads would be better in a different language). Thats a pretty self defeating argument. Every single possible ruby workload would perform better in another language. >So, asking for GIL removal is asking all users of the language to pay a 'threading tax', First, the existence of the multithreading support already has a threading tax, the GIL. Second, sprinkling in the use of some more mutexes wouldn't make a significant performance impact, especially not in a interpreted language that we already agree is not good for performance. Of course, if the runtime knew it was only running one thread it could ignore much the synchronization code altogether, but we don't want to pay the tax of a single branch decision, do we? >IMHO, if the resource difference between processes and threads is significant [...] What about all situations where you want to use threads, but not due to resource constraints? I would assume your answer would be "don't use threads" but you say you're not rabidly anti-thread.

drnicwilliams 15y ago

Three quotes, hopefully not out of useful context, that seem incongruous: "Rubinius is about the join JRuby and MacRuby in the realm of GIL-less Ruby implementations" "I spend my free time working on an alternative Ruby implementation which doesn’t use a GIL (MacRuby)" "I respect Matz’ decision to keep the GIL even though, I would personally prefer to push the data safety responsibility to the developers. However, I do know that many Ruby developers would end up shooting themselves in the foot" So developers using GIL-less MacRuby, JRuby and Rubinius are prone to foot shooting? I wish they'd blog about this more, I've never once heard a MacRuby or JRuby developer blog saying "I went back to MRI because I needed my Ruby code to be run more safely".

jballanc 15y ago
I can't speak for JRuby or Rubinius, but in MacRuby one of the main reasons that lack-of-GIL is not a more serious problem is because MacRuby has the Dispatch library (based on libdispatch, a.k.a. Grand Central Dispatch) which makes working with multiple threads safe again. If you were to invoke Ruby's Thread library directly in MacRuby, you would find that things get crashy rather quickly!
regularfry 15y ago
We run heavily-threaded code in jruby in production, but we also test on MRI because it detects and explicitly fails on deadlocks rather than just locking up.