6 ms·

With most web architectures, your state is outside the process (RDBMS, NoSQL, filesystem, whatever persistent store you're using). So the main benefit using th

by jbert 15y ago

With most web architectures, your state is outside the process (RDBMS, NoSQL, filesystem, whatever persistent store you're using). So the main benefit using threads (easy sharing of in-process state) goes away. For web architectures, concurrent request handling seems to me to be best achieved by multi-process, rather than multi-thread. There are some minor benefits to threading (e.g. an in-memory cache of global state can be shared amongst multiple threads, rather than being replicated among multiple processes) but: - there are out-of-process cacheing technologies (memcached, redis etc) - threading doesn't come "for free", in that you need to pay a performance cost in terms of locking in your interpreter and/or a code complexity cost in terms of access to your shared data structures.

wladimir 15y ago
Also, don't forget that threads don't scale beyond one machine; so when you want expand beyond one machine you need to take care of two levels of scaling, threads and processes. This can be avoided when using multiple processes that communicate or share state some other way in the first place. Even with one CPU, with many cores the internal synchronization that happens to "simulate" a shared memory space can be expensive, if you're not careful your cores get clogged ping-ponging memory pages between each other. It might sound more convenient to use threads instead of processes, but in the end I'm not sure that all the work to remove the GIL (and introduce shitloads of finer-grained synchronization primitives) is worth it.
- bad_user 15y ago
  Threads are important not because you want to scale horizontally, but because you want to also scale vertically, as in having an optimum ratio of performance per Watts or CPU cores or RAM MB. Threads are much, much more efficient than processes - threads consume less memory (even with COW, VMs with garbage-collectors basically prevent COW from being effective), threads have faster context-switch, and threads achieve better cache-locality. And shared memory, which is considered the plague of the software industry along with pointers to memory, is actually a mirror of our current hardware architecture. The fact is that hard-disks and SSDs are much slower than RAM, RAM memory is much slower than L2 cache, which is slower than L1 cache, which is slower than when registers get accessed. You achieve good performance characteristics when you keep your data in one place, keeping your most accessed items in the CPU's cache, or at least prevent those items from being stored in swap. Food for thought: the NoSQL solutions so loved by Rails or Node.js developers are mostly written in C/C++ and rely on Posix threads, with some Erlang here and there. might sound more convenient to use threads instead of processes That's because it IS more convenient to use threads instead of processes in most cases. You can mostly get away with it only when the processes don't need to synchronize (i.e. your workload can be processed fully in parallel), but when processes do need to synchronize lots of bad shit can happen, as processes are more unreliable than threads. Of course, I'm referring to real kernel-managed POSIX threads and processes, not what Erlang does, but then again, the Erlang's light-weight processes are just an abstraction over POSIX threads, not processes as that would have been dumb.
  - wladimir 15y ago
    Threads are much, much more efficient than processes - threads consume less memory (even with COW, VMs with garbage-collectors basically prevent COW from being effective), threads have faster context-switch, and threads achieve better cache-locality. There is hardly any "context switch" if you run one process per core (compared to one thread per core, it makes no difference). And yes, you'll save memory if you do everything in one process (even with COW), but don't forget garbage collection with multiple threads is a much more complex beast than single-threaded, resulting in extra overhead. I'm not denying that threads have some advantages, for example, when parallelizing heavy number-crunching work, but the point I was trying to make is that "they are much more efficient" is not that clear-cut and universally true. That's because it IS more convenient to use threads instead of processes in most cases Not necessarily true either. When using threads there is a much larger chance of introducing race conditions and deadlocks and other synchronization fails. It might look more convenient when writing the code, but debugging and supporting it certainly isn't.
    - buff-a 15y ago
      There is hardly any "context switch" if you run one process per core There is hardly any context switch if I don't run any software at all on my server. Zero bugs either. Back to the real world, and the point of this discussion, we run multiple Ruby processes per CPU so that when one process is waiting for the db or memcache, another can be doing useful work on the same CPU. When using threads there is a much larger chance of introducing race conditions and deadlocks and other synchronization fails As opposed to dropping the problem into memcache and getting it wrong: your code "works", you get no crashes, but customers lose data, lose posts, or lose money. If you're advocating using memcache because multithreading is too hard for your programmer, then your customers are fucked.
      - wladimir 15y ago
        We run multiple Ruby processes per CPU so that when one process is waiting for the db or memcache, another can be doing useful work on the same CPU That's the trivial case. When waiting for the db or memcache (I/O in general), the GIL is lifted, so it doesn't get in the way. It only gets in the way if multiple threads are inside the ruby VM doing work.
buff-a 15y ago
threading doesn't come "for free", in that you need to pay a performance cost in terms of locking in your interpreter and/or a code complexity cost in terms of access to your shared data structures vs there are out-of-process cacheing technologies (memcached, redis etc) If you subject "out-of-process cacheing technologies" to the same measurements (performance cost, code complexity), as threading solutions, what do you find?
- simonw 15y ago
  That they're massively easier to work with?
- jbert 15y ago
  > If you subject "out-of-process cacheing technologies" to the same measurements (performance cost, code complexity), as threading solutions, what do you find? Yes, I think you find that shared state is difficult. So it's the old process (private by default, shared as the exception) versus threads (shared by default) split. imho, "less shared state" == "simpler". And imho, "private by default" == "less shared state". And you have an easier time of it the shared state helps you with concurrent access. This is one benefit of RDBMSs (transactions) and also systems like redis. The fact that your state is external means it is more likely to have been designed to be conncurrency-safe.
  - antrix 15y ago
    I don't get this argument. If I replace my memcache logic with an in-memory thread-safe hashmap, it will always be faster. I can't see how it can be slow. As for 'simpler', as long as you aren't actually implementing the thread safe map, it is simpler than using memcache too. So as long as you use well written shared state implementations, (e.g. Java's concurrent collections), shared state isn't as hard as you make it out to be.
    - jbert 15y ago
      > If I replace my memcache logic with an in-memory thread-safe hashmap, it will always be faster. I can't see how it can be slow. Because in a threaded application context, you have to either: - have locking/synchronization on all data structures (performance cost) OR - manage which of your data structures are shared and which are not (complexity cost/race bugs) With external, shared state (RDBMs, memcached, etc) you get fast in-process access to your (private) data structures (no performance cost due to locking) and a (mostly) concurrency-safe, explicit datastore. And as pointed out elsewhere, to scale beyond a single process you need the external state anyway. There are other approaches to this, software transactional memory, functional programming, clojure paradigm etc. But it's a hard problem and it's not clear these are the right solution at scale.
jerf 15y ago
"With most web architectures, your state is outside the process (RDBMS, NoSQL, filesystem, whatever persistent store you're using)." I submit the fact that is nearly 100% is effect, not cause. If web technologies weren't so bad at handling multiple cores, you'd be more likely to move your web apps up to where ever the "real" computation is taking place, so as to remove a link in the chain. But since that is almost entirely not an option, nobody does it. It doesn't prove it's the "right" way to do it if it's the only way to do it right now. Using Erlang and Haskell tech, I've actually had some great fun writing applications that are heavily multithreaded and happen to include a web server too, built in. It isn't the solution to every problem, but damn it's fun and nice not to spend so much time marshalling things over a database gap for what is basically no reason. The database goes back to just being persistence and querying, instead of communication as well.
- jbert 15y ago
  > I submit the fact that is nearly 100% is effect, not cause. Well, there's also the point that your shared state needs to be process-external as soon as you want to scale beyond a single process. (e.g. need to run on more than one server). Yes - most sites/apps don't need that, but given that a lot of the web site framework and tooling is set up to make this relatively easy to do, it's nice to not have to rework your app fairly fundamentally once you hit that scaling wall.
  - gcampbell 15y ago
    Depending on the application, there are some options for process-internal multi-machine shared state - Erlang's mnesia distributed database, for instance.
    - jbert 15y ago
      That's interesting, thanks.

by jbert 15y ago

wladimir 15y ago

Also, don't forget that threads don't scale beyond one machine; so when you want expand beyond one machine you need to take care of two levels of scaling, threads and processes. This can be avoided when using multiple processes that communicate or share state some other way in the first place. Even with one CPU, with many cores the internal synchronization that happens to "simulate" a shared memory space can be expensive, if you're not careful your cores get clogged ping-ponging memory pages between each other. It might sound more convenient to use threads instead of processes, but in the end I'm not sure that all the work to remove the GIL (and introduce shitloads of finer-grained synchronization primitives) is worth it.

bad_user 15y ago
Threads are important not because you want to scale horizontally, but because you want to also scale vertically, as in having an optimum ratio of performance per Watts or CPU cores or RAM MB. Threads are much, much more efficient than processes - threads consume less memory (even with COW, VMs with garbage-collectors basically prevent COW from being effective), threads have faster context-switch, and threads achieve better cache-locality. And shared memory, which is considered the plague of the software industry along with pointers to memory, is actually a mirror of our current hardware architecture. The fact is that hard-disks and SSDs are much slower than RAM, RAM memory is much slower than L2 cache, which is slower than L1 cache, which is slower than when registers get accessed. You achieve good performance characteristics when you keep your data in one place, keeping your most accessed items in the CPU's cache, or at least prevent those items from being stored in swap. Food for thought: the NoSQL solutions so loved by Rails or Node.js developers are mostly written in C/C++ and rely on Posix threads, with some Erlang here and there. might sound more convenient to use threads instead of processes That's because it IS more convenient to use threads instead of processes in most cases. You can mostly get away with it only when the processes don't need to synchronize (i.e. your workload can be processed fully in parallel), but when processes do need to synchronize lots of bad shit can happen, as processes are more unreliable than threads. Of course, I'm referring to real kernel-managed POSIX threads and processes, not what Erlang does, but then again, the Erlang's light-weight processes are just an abstraction over POSIX threads, not processes as that would have been dumb.
- wladimir 15y ago
  Threads are much, much more efficient than processes - threads consume less memory (even with COW, VMs with garbage-collectors basically prevent COW from being effective), threads have faster context-switch, and threads achieve better cache-locality. There is hardly any "context switch" if you run one process per core (compared to one thread per core, it makes no difference). And yes, you'll save memory if you do everything in one process (even with COW), but don't forget garbage collection with multiple threads is a much more complex beast than single-threaded, resulting in extra overhead. I'm not denying that threads have some advantages, for example, when parallelizing heavy number-crunching work, but the point I was trying to make is that "they are much more efficient" is not that clear-cut and universally true. That's because it IS more convenient to use threads instead of processes in most cases Not necessarily true either. When using threads there is a much larger chance of introducing race conditions and deadlocks and other synchronization fails. It might look more convenient when writing the code, but debugging and supporting it certainly isn't.
  - buff-a 15y ago
    There is hardly any "context switch" if you run one process per core There is hardly any context switch if I don't run any software at all on my server. Zero bugs either. Back to the real world, and the point of this discussion, we run multiple Ruby processes per CPU so that when one process is waiting for the db or memcache, another can be doing useful work on the same CPU. When using threads there is a much larger chance of introducing race conditions and deadlocks and other synchronization fails As opposed to dropping the problem into memcache and getting it wrong: your code "works", you get no crashes, but customers lose data, lose posts, or lose money. If you're advocating using memcache because multithreading is too hard for your programmer, then your customers are fucked.
    - wladimir 15y ago
      We run multiple Ruby processes per CPU so that when one process is waiting for the db or memcache, another can be doing useful work on the same CPU That's the trivial case. When waiting for the db or memcache (I/O in general), the GIL is lifted, so it doesn't get in the way. It only gets in the way if multiple threads are inside the ruby VM doing work.

buff-a 15y ago

threading doesn't come "for free", in that you need to pay a performance cost in terms of locking in your interpreter and/or a code complexity cost in terms of access to your shared data structures vs there are out-of-process cacheing technologies (memcached, redis etc) If you subject "out-of-process cacheing technologies" to the same measurements (performance cost, code complexity), as threading solutions, what do you find?

simonw 15y ago
That they're massively easier to work with?
jbert 15y ago
> If you subject "out-of-process cacheing technologies" to the same measurements (performance cost, code complexity), as threading solutions, what do you find? Yes, I think you find that shared state is difficult. So it's the old process (private by default, shared as the exception) versus threads (shared by default) split. imho, "less shared state" == "simpler". And imho, "private by default" == "less shared state". And you have an easier time of it the shared state helps you with concurrent access. This is one benefit of RDBMSs (transactions) and also systems like redis. The fact that your state is external means it is more likely to have been designed to be conncurrency-safe.
- antrix 15y ago
  I don't get this argument. If I replace my memcache logic with an in-memory thread-safe hashmap, it will always be faster. I can't see how it can be slow. As for 'simpler', as long as you aren't actually implementing the thread safe map, it is simpler than using memcache too. So as long as you use well written shared state implementations, (e.g. Java's concurrent collections), shared state isn't as hard as you make it out to be.
  - jbert 15y ago
    > If I replace my memcache logic with an in-memory thread-safe hashmap, it will always be faster. I can't see how it can be slow. Because in a threaded application context, you have to either: - have locking/synchronization on all data structures (performance cost) OR - manage which of your data structures are shared and which are not (complexity cost/race bugs) With external, shared state (RDBMs, memcached, etc) you get fast in-process access to your (private) data structures (no performance cost due to locking) and a (mostly) concurrency-safe, explicit datastore. And as pointed out elsewhere, to scale beyond a single process you need the external state anyway. There are other approaches to this, software transactional memory, functional programming, clojure paradigm etc. But it's a hard problem and it's not clear these are the right solution at scale.

jerf 15y ago

"With most web architectures, your state is outside the process (RDBMS, NoSQL, filesystem, whatever persistent store you're using)." I submit the fact that is nearly 100% is effect, not cause. If web technologies weren't so bad at handling multiple cores, you'd be more likely to move your web apps up to where ever the "real" computation is taking place, so as to remove a link in the chain. But since that is almost entirely not an option, nobody does it. It doesn't prove it's the "right" way to do it if it's the only way to do it right now. Using Erlang and Haskell tech, I've actually had some great fun writing applications that are heavily multithreaded and happen to include a web server too, built in. It isn't the solution to every problem, but damn it's fun and nice not to spend so much time marshalling things over a database gap for what is basically no reason. The database goes back to just being persistence and querying, instead of communication as well.

jbert 15y ago
> I submit the fact that is nearly 100% is effect, not cause. Well, there's also the point that your shared state needs to be process-external as soon as you want to scale beyond a single process. (e.g. need to run on more than one server). Yes - most sites/apps don't need that, but given that a lot of the web site framework and tooling is set up to make this relatively easy to do, it's nice to not have to rework your app fairly fundamentally once you hit that scaling wall.
- gcampbell 15y ago
  Depending on the application, there are some options for process-internal multi-machine shared state - Erlang's mnesia distributed database, for instance.
  - jbert 15y ago
    That's interesting, thanks.