5 ms·

I keep hearing assertions that threaded can be slow or does not scale. Can somebody describe the reasons here or point to a good document on the subject? I oft

I keep hearing assertions that threaded can be slow or does not scale. Can somebody describe the reasons here or point to a good document on the subject? I often see "scalable" and "fast" used interchangeably in these discussions. These are different concepts. Is the contention that threading is not as scalable as evented, not as fast as evented, or both?

bnoordhuis 16y ago
Both threaded and event-driven I/O are scalable if you throw enough hardware at it and - all other things being equal - are about as fast. The one-thread-per-connection model is conceptually simpler at the cost of a higher resource footprint, the event-driven approach adds complexity but allows you to process more concurrent connections. Overly broad and simplified but that's about the gist of it. Like everything non-trivial, the real answer is: It depends.
- frognibble 16y ago
  Threaded does not imply one thread per connection. As an example, Java Servlet 3.0 allows a request handler to suspend and resume execution of a request handler on a thread. This feature allows the application to relinquish the thread for the occasional long operation and use blocking threaded code for most other code. Threaded applications written in this style can support a large number of concurrent connections with modest resource use and no callback spaghetti. Models like Java Servlet 3.0 allow threaded servers to handle a large number of concurrent connections. Are there other reasons why threaded servlets are slow or not scalable? Most web applications don't have a large number of concurrent connections because they application request handlers do not block for a long time. The resource use for thread stacks is very modest for these applications even without a suspend & resume feature.
  - Tichy 16y ago
    I don't think the Servlet 3.0 example counts as an example for thread based request handling to be working. After all, it introduces event based request handling to circumvent the problems of threaded event handling.
    - frognibble 16y ago
      Because Servlet 3.0 does not dictate how an application suspends or resumes request handling, I wouldn't say that Servlet 3.0 introduces event based request handling (other than the first invocation of the handler as in Servlet 1.0). I guess one can claim that any suspend and resume logic in the application is a form of eventing, but it's different from node.js in that it's not pervasive through the application. It's also different from node.js in that the server infrastructure is not involved in the "events".
_delirium 16y ago
One of the traditional threads-don't-scale arguments, which led to the early-2000s wave of evented competitors to Apache (lighthttpd, nginx), was that threads have per-thread call-stack overhead that's hard to minimize without either OS support for dynamically sized call stacks, or choosing really small static stack sizes that make writing code that doesn't stack-overflow tricky. A different issue was that OSs used to be very bad at scheduling large numbers of threads--- if you spawned 10,000 threads, the system scheduler blew up. I believe that's mostly much improved these days. Most of the threads-don't-scale arguments also assume OS threads; user-level threads are another story.
Tichy 16y ago
I think traditionally, something like Tomcat used to have perhaps 30 threads? Especially with todays Ajax-driven web sites, it is very easy to exhaust them. I can only point to a stupid app of mine, which triggered Twitter searches for a list of search results via Ajax (so one page of search results would trigger, say, 10 search reguests to Twitter via Ajax, proxied through the server). Since that app was written for a competition and non-commercial, I hosted it on the free Heroku plan (it was a Rails app). That is how I found out that apparently that Heroku plan has only one process, and Rails doesn't use threads. So not only would the search requests to Twitter only be handled sequentially, while those threads were running, no other visitors would get to use the site (the 10 Ajax requests would already use up up to 10 threads, and I had only one). Just a stupid example - with event based request handling, it would not have been an issue because the search requests to Twitter would not have blocked everything else.
- dedward 16y ago
  But a request to twitter blocking everything else would be an issue, right? Aside: haproxy, configured cleverly, can deal with limiting the number of concurrent connections permitted to your app on a URL basis (or just about any other part of the request you can deconstruct) to allow your app to not get hung up on slow queries and keep fast queries going where you want them.
  - Tichy 16y ago
    With something like node, a request to Twitter would not block. Not sure how haproxy would help?
- epochwolf 16y ago
  Uh... rails does thread. Not sure if it's a good idea but it does have support for threads. http://guides.rubyonrails.org/2_2_release_notes.html#thread-safety http://guides.rubyonrails.org/2_2_release_notes.html#thread-...
  - Tichy 16y ago
    OK, but I don't think the traditional hosting setups use that. Anyway, the point was that with a limited amount of threads, you can run into trouble.
dedward 16y ago
Seems like kind of apples-to-oranges to me as well.... Threaded, done right (which isn't always easy) - can scale on a single machine to use up all the cores you can throw at it. This can be fairly optimal on the machine in question, depending on the app. Traditionally when dealing with threaded programming you have a whole lot of locking and whatnot to keep track of - and things can get ugly fast. It's HARD. And all that hard work, while it might help you scale up to the biggest multi-processor beast you can find - doesn't help you one bit when it comes to scaling horizontally to multiple nodes..... I'd wager, depending on the situation, that it gets right up in your face and in the way. But tha'ts back to comparing apples and oranges. "scalable" is about overall design - what needs to talk to what, how will it scale, will it be a purely shared-nothing environment like erlang where we're passing messages and can scale out horizontally like mad, but where every function call is effectively a full memory copy, possibly over a network? Or do we nail it down to a precisely crafted C++ threaded app that our expert designers know down to it's finest detail, that's faster than sh*t througha goose, and hey, it uses some evented stuff as well. It seems the real rise interest in the event-driven nature of node.js is that it makes it obvious to even beginner programmers where the difference is - you are setting up some actions, and letting the OS kernel handle listening for the events for you in the fastest possible way. You could do all that on your own in another language, but here it is all neatly wrapped up, in a language you already know.
rektide 16y ago
Threaded model has two detractors; Foremost, context switching. This is a non-trivial process: on-die caches get flushed, virtual memory has to get re-mapped, registers re-loaded. This is by definition lost time, time the OS is spent not doing your workload. If the number of context switches gets too large, a system is spending an inordinate amount of time performing context switches, and less time doing "real work" inside threads. You cite two isues, slow and not scaling. Second, memory usage. Each thread has its own stack space, which consumes a fixed memory size. As to "slow" and "does not scale;" slow is relative; the amount of time spent context switching is very dependent on a variety of workload factors. Threading _can_ be slow. Scaling, on the other hand, is less unequivocal. A quarter million active threads, each doing very little, is not a good thing: too much memory, too many context switches, and too high a latency. The problem is not threading per se. It's pre-emptive multi-threading, where a programs state is transparently pulled from the CPU by the OS and serialized to someplace else, and another programs state is de-serialized and loaded in transparent; that is, without the loaded or unloaded program knowing. Erlang and tasklets are examples of userspace or lightweighs threads where pre-emption is not permitted, and the context switching penalty is negligible. Instead of threads being ignorant, and being preemptively swapped in and out, they are coordinated and actively yield themselves back to the scheduler, thereby reducing the context switching penalty.

bnoordhuis 16y ago

Both threaded and event-driven I/O are scalable if you throw enough hardware at it and - all other things being equal - are about as fast. The one-thread-per-connection model is conceptually simpler at the cost of a higher resource footprint, the event-driven approach adds complexity but allows you to process more concurrent connections. Overly broad and simplified but that's about the gist of it. Like everything non-trivial, the real answer is: It depends.

frognibble 16y ago
Threaded does not imply one thread per connection. As an example, Java Servlet 3.0 allows a request handler to suspend and resume execution of a request handler on a thread. This feature allows the application to relinquish the thread for the occasional long operation and use blocking threaded code for most other code. Threaded applications written in this style can support a large number of concurrent connections with modest resource use and no callback spaghetti. Models like Java Servlet 3.0 allow threaded servers to handle a large number of concurrent connections. Are there other reasons why threaded servlets are slow or not scalable? Most web applications don't have a large number of concurrent connections because they application request handlers do not block for a long time. The resource use for thread stacks is very modest for these applications even without a suspend & resume feature.
- Tichy 16y ago
  I don't think the Servlet 3.0 example counts as an example for thread based request handling to be working. After all, it introduces event based request handling to circumvent the problems of threaded event handling.
  - frognibble 16y ago
    Because Servlet 3.0 does not dictate how an application suspends or resumes request handling, I wouldn't say that Servlet 3.0 introduces event based request handling (other than the first invocation of the handler as in Servlet 1.0). I guess one can claim that any suspend and resume logic in the application is a form of eventing, but it's different from node.js in that it's not pervasive through the application. It's also different from node.js in that the server infrastructure is not involved in the "events".

_delirium 16y ago

One of the traditional threads-don't-scale arguments, which led to the early-2000s wave of evented competitors to Apache (lighthttpd, nginx), was that threads have per-thread call-stack overhead that's hard to minimize without either OS support for dynamically sized call stacks, or choosing really small static stack sizes that make writing code that doesn't stack-overflow tricky. A different issue was that OSs used to be very bad at scheduling large numbers of threads--- if you spawned 10,000 threads, the system scheduler blew up. I believe that's mostly much improved these days. Most of the threads-don't-scale arguments also assume OS threads; user-level threads are another story.

Tichy 16y ago

I think traditionally, something like Tomcat used to have perhaps 30 threads? Especially with todays Ajax-driven web sites, it is very easy to exhaust them. I can only point to a stupid app of mine, which triggered Twitter searches for a list of search results via Ajax (so one page of search results would trigger, say, 10 search reguests to Twitter via Ajax, proxied through the server). Since that app was written for a competition and non-commercial, I hosted it on the free Heroku plan (it was a Rails app). That is how I found out that apparently that Heroku plan has only one process, and Rails doesn't use threads. So not only would the search requests to Twitter only be handled sequentially, while those threads were running, no other visitors would get to use the site (the 10 Ajax requests would already use up up to 10 threads, and I had only one). Just a stupid example - with event based request handling, it would not have been an issue because the search requests to Twitter would not have blocked everything else.

dedward 16y ago
But a request to twitter blocking everything else would be an issue, right? Aside: haproxy, configured cleverly, can deal with limiting the number of concurrent connections permitted to your app on a URL basis (or just about any other part of the request you can deconstruct) to allow your app to not get hung up on slow queries and keep fast queries going where you want them.
- Tichy 16y ago
  With something like node, a request to Twitter would not block. Not sure how haproxy would help?
epochwolf 16y ago
Uh... rails does thread. Not sure if it's a good idea but it does have support for threads. http://guides.rubyonrails.org/2_2_release_notes.html#thread-safety http://guides.rubyonrails.org/2_2_release_notes.html#thread-...
- Tichy 16y ago
  OK, but I don't think the traditional hosting setups use that. Anyway, the point was that with a limited amount of threads, you can run into trouble.

dedward 16y ago

Seems like kind of apples-to-oranges to me as well.... Threaded, done right (which isn't always easy) - can scale on a single machine to use up all the cores you can throw at it. This can be fairly optimal on the machine in question, depending on the app. Traditionally when dealing with threaded programming you have a whole lot of locking and whatnot to keep track of - and things can get ugly fast. It's HARD. And all that hard work, while it might help you scale up to the biggest multi-processor beast you can find - doesn't help you one bit when it comes to scaling horizontally to multiple nodes..... I'd wager, depending on the situation, that it gets right up in your face and in the way. But tha'ts back to comparing apples and oranges. "scalable" is about overall design - what needs to talk to what, how will it scale, will it be a purely shared-nothing environment like erlang where we're passing messages and can scale out horizontally like mad, but where every function call is effectively a full memory copy, possibly over a network? Or do we nail it down to a precisely crafted C++ threaded app that our expert designers know down to it's finest detail, that's faster than sh*t througha goose, and hey, it uses some evented stuff as well. It seems the real rise interest in the event-driven nature of node.js is that it makes it obvious to even beginner programmers where the difference is - you are setting up some actions, and letting the OS kernel handle listening for the events for you in the fastest possible way. You could do all that on your own in another language, but here it is all neatly wrapped up, in a language you already know.

rektide 16y ago

Threaded model has two detractors; Foremost, context switching. This is a non-trivial process: on-die caches get flushed, virtual memory has to get re-mapped, registers re-loaded. This is by definition lost time, time the OS is spent not doing your workload. If the number of context switches gets too large, a system is spending an inordinate amount of time performing context switches, and less time doing "real work" inside threads. You cite two isues, slow and not scaling. Second, memory usage. Each thread has its own stack space, which consumes a fixed memory size. As to "slow" and "does not scale;" slow is relative; the amount of time spent context switching is very dependent on a variety of workload factors. Threading _can_ be slow. Scaling, on the other hand, is less unequivocal. A quarter million active threads, each doing very little, is not a good thing: too much memory, too many context switches, and too high a latency. The problem is not threading per se. It's pre-emptive multi-threading, where a programs state is transparently pulled from the CPU by the OS and serialized to someplace else, and another programs state is de-serialized and loaded in transparent; that is, without the loaded or unloaded program knowing. Erlang and tasklets are examples of userspace or lightweighs threads where pre-emption is not permitted, and the context switching penalty is negligible. Instead of threads being ignorant, and being preemptively swapped in and out, they are coordinated and actively yield themselves back to the scheduler, thereby reducing the context switching penalty.