5 ms·

> Lets say you have 20 threads. You improved performance 20 times, so the rate is now 4 request per second. Still, way too small. You can keep throwing threads

by ZoFreX 16y ago

> Lets say you have 20 threads. You improved performance 20 times, so the rate is now 4 request per second. Still, way too small. You can keep throwing threads at the problem, but threads are expensive in terms of memory usage and scheduling. I doubt you’ll ever reach hundreds of requests per second this way. Let's say I'm using Java instead of Python. Lets say I use a lot more threads than 20. I will reach thousands of requests per second this way. There are real problems with that approach, and event-driven approaches have genuine advantages, but can people please stop straight-up lying and saying that you can't just throw a few thousand threads at a problem, because you can, and people do.

wildmXranat 16y ago
I agree with you mostly, and I think there is a misunderstanding of the terms. It's possible to serve large amounts of requests with threads as well as with event loops, but there is a distinction between the two on how they consume resources. The author is wrong in assuming threads will not scale, as they will, but a Java web framework using threads will eat up hundreds of megs under heavy load, while an event loop will keep on churning away with progressively longer response times. It's not so much a case of lying as much not knowing when to stop presuming instead of investigating the actual facts.
- ZoFreX 16y ago
  There are resource issues either way, in both cases you have to store state. With threads this can quickly become expensive if you keep the stack size for all those threads at the default amount (a whopping 48K if memory serves). I would say it's easier to use smaller amounts of memory with event-driven programming, but you still have to keep some data around per-client. Of course, if the situation is very asynchronous (seconds instead of milliseconds) then you don't necessarily need all of this hanging around in RAM.
ericflo 16y ago
Yeah, whenever people say threads don't work, point them to Varnish. It uses threads intelligently and everyone agrees it's fast and awesome.
ankrgyl 16y ago
Even with Python (CGI backed by Apache), you can scale with threads. The issue of scaling with threads vs. events is a pretty hot debate, and I think the author sets up this kind of criticism by addressing it poorly. Personally, I fall into the event-driven camp, because it involves the operating system as little as possible (only file descriptors). The C10K link has a great overview of the threaded approach (http://www.kegel.com/c10k.html#threaded http://www.kegel.com/c10k.html#threaded). Read through the rest of the page for a discussion on the pros/cons of other approaches.
- cdavid 16y ago
  Another issue with evented frameworks in python is that you have to ensure nothing is blocking anywhere in your code, which becomes harder the more complex your application becomes. People will often mention IO, database, etc... forgetting that it is also an issue if your request handler takes CPU for N ms (e.g. encoding a relatively large payload in json, etc...). everything needs to be written with async in mind. Languages/frameworks where async IO is an implementation detail (like some web frameworks in haskell) don't have this issue.
  - ntoshev 16y ago
    Tornado devs recommend using blocking mysql drivers and disk I/O. These operations have relatively low and predictable latencies. You can run more Tornado processes than the number of cores is if CPU is underutilized.
  - ankrgyl 16y ago
    I don't think of this as a problem. Tornado is just a small and lightweight non-blocking HTTP server with a couple of convenient libraries. There are some pretty awesome, elegant solutions to the problem you point out, and I like to think of these as extensions to the small Tornado framework. One really cool one is http://thomas.pelletier.im/2010/08/websocket-tornado-redis/ http://thomas.pelletier.im/2010/08/websocket-tornado-redis/, which demonstrates how to use threading to support Redis pub/sub. For heavy computational tasks that aren't time-critical, you could have an accessory worker thread that chugs queued computations (yay first-class functions) in computational downtime.
bdarnell 16y ago
Agreed. If requests per second is the metric you're optimizing for, then threads are a perfectly fine solution (see e.g. Paul Tyma's presentations about how threads are superior to event-driven approaches in Java). The strongest case for event-driven servers is for cases like long polling, where a connection will sit idle for a minute or more and the number of connected idle clients becomes an important metric.
thejbf 16y ago
Yes, but I can still spawn thousands of event-based Scala actors on the same machine. The actual problem with author: He believes running long and costly operations on an HTTP request is OK which is never and ever right. If you need to wait 5 seconds to perform a task, make it asynchronous.
barrkel 16y ago
Something I'd like people to understand, at a very deep level: event-based async programming, and blocking thread-based programming, are fundamentally the same. The problem of maintaining your state in between event triggers is explicit in the first style; it's implicit, in the thread stack and CPU instruction pointer, in the second style. In the second style, it's the OS which is running the event loop, and it dispatches events by resuming a continuation - a continuation that starts by "returning" from the blocking call. Much of the translation from blocking style to event-based style is moving the work of dispatching and looping from kernel to userland. Other ancillary benefits, like reduced address space usage by blocked threads, are in principle also achievable in a threading model - e.g. by storing stack frames on the heap and being more aggressive about collecting them (assuming GC). Other benefits of async - such as overlapping work - are also fairly trivially possible with threading, though less deterministic.
- _delirium 16y ago
  That's essentially the thesis of this 1978 paper, though they group evented programming under the broader "message-passing" style: http://www.sics.se/~adam/pt/duality78.pdf http://www.sics.se/~adam/pt/duality78.pdf
  - barrkel 16y ago
    Indeed. And the corollary is that every possible performance benefit of non-blocking code is in principle achievable with threaded code, though some of the scheduler may need to be implemented in userland, with more explicit hinting of things like lifetimes of local variables in stack frames, etc., to get there. The true benefit of event-oriented network programming may be simply in what it makes explicit vs implicit. By making the tracking of state awkward, it forces its minimization and consolidation. By a process of Worse is Better, it ends up being more performant for scaling despite (usually) being more awkward to program (especially if you have any kind of nesting or recursion structure in your protocol).
    - ankrgyl 16y ago
      Remember that modern implementations of kernel threads are designed to be general-purpose. So, while a threaded model would solve the problems, it also incurs necessary overhead (that works towards solving irrelevant problems). The event-driven approach can be thought of as minimalistic user-space threads for this context. Libraries like Python Eventlet (http://eventlet.net/ http://eventlet.net/) and CML (http://cml.cs.uchicago.edu/ http://cml.cs.uchicago.edu/) are trying to reabstract this into the conveniences of thread-style programming.
jrockway 16y ago
But I'll bet that a node.js HTTP server handling 10,000 slow downloads is going to use a lot less memory than 10,000 Java+OS threads handling those same downloads.

by ZoFreX 16y ago

wildmXranat 16y ago

I agree with you mostly, and I think there is a misunderstanding of the terms. It's possible to serve large amounts of requests with threads as well as with event loops, but there is a distinction between the two on how they consume resources. The author is wrong in assuming threads will not scale, as they will, but a Java web framework using threads will eat up hundreds of megs under heavy load, while an event loop will keep on churning away with progressively longer response times. It's not so much a case of lying as much not knowing when to stop presuming instead of investigating the actual facts.

ZoFreX 16y ago
There are resource issues either way, in both cases you have to store state. With threads this can quickly become expensive if you keep the stack size for all those threads at the default amount (a whopping 48K if memory serves). I would say it's easier to use smaller amounts of memory with event-driven programming, but you still have to keep some data around per-client. Of course, if the situation is very asynchronous (seconds instead of milliseconds) then you don't necessarily need all of this hanging around in RAM.

ericflo 16y ago

Yeah, whenever people say threads don't work, point them to Varnish. It uses threads intelligently and everyone agrees it's fast and awesome.

ankrgyl 16y ago

Even with Python (CGI backed by Apache), you can scale with threads. The issue of scaling with threads vs. events is a pretty hot debate, and I think the author sets up this kind of criticism by addressing it poorly. Personally, I fall into the event-driven camp, because it involves the operating system as little as possible (only file descriptors). The C10K link has a great overview of the threaded approach (http://www.kegel.com/c10k.html#threaded http://www.kegel.com/c10k.html#threaded). Read through the rest of the page for a discussion on the pros/cons of other approaches.

cdavid 16y ago
Another issue with evented frameworks in python is that you have to ensure nothing is blocking anywhere in your code, which becomes harder the more complex your application becomes. People will often mention IO, database, etc... forgetting that it is also an issue if your request handler takes CPU for N ms (e.g. encoding a relatively large payload in json, etc...). everything needs to be written with async in mind. Languages/frameworks where async IO is an implementation detail (like some web frameworks in haskell) don't have this issue.
- ntoshev 16y ago
  Tornado devs recommend using blocking mysql drivers and disk I/O. These operations have relatively low and predictable latencies. You can run more Tornado processes than the number of cores is if CPU is underutilized.
- ankrgyl 16y ago
  I don't think of this as a problem. Tornado is just a small and lightweight non-blocking HTTP server with a couple of convenient libraries. There are some pretty awesome, elegant solutions to the problem you point out, and I like to think of these as extensions to the small Tornado framework. One really cool one is http://thomas.pelletier.im/2010/08/websocket-tornado-redis/ http://thomas.pelletier.im/2010/08/websocket-tornado-redis/, which demonstrates how to use threading to support Redis pub/sub. For heavy computational tasks that aren't time-critical, you could have an accessory worker thread that chugs queued computations (yay first-class functions) in computational downtime.

bdarnell 16y ago

Agreed. If requests per second is the metric you're optimizing for, then threads are a perfectly fine solution (see e.g. Paul Tyma's presentations about how threads are superior to event-driven approaches in Java). The strongest case for event-driven servers is for cases like long polling, where a connection will sit idle for a minute or more and the number of connected idle clients becomes an important metric.

thejbf 16y ago

Yes, but I can still spawn thousands of event-based Scala actors on the same machine. The actual problem with author: He believes running long and costly operations on an HTTP request is OK which is never and ever right. If you need to wait 5 seconds to perform a task, make it asynchronous.

barrkel 16y ago

Something I'd like people to understand, at a very deep level: event-based async programming, and blocking thread-based programming, are fundamentally the same. The problem of maintaining your state in between event triggers is explicit in the first style; it's implicit, in the thread stack and CPU instruction pointer, in the second style. In the second style, it's the OS which is running the event loop, and it dispatches events by resuming a continuation - a continuation that starts by "returning" from the blocking call. Much of the translation from blocking style to event-based style is moving the work of dispatching and looping from kernel to userland. Other ancillary benefits, like reduced address space usage by blocked threads, are in principle also achievable in a threading model - e.g. by storing stack frames on the heap and being more aggressive about collecting them (assuming GC). Other benefits of async - such as overlapping work - are also fairly trivially possible with threading, though less deterministic.

_delirium 16y ago
That's essentially the thesis of this 1978 paper, though they group evented programming under the broader "message-passing" style: http://www.sics.se/~adam/pt/duality78.pdf http://www.sics.se/~adam/pt/duality78.pdf
- barrkel 16y ago
  Indeed. And the corollary is that every possible performance benefit of non-blocking code is in principle achievable with threaded code, though some of the scheduler may need to be implemented in userland, with more explicit hinting of things like lifetimes of local variables in stack frames, etc., to get there. The true benefit of event-oriented network programming may be simply in what it makes explicit vs implicit. By making the tracking of state awkward, it forces its minimization and consolidation. By a process of Worse is Better, it ends up being more performant for scaling despite (usually) being more awkward to program (especially if you have any kind of nesting or recursion structure in your protocol).
  - ankrgyl 16y ago
    Remember that modern implementations of kernel threads are designed to be general-purpose. So, while a threaded model would solve the problems, it also incurs necessary overhead (that works towards solving irrelevant problems). The event-driven approach can be thought of as minimalistic user-space threads for this context. Libraries like Python Eventlet (http://eventlet.net/ http://eventlet.net/) and CML (http://cml.cs.uchicago.edu/ http://cml.cs.uchicago.edu/) are trying to reabstract this into the conveniences of thread-style programming.

jrockway 16y ago

But I'll bet that a node.js HTTP server handling 10,000 slow downloads is going to use a lot less memory than 10,000 Java+OS threads handling those same downloads.