4 ms·

The argument encompassed two aspects of the model: 1) "Threads are out -- processes are better than threads." 2) Process-per-connection architectures. The fi

The argument encompassed two aspects of the model: 1) "Threads are out -- processes are better than threads." 2) Process-per-connection architectures. The first is demonstrably false -- for instance, look at Erlang, which maps lightweight erlang processes to operating system threads, providing SMP scalability at a low cost without running into Github's issues with mongrel "thread-killing". More broadly used, look at Servlets and the Servlet 3.0 support for async comet-style event-based request handling. Each request is handled on a thread as necessary. Inter-thread communication (where necessary) is cheap, and this scales just fine. If you use the fork() model and a long-running request blocks the entire process, comet is basically a non-starter. This is why people are interested (and implement) lightweight threads, coroutines, and restartable request implementations. For a conceptual challenge, consider how you would implement a live web chat system that can scale up to a considerable number of clients, with extremely low resource usage, and instant message distribution (no polling). Locally, we implemented this with async servlet support and M:N scheduled scala actors -- a blocking HTTP request doesn't hold a thread or process hostage in the web server or the application, and we can scale up to enormous number of live clients on one machine. The second was merely a lack of understanding of the model. In implementations where fork() is used as an alternative to threads and multiple connections are not handled per sub-process, you quickly run into scaling issues with subprocess memory utilization. In cases where subprocesses handle multiple connections via an event mechanism, you're just using fork() instead of threads. I'm wondering how many servers github is using to keep up with load -- If each 8 core 16 gigabyte (!!!) server can actually only handle 16 concurrent requests via a pool of 16 workers, that's an incredibly poor (and expensive) scaling model.

mrshoe 17y ago
Where were all you people in this thread the other day when I seemed to be the only one who understood the other side of this debate?!? :-P I'm also very curious about the 16 workers per box model. Frankly, that github would ditch haproxy in favor of a pre-fork Ruby web server is shocking. I'm just glad we have CherryPy in the Python world.
- mojombo 17y ago
  Why is that shocking? Can you elaborate?
  - mrshoe 17y ago
    Sure, now that HN has decided that enough time has elapsed to allow me to reply. :-) Actually, antonovka did an excellent job explaining it above. The most important aspect is that threads are "lighter weight" than processes. They use less memory and are quicker to context switch (usually). The result of this lighter weight is that you can spawn more threads than you could processes, on the same hardware. And when you're using one thread/process per connection, that means more concurrent connections on the same hardware. So, if Unicorn used its exact same architecture, but replaced the worker processes with worker threads, you could scale much better. Secondly, haproxy is written in C, which generally means it's going to perform much better and use a lot less memory than a Ruby webserver. This translates, once again, to more output from the same amount of hardware. That's why I was pretty surprised to see that github would choose both Ruby and pre-fork over C and threads (or in the case of haproxy, async, which scales even better).
    - joevandyk 17y ago
      The limiting factor of concurrency in a Rails application is the slowness of Ruby, not processes vs threads. If the bottleneck of Ruby were fixed, then the next bottleneck would be the database.
    - grandalf 17y ago
      I think you missed the point. The point was not that github stopped using haproxy, but that it no longer needs to use it. Assuming haproxy can work with unix sockets, github could configure nginx/unicorn to use it, but with unicorn its current work load does not require additional load balancing acrobatics beyond what the unix socket does. Similarly, many rails sites use thin/nginx or mongrel/nginx and do not even have a workload that necessitates haproxy. Plus, haproxy adds another layer of complexity, configuration, and management, which is nice to avoid if you can.
    - 17y ago
mojombo 17y ago
Each box can handle plenty more than 16 workers, but we currently don't need that many, so it's pointless to run them. Our frontend machines generally sit about 60% idle. We wanted to have plenty of headroom on our new setup. Also, our Ruby application code spends very little time blocking on external resources (compared to time spent in Ruby execution), so having async execution on these wouldn't give us that much more efficiency. It's true that running Ruby code is generally more expensive (dollar-wise) per given task than other languages, but we made a conscious decision to make that tradeoff for the productivity gains that Ruby affords us. We use Erlang and EventMachine in other pieces of the architecture where they make sense. The simple fact is that Unicorn works better for us (and our specific use case) than anything else we've tried, and so we're using it.
- antonovka 17y ago
  Each box can handle plenty more than 16 workers, but we currently don't need that many, so it's pointless to run them. Our frontend machines generally sit about 60% idle. We wanted to have plenty of headroom on our new setup. At those numbers, it sounds like you'd max out at roughly 32 workers (so 32 concurrent requests), or am I missing something important? It's true that running Ruby code is generally more expensive (dollar-wise) per given task than other languages, but we made a conscious decision to make that tradeoff for the productivity gains that Ruby affords us. It's really easy to measure performance, but productivity claims seem difficult to support and highly susceptible to confirmation bias. I think that Ruby, Scala, and Clojure (just for instance) would stack up against each other very well in terms of productivity, but I wouldn't know how to prove it.
  - joevandyk 17y ago
    The github developers are great ruby developers, and I'd be very surprised if they would be as productive in the short term in a different language. I do not see a business case for switching platforms for them. You seem hung up on the number of concurrent requests. An 8 core machine can only do 8 things at once, no matter if you are using threads or processes.
joevandyk 17y ago
The 16 worker processes only handle dynamic requests, not static. With proper http caching, ESI, etc, it's quite likely that amount amount of concurrency is more than enough for their needs. Say that when all the workers are being utilized, the average dynamic request takes 50ms (which is on the high end). That means that each box can handle 320 dynamic requests per second. Which is a decent amount, and I'd be surprised if they see that much traffic. If they are using http caching and ESI properly, the average request time would be significantly lower, and the requests per second would probably be > 500/s on that box.
- deleted 17y ago
  [deleted]
- antonovka 17y ago
  If they are using http caching and ESI properly, the average request time would be significantly lower, and the requests per second would probably be > 500/s on that box. 500 req/s is pretty abysmal for a 8 core 16 gigabyte server. I realize that some of that has to do with Ruby performance, but yikes -- that is frighteningly bad scaling, regardless of the cause.
  - mojombo 17y ago
    A number like 500 req/s is irrelevant without an understanding of what the site has to do. For a simple site that does simple database lookups, I'd expect more than that. For a site like GitHub that has to do database lookups and pull large amounts of data from Git repositories, it's an entirely different story. Without knowing the split between cache hits and misses, the number is even more irrelevant. You're arguing in abstracts, whereas I am constrained to actually working with real life. I'd love to see some examples of the type of sites you're running and the solutions you employ wherein 500 req/s of dynamic requests on an 8 core machine are considered "frighteningly bad scaling."
  - joevandyk 17y ago
    How can you say it's bad without knowing how knowing what each request is doing? Github isn't a microbenchmark. A concrete example: I can say that one of my rails sites handles 850 dynamic requests per second running on a single small 1 cpu server. That's because all that particular request does is lookup 4k of data from memcached and returns it. (i.e. http://www.tanga.com/feeds/current_deal.xml http://www.tanga.com/feeds/current_deal.xml) However, as a general rule, I know that each small server can handle about 30-50 pages per second, because each page takes a lot of data crunching to generate (and because I haven't been bothered to make it as efficient as possible, it's fast enough as is). If all I was doing was returning a small bit of text that didn't require much lookups or calculation, then sure, a 8 core cpu with rails could probably do 4-5000 requests per second easily.
mbrubeck 17y ago
"Look at Erlang, which maps lightweight erlang processes to operating system threads, providing SMP scalability at a low cost..." Erlang can use native threads, but they do not map to its lightweight processes. Erlang's lightweight processes are basically green threads. Native threads are used for SMP scheduling only. So on a four-core system, you might have thousands of Erlang lightweight processes running on just four OS threads in one OS process.
- antonovka 17y ago
  Erlang can use native threads, but they do not map to its lightweight processes. That's not what I said; I said: "[Erlang] maps lightweight erlang processes to operating system threads." Erlang processes are green threads mapped to OS threads. They're called processes in erlang, but they're M:N scheduled threads. So on a four-core system, you might have one thousands of Erlang lightweight processes running on just four OS threads in one OS process. Or you could have thousands of Erlang lightweight processes mapped to 185 OS threads (not that it will, but it could). The fact that they're green threads is irrelevant in terms of scalability -- they're still threads, but even less resource intensive than a direct mapping to OS threads. M:N scheduled green threading is also very hard to do in the general case, which is why you don't see more of it (see the abandoned attempts to implement generic M:N OS:green thread scheduling in FreeBSD, for instance). You could also be using Scala, where an actor only maps to a thread when it's blocked inside of its react loop, but in that case will consume a whole thread. What's your point? :)
  - mbrubeck 17y ago
    Sorry, I misunderstood what you were saying. You are right.
davidw 17y ago
Erlang is not really an argument for threads. The Erlang model is to utilize poll/select/whatever together with a language that contains its own scheduler. If you execute arbitrary python/ruby/tcl/php code, there's a chance it will block. If you execute arbitrary Erlang code, the chance of that is very low. This means that you can handle all kinds of long running calculations in the same OS process, while you continue to handle incoming requests. The fact that Erlang farms out a few threads is a later optimization added to the language several years back in order to take advantage of SMP. However, it only creates a number of threads to match the number of processes, IIRC. After that it shouldn't spawn any more, so I don't think you would particularly call it an example of a 'threaded model'. Erlang's big advantage is the internal scheduler, and a systematic approach to writing code that will never block the Erlang OS process.
- antonovka 17y ago
  Erlang is not really an argument for threads. Solaris 2-8 implemented a general purpose M:N thread scheduler mapping user-space threads to kernel threads. This is the exact same solution Erlang implements for the less general purpose use-case of actor messaging. Is Solaris' M:N thread implementation somehow not 'threaded'? If it is threaded (and it is), then how is Erlang's implementation not also an argument for threads? To elucidate rather than rely on Erlang as an example, the argument for threads over processes: 1) Extremely low-cost alternative to IPC. Threads allow (but do not require the use of) shared mutable state. This is a much, much cheaper way to communicate between concurrent entities. You can achieve the same effect with processes and shared memory, but it's significantly more complex to implement and subject to the disadvantages listed below. 2) Extremely low memory foot-print. A thread costs a stack plus minor OS book keeping. A fork(2) can leverage COW pages, but almost invariably the number of non-shared pages will be significantly higher than with a thread. If an operation blocks a thread, it's cheap to create more. If an operation blocks a process, you'll hit resource constraints far more quickly trying to fork() more. Of course, leveraging shared memory and other operating system tools, you can turn a fork(2) implementation into a thread alternative -- but then, you'd have ... threads. This is what Linux's clone(2) syscall was intended for -- a thread-implementation friendly fork(2) alternative -- and the pthread library was built on top of it. ... for a while, you could call setuid() in a Linux "thread" and the new uid would be a thread-local change, because the thread was actually a 'process'. This is effectively not an argument against fork(2) (although it's an expensive route to the same solution offered by threads), but rather against scaling models that will block an entire OS process for the sake of a single request.

mrshoe 17y ago

Where were all you people in this thread the other day when I seemed to be the only one who understood the other side of this debate?!? :-P I'm also very curious about the 16 workers per box model. Frankly, that github would ditch haproxy in favor of a pre-fork Ruby web server is shocking. I'm just glad we have CherryPy in the Python world.

mojombo 17y ago
Why is that shocking? Can you elaborate?
- mrshoe 17y ago
  Sure, now that HN has decided that enough time has elapsed to allow me to reply. :-) Actually, antonovka did an excellent job explaining it above. The most important aspect is that threads are "lighter weight" than processes. They use less memory and are quicker to context switch (usually). The result of this lighter weight is that you can spawn more threads than you could processes, on the same hardware. And when you're using one thread/process per connection, that means more concurrent connections on the same hardware. So, if Unicorn used its exact same architecture, but replaced the worker processes with worker threads, you could scale much better. Secondly, haproxy is written in C, which generally means it's going to perform much better and use a lot less memory than a Ruby webserver. This translates, once again, to more output from the same amount of hardware. That's why I was pretty surprised to see that github would choose both Ruby and pre-fork over C and threads (or in the case of haproxy, async, which scales even better).
  - joevandyk 17y ago
    The limiting factor of concurrency in a Rails application is the slowness of Ruby, not processes vs threads. If the bottleneck of Ruby were fixed, then the next bottleneck would be the database.
  - grandalf 17y ago
    I think you missed the point. The point was not that github stopped using haproxy, but that it no longer needs to use it. Assuming haproxy can work with unix sockets, github could configure nginx/unicorn to use it, but with unicorn its current work load does not require additional load balancing acrobatics beyond what the unix socket does. Similarly, many rails sites use thin/nginx or mongrel/nginx and do not even have a workload that necessitates haproxy. Plus, haproxy adds another layer of complexity, configuration, and management, which is nice to avoid if you can.
  - 17y ago

mojombo 17y ago

Each box can handle plenty more than 16 workers, but we currently don't need that many, so it's pointless to run them. Our frontend machines generally sit about 60% idle. We wanted to have plenty of headroom on our new setup. Also, our Ruby application code spends very little time blocking on external resources (compared to time spent in Ruby execution), so having async execution on these wouldn't give us that much more efficiency. It's true that running Ruby code is generally more expensive (dollar-wise) per given task than other languages, but we made a conscious decision to make that tradeoff for the productivity gains that Ruby affords us. We use Erlang and EventMachine in other pieces of the architecture where they make sense. The simple fact is that Unicorn works better for us (and our specific use case) than anything else we've tried, and so we're using it.

antonovka 17y ago
Each box can handle plenty more than 16 workers, but we currently don't need that many, so it's pointless to run them. Our frontend machines generally sit about 60% idle. We wanted to have plenty of headroom on our new setup. At those numbers, it sounds like you'd max out at roughly 32 workers (so 32 concurrent requests), or am I missing something important? It's true that running Ruby code is generally more expensive (dollar-wise) per given task than other languages, but we made a conscious decision to make that tradeoff for the productivity gains that Ruby affords us. It's really easy to measure performance, but productivity claims seem difficult to support and highly susceptible to confirmation bias. I think that Ruby, Scala, and Clojure (just for instance) would stack up against each other very well in terms of productivity, but I wouldn't know how to prove it.
- joevandyk 17y ago
  The github developers are great ruby developers, and I'd be very surprised if they would be as productive in the short term in a different language. I do not see a business case for switching platforms for them. You seem hung up on the number of concurrent requests. An 8 core machine can only do 8 things at once, no matter if you are using threads or processes.

joevandyk 17y ago

The 16 worker processes only handle dynamic requests, not static. With proper http caching, ESI, etc, it's quite likely that amount amount of concurrency is more than enough for their needs. Say that when all the workers are being utilized, the average dynamic request takes 50ms (which is on the high end). That means that each box can handle 320 dynamic requests per second. Which is a decent amount, and I'd be surprised if they see that much traffic. If they are using http caching and ESI properly, the average request time would be significantly lower, and the requests per second would probably be > 500/s on that box.

deleted 17y ago
[deleted]
antonovka 17y ago
If they are using http caching and ESI properly, the average request time would be significantly lower, and the requests per second would probably be > 500/s on that box. 500 req/s is pretty abysmal for a 8 core 16 gigabyte server. I realize that some of that has to do with Ruby performance, but yikes -- that is frighteningly bad scaling, regardless of the cause.
- mojombo 17y ago
  A number like 500 req/s is irrelevant without an understanding of what the site has to do. For a simple site that does simple database lookups, I'd expect more than that. For a site like GitHub that has to do database lookups and pull large amounts of data from Git repositories, it's an entirely different story. Without knowing the split between cache hits and misses, the number is even more irrelevant. You're arguing in abstracts, whereas I am constrained to actually working with real life. I'd love to see some examples of the type of sites you're running and the solutions you employ wherein 500 req/s of dynamic requests on an 8 core machine are considered "frighteningly bad scaling."
- joevandyk 17y ago
  How can you say it's bad without knowing how knowing what each request is doing? Github isn't a microbenchmark. A concrete example: I can say that one of my rails sites handles 850 dynamic requests per second running on a single small 1 cpu server. That's because all that particular request does is lookup 4k of data from memcached and returns it. (i.e. http://www.tanga.com/feeds/current_deal.xml http://www.tanga.com/feeds/current_deal.xml) However, as a general rule, I know that each small server can handle about 30-50 pages per second, because each page takes a lot of data crunching to generate (and because I haven't been bothered to make it as efficient as possible, it's fast enough as is). If all I was doing was returning a small bit of text that didn't require much lookups or calculation, then sure, a 8 core cpu with rails could probably do 4-5000 requests per second easily.

mbrubeck 17y ago

"Look at Erlang, which maps lightweight erlang processes to operating system threads, providing SMP scalability at a low cost..." Erlang can use native threads, but they do not map to its lightweight processes. Erlang's lightweight processes are basically green threads. Native threads are used for SMP scheduling only. So on a four-core system, you might have thousands of Erlang lightweight processes running on just four OS threads in one OS process.

antonovka 17y ago
Erlang can use native threads, but they do not map to its lightweight processes. That's not what I said; I said: "[Erlang] maps lightweight erlang processes to operating system threads." Erlang processes are green threads mapped to OS threads. They're called processes in erlang, but they're M:N scheduled threads. So on a four-core system, you might have one thousands of Erlang lightweight processes running on just four OS threads in one OS process. Or you could have thousands of Erlang lightweight processes mapped to 185 OS threads (not that it will, but it could). The fact that they're green threads is irrelevant in terms of scalability -- they're still threads, but even less resource intensive than a direct mapping to OS threads. M:N scheduled green threading is also very hard to do in the general case, which is why you don't see more of it (see the abandoned attempts to implement generic M:N OS:green thread scheduling in FreeBSD, for instance). You could also be using Scala, where an actor only maps to a thread when it's blocked inside of its react loop, but in that case will consume a whole thread. What's your point? :)
- mbrubeck 17y ago
  Sorry, I misunderstood what you were saying. You are right.

davidw 17y ago

Erlang is not really an argument for threads. The Erlang model is to utilize poll/select/whatever together with a language that contains its own scheduler. If you execute arbitrary python/ruby/tcl/php code, there's a chance it will block. If you execute arbitrary Erlang code, the chance of that is very low. This means that you can handle all kinds of long running calculations in the same OS process, while you continue to handle incoming requests. The fact that Erlang farms out a few threads is a later optimization added to the language several years back in order to take advantage of SMP. However, it only creates a number of threads to match the number of processes, IIRC. After that it shouldn't spawn any more, so I don't think you would particularly call it an example of a 'threaded model'. Erlang's big advantage is the internal scheduler, and a systematic approach to writing code that will never block the Erlang OS process.

antonovka 17y ago
Erlang is not really an argument for threads. Solaris 2-8 implemented a general purpose M:N thread scheduler mapping user-space threads to kernel threads. This is the exact same solution Erlang implements for the less general purpose use-case of actor messaging. Is Solaris' M:N thread implementation somehow not 'threaded'? If it is threaded (and it is), then how is Erlang's implementation not also an argument for threads? To elucidate rather than rely on Erlang as an example, the argument for threads over processes: 1) Extremely low-cost alternative to IPC. Threads allow (but do not require the use of) shared mutable state. This is a much, much cheaper way to communicate between concurrent entities. You can achieve the same effect with processes and shared memory, but it's significantly more complex to implement and subject to the disadvantages listed below. 2) Extremely low memory foot-print. A thread costs a stack plus minor OS book keeping. A fork(2) can leverage COW pages, but almost invariably the number of non-shared pages will be significantly higher than with a thread. If an operation blocks a thread, it's cheap to create more. If an operation blocks a process, you'll hit resource constraints far more quickly trying to fork() more. Of course, leveraging shared memory and other operating system tools, you can turn a fork(2) implementation into a thread alternative -- but then, you'd have ... threads. This is what Linux's clone(2) syscall was intended for -- a thread-implementation friendly fork(2) alternative -- and the pthread library was built on top of it. ... for a while, you could call setuid() in a Linux "thread" and the new uid would be a thread-local change, because the thread was actually a 'process'. This is effectively not an argument against fork(2) (although it's an expensive route to the same solution offered by threads), but rather against scaling models that will block an entire OS process for the sake of a single request.