7 ms·

GitHub's Unicorn Setup

162 points by mqt 17y ago

boundlessdreamz 17y ago
How does passenger handle restarts? Does it also allow a zero downtime restart ?
- joevandyk 17y ago
  You run "touch #{RAILS_ROOT}/tmp/restart.txt". That will restart the rails processes. No connections are dropped, and I've not seen any downtime.
  - latortuga 17y ago
    To clarify, my understanding is that while rails is restarting, Passenger will queue all the requests that come in and begin processing them as soon as rails is ready. But yeah, zero downtime, it's pretty awesome.
    - joevandyk 17y ago
      that is correct.
    - fizx 17y ago
      Sites do tend to lag for at least several seconds after the restart though.
      - caseyf 17y ago
        Yeah. We load balance across 5 Apache/Passengers and I do rolling deploys (all in Capistrano) by removing a Passenger from load balancing, updating the app, restarting Apache, and adding it back into load balancing with a 10 second delay between each. We tried the Passenger touch restart.txt and that didn't go well at all when we were under load.
        chadr 17y ago
        Have you tried contacting the Passenger developers about this? I won't be surprised if they come out with a fix for this.
        boundlessdreamz 17y ago
        Can you share your setup please?
        brett 17y ago
        He has been kind enough to share about his setup quite a lot about lately. In fact, looks like it's on the HN homepage right now: http://news.ycombinator.com/item?id=872301 http://news.ycombinator.com/item?id=872301
        caseyf 17y ago
        I wrote a short post about our setup in March and it hasn't really changed since then: http://codemonkey.ravelry.com/2009/03/10/quick-update-ravelry-runs-on/ http://codemonkey.ravelry.com/2009/03/10/quick-update-ravelr... If you have any questions or are looking for details or something, ask away - just stick a comment on the blog post.
      - EvilTrout 17y ago
        Also, if you're using Monit to watch the memory bloat of your passenger processes, when you kill one process it tends to do the same stalling thing on all your processes until the ONE you killed is back up.
dschobel 17y ago
When the Unicorn master starts, it loads our app into memory. As soon as it’s ready to serve requests it forks 16 workers. Those workers then select() on the socket, only serving requests they’re capable of handling. In this way the kernel handles the load balancing for us. Wasn't there a heated debate here just the other day about the prefork model? Guess it's at least back en vogue @github.
- seiji 17y ago
  It's connection pooling, not a fork-per-accept web server. Each worker is select/epoll/kqueue'ing for individual requests.
  - aaronblohowiak 17y ago
    Right, but it is being handled by processes and not threads, which the argument was about the other day.
    - antonovka 17y ago
      The argument encompassed two aspects of the model: 1) "Threads are out -- processes are better than threads." 2) Process-per-connection architectures. The first is demonstrably false -- for instance, look at Erlang, which maps lightweight erlang processes to operating system threads, providing SMP scalability at a low cost without running into Github's issues with mongrel "thread-killing". More broadly used, look at Servlets and the Servlet 3.0 support for async comet-style event-based request handling. Each request is handled on a thread as necessary. Inter-thread communication (where necessary) is cheap, and this scales just fine. If you use the fork() model and a long-running request blocks the entire process, comet is basically a non-starter. This is why people are interested (and implement) lightweight threads, coroutines, and restartable request implementations. For a conceptual challenge, consider how you would implement a live web chat system that can scale up to a considerable number of clients, with extremely low resource usage, and instant message distribution (no polling). Locally, we implemented this with async servlet support and M:N scheduled scala actors -- a blocking HTTP request doesn't hold a thread or process hostage in the web server or the application, and we can scale up to enormous number of live clients on one machine. The second was merely a lack of understanding of the model. In implementations where fork() is used as an alternative to threads and multiple connections are not handled per sub-process, you quickly run into scaling issues with subprocess memory utilization. In cases where subprocesses handle multiple connections via an event mechanism, you're just using fork() instead of threads. I'm wondering how many servers github is using to keep up with load -- If each 8 core 16 gigabyte (!!!) server can actually only handle 16 concurrent requests via a pool of 16 workers, that's an incredibly poor (and expensive) scaling model.
cakeface 17y ago
Its great to see companies sharing what they've learned from experience about system architecture. So many times this sort of stuff is very difficult to plan out and the only real way to get it right is through experimentation. Having first hand descriptions like this is a great resource if you are setting something up the first time. I like that they posted their unicorn config file too!
- defunkt 17y ago
  Also it doubles as documentation for the members of our team who aren't familiar with this part of the system :)
blasdel 17y ago
I don't understand why people insist on architectures where otherwise-independent processes share a single socket. You're already running a reverse proxy in front of them! There's no reason each Unicorn couldn't be listening on a different port. Does that third layer of local load-balancing between the HTTP proxy and the event-driven app server actually get you anything?
- wmf 17y ago
  Replacing N ports with one simplifies configuration. I never understood the complex HAProxy in front of Apache in front of Nginx in front of Mongrel type setups that seem to be popular in the Rails world. Why not just use Unicorn? What value is GitHub getting from having Nginx in front?
  - defunkt 17y ago
    Unicorn is not for slow clients or static assets. That's what nginx is for. See http://unicorn.bogomips.org/PHILOSOPHY.html http://unicorn.bogomips.org/PHILOSOPHY.html for info on Unicorn and slow clients. nginx also has features like ESI, serving from memcached, and rate limiting which Unicorn does not (and doesn't need).
  - jasonwatkinspdx 17y ago
    Because Ruby 1.8 threading sucks you pay a large memory price (ie a process) for each concurrent request in flight. A fronting proxy allows your backends to write out the response as fast as possible and move on to another request while the proxy spoon feeds the response to slow clients. Also, nginx is going to be more efficient for serving static files, though most larger apps will have broken such requests out to a separate set of domains likely serviced by a cdn.
- defunkt 17y ago
  If each Unicorn worker listened on a different port, we'd have to use one of nginx's load balancing strategies (unless I'm misreading your comment). We have not had success with them in the past which is why we employed HAProxy with mongrel.
atambo 17y ago
Is there any benefit in using unicorn over passenger?
- fizx 17y ago
  Unicorn doesn't make your site very slow for the 5 seconds-2 minutes (YMMV) after you deploy.
  - chadr 17y ago
    It seems to me like the Passenger guys could easily add an option so that on a "touch tmp/restart.txt" a set of new worker processes is started before the old ones are killed off. I imagine this would make this slowness a thing of the past. For the record, my apps experiencing this momentary queueing and slowness on restart (5 seconds max).
Cornify 17y ago
The Grand Unicorn is proud!

162 points by mqt 17y ago

boundlessdreamz 17y ago

How does passenger handle restarts? Does it also allow a zero downtime restart ?

joevandyk 17y ago
You run "touch #{RAILS_ROOT}/tmp/restart.txt". That will restart the rails processes. No connections are dropped, and I've not seen any downtime.
- latortuga 17y ago
  To clarify, my understanding is that while rails is restarting, Passenger will queue all the requests that come in and begin processing them as soon as rails is ready. But yeah, zero downtime, it's pretty awesome.
  - joevandyk 17y ago
    that is correct.
  - fizx 17y ago
    Sites do tend to lag for at least several seconds after the restart though.
    - caseyf 17y ago
      Yeah. We load balance across 5 Apache/Passengers and I do rolling deploys (all in Capistrano) by removing a Passenger from load balancing, updating the app, restarting Apache, and adding it back into load balancing with a 10 second delay between each. We tried the Passenger touch restart.txt and that didn't go well at all when we were under load.
      - chadr 17y ago
        Have you tried contacting the Passenger developers about this? I won't be surprised if they come out with a fix for this.
      - boundlessdreamz 17y ago
        Can you share your setup please?
        brett 17y ago
        He has been kind enough to share about his setup quite a lot about lately. In fact, looks like it's on the HN homepage right now: http://news.ycombinator.com/item?id=872301 http://news.ycombinator.com/item?id=872301
        caseyf 17y ago
        I wrote a short post about our setup in March and it hasn't really changed since then: http://codemonkey.ravelry.com/2009/03/10/quick-update-ravelry-runs-on/ http://codemonkey.ravelry.com/2009/03/10/quick-update-ravelr... If you have any questions or are looking for details or something, ask away - just stick a comment on the blog post.
    - EvilTrout 17y ago
      Also, if you're using Monit to watch the memory bloat of your passenger processes, when you kill one process it tends to do the same stalling thing on all your processes until the ONE you killed is back up.

dschobel 17y ago

When the Unicorn master starts, it loads our app into memory. As soon as it’s ready to serve requests it forks 16 workers. Those workers then select() on the socket, only serving requests they’re capable of handling. In this way the kernel handles the load balancing for us. Wasn't there a heated debate here just the other day about the prefork model? Guess it's at least back en vogue @github.

seiji 17y ago
It's connection pooling, not a fork-per-accept web server. Each worker is select/epoll/kqueue'ing for individual requests.
- aaronblohowiak 17y ago
  Right, but it is being handled by processes and not threads, which the argument was about the other day.
  - antonovka 17y ago
    The argument encompassed two aspects of the model: 1) "Threads are out -- processes are better than threads." 2) Process-per-connection architectures. The first is demonstrably false -- for instance, look at Erlang, which maps lightweight erlang processes to operating system threads, providing SMP scalability at a low cost without running into Github's issues with mongrel "thread-killing". More broadly used, look at Servlets and the Servlet 3.0 support for async comet-style event-based request handling. Each request is handled on a thread as necessary. Inter-thread communication (where necessary) is cheap, and this scales just fine. If you use the fork() model and a long-running request blocks the entire process, comet is basically a non-starter. This is why people are interested (and implement) lightweight threads, coroutines, and restartable request implementations. For a conceptual challenge, consider how you would implement a live web chat system that can scale up to a considerable number of clients, with extremely low resource usage, and instant message distribution (no polling). Locally, we implemented this with async servlet support and M:N scheduled scala actors -- a blocking HTTP request doesn't hold a thread or process hostage in the web server or the application, and we can scale up to enormous number of live clients on one machine. The second was merely a lack of understanding of the model. In implementations where fork() is used as an alternative to threads and multiple connections are not handled per sub-process, you quickly run into scaling issues with subprocess memory utilization. In cases where subprocesses handle multiple connections via an event mechanism, you're just using fork() instead of threads. I'm wondering how many servers github is using to keep up with load -- If each 8 core 16 gigabyte (!!!) server can actually only handle 16 concurrent requests via a pool of 16 workers, that's an incredibly poor (and expensive) scaling model.

cakeface 17y ago

Its great to see companies sharing what they've learned from experience about system architecture. So many times this sort of stuff is very difficult to plan out and the only real way to get it right is through experimentation. Having first hand descriptions like this is a great resource if you are setting something up the first time. I like that they posted their unicorn config file too!

defunkt 17y ago
Also it doubles as documentation for the members of our team who aren't familiar with this part of the system :)

blasdel 17y ago

I don't understand why people insist on architectures where otherwise-independent processes share a single socket. You're already running a reverse proxy in front of them! There's no reason each Unicorn couldn't be listening on a different port. Does that third layer of local load-balancing between the HTTP proxy and the event-driven app server actually get you anything?

wmf 17y ago
Replacing N ports with one simplifies configuration. I never understood the complex HAProxy in front of Apache in front of Nginx in front of Mongrel type setups that seem to be popular in the Rails world. Why not just use Unicorn? What value is GitHub getting from having Nginx in front?
- defunkt 17y ago
  Unicorn is not for slow clients or static assets. That's what nginx is for. See http://unicorn.bogomips.org/PHILOSOPHY.html http://unicorn.bogomips.org/PHILOSOPHY.html for info on Unicorn and slow clients. nginx also has features like ESI, serving from memcached, and rate limiting which Unicorn does not (and doesn't need).
- jasonwatkinspdx 17y ago
  Because Ruby 1.8 threading sucks you pay a large memory price (ie a process) for each concurrent request in flight. A fronting proxy allows your backends to write out the response as fast as possible and move on to another request while the proxy spoon feeds the response to slow clients. Also, nginx is going to be more efficient for serving static files, though most larger apps will have broken such requests out to a separate set of domains likely serviced by a cdn.
defunkt 17y ago
If each Unicorn worker listened on a different port, we'd have to use one of nginx's load balancing strategies (unless I'm misreading your comment). We have not had success with them in the past which is why we employed HAProxy with mongrel.

atambo 17y ago

Is there any benefit in using unicorn over passenger?

fizx 17y ago
Unicorn doesn't make your site very slow for the 5 seconds-2 minutes (YMMV) after you deploy.
- chadr 17y ago
  It seems to me like the Passenger guys could easily add an option so that on a "touch tmp/restart.txt" a set of new worker processes is started before the old ones are killed off. I imagine this would make this slowness a thing of the past. For the record, my apps experiencing this momentary queueing and slowness on restart (5 seconds max).

Cornify 17y ago

The Grand Unicorn is proud!