5 ms·

Can anyone who's used Puma on Heroku comment on how it compares to Unicorn?

enoptix 13y ago
Much better memory usage and (for us at least) better concurrency. We could only run 4 unicorn workers on a single dyno. But with Puma we run 16 threads with ease.
- cpursley 13y ago
  Thanks, this is all very relevant at the moment. Trying to optimize page load times for a Heroku Rails app. Web server is one of the bottlenecks.
- nixme 13y ago
  Have you tried running 3 or even 4 puma workers (each with 8-16) threads on a dyno? That way you can get more concurrency on CPU-bound requests in addition to IO-bound concurrency (assuming MRI).
  - jrochkind1 13y ago
    I tried that in my attempt at benchmarking a standardized app on heroku, and, yep, it works great: https://github.com/jrochkind/fake_work_app/blob/master/README.md https://github.com/jrochkind/fake_work_app/blob/master/READM...
nlh 13y ago
I switched from Unicorn to Puma for my relatively low-load Heroku-based Rails app and have seen nothing but improvements across the board. Less memory usage, lower response times, overall better performance. It's working very well. Note: I'm an amateur and I know I'm not nearly optimized across many areas -- the app could be better, I'm not using JRuby (where multi-threaded shines, or so I read), so this is from a pretty novice perspective. But from here, Puma has been great.
- hhenn 13y ago
  What did you use for profiling? Their newrelic add-on?
cmelbye 13y ago
Puma works great on Heroku. Rubinius, on the other hand, doesn't. It was incredibly slow, the app took more than a minute to boot up which caused Heroku to think it crashed. However, you can still use Puma with MRI 1.9.3 or something else and it'll still work great.
jrochkind1 13y ago
I attempted to do some generic benchmarking of just this question and found Puma way outperformed Unicorn under heavy load if your request processing are largely io-bound. Didn't get much attention when I tried to post it to HN when I did it a couple months ago: https://github.com/jrochkind/fake_work_app/blob/master/README.md https://github.com/jrochkind/fake_work_app/blob/master/READM...
- tinco 13y ago
  Your application is not fair at all. It looks like your sleep mimics I/O boundedness, but in reality it does nothing but set an arbitrary request length. In a real world application I/O requests contend for resources with eachother. Sleep calls don't contend with each other at all. This means that application servers can just keep stacking concurrent requests with perfect performance. If there was actually I/O being done that took 250ms, then you would find that it was the real bottleneck, and it would be the dominant factor in your benchmark, with the differences between app servers all but disappearing.
  - FooBarWidget 13y ago
    That benchmark is fair if you consider the context: I/O bound workoads. On those kinds of workoads, your app spends a large amount of time waiting for I/O. While waiting, it does not use any CPU. This is very well simulated using sleep calls.
  - jrochkind1 13y ago
    It is not clear to me that my application fails at realistically simulating I/O (I agree with FooBarWidget below) (and I'm not sure where the word 'fair' comes from or what it means here, that's your word not mine), but I tried to provide as much code and information about exactly what I tested so you could decide for yourself, so fair enough. (I agree that with real I/O, there wouldn't be _exactly_ (eg) 50ms of waiting in every request; it would differ from request to request, and might slow down under heavier load. That's true, but I don't think it matters for what I was trying to test -- I was not trying to test how well a given rdbms can handle load, for instance; I was trying to standardize that, to test the app's performance. My simulation assumes an (eg) 50ms _average_ iowait time, by just assigning every request 50ms of wait. Or instead of 50ms, whatever average iowait you wanted to posit and test, and I tested a few. It is indeed a simulation, but I think it captures the significant things for what we wanted to measure). However! I would very much welcome seeing the results from a test done by you (or anyone else), with an application that simulates (or actually does) I/O in a way that you believe is more realistic and leads to more instructive results. It certainly would be interesting to see if it led to different results than my tests, or not. I suspect it would not (because I think my app is realistically simulating the parts that matter, naturally! but I've been wrong before!) Feel free to fork my code to do so, if my code is a useful starting point for you. Please do share your results! Of course, you could also run tests with some actual real world app as well, instead of an app simulating a real world load in a standardized way like I used, and see what those results look like. I'd certainly welcome seeing that too! There's certainly plenty of different sorts of tests that could be done. I'd love it if people started doing some of them and sharing their results. I tried to contribute my best effort to that.

Much better memory usage and (for us at least) better concurrency. We could only run 4 unicorn workers on a single dyno. But with Puma we run 16 threads with ease.

cpursley 13y ago
Thanks, this is all very relevant at the moment. Trying to optimize page load times for a Heroku Rails app. Web server is one of the bottlenecks.
nixme 13y ago
Have you tried running 3 or even 4 puma workers (each with 8-16) threads on a dyno? That way you can get more concurrency on CPU-bound requests in addition to IO-bound concurrency (assuming MRI).
- jrochkind1 13y ago
  I tried that in my attempt at benchmarking a standardized app on heroku, and, yep, it works great: https://github.com/jrochkind/fake_work_app/blob/master/README.md https://github.com/jrochkind/fake_work_app/blob/master/READM...

nlh 13y ago

I switched from Unicorn to Puma for my relatively low-load Heroku-based Rails app and have seen nothing but improvements across the board. Less memory usage, lower response times, overall better performance. It's working very well. Note: I'm an amateur and I know I'm not nearly optimized across many areas -- the app could be better, I'm not using JRuby (where multi-threaded shines, or so I read), so this is from a pretty novice perspective. But from here, Puma has been great.

hhenn 13y ago
What did you use for profiling? Their newrelic add-on?

cmelbye 13y ago

Puma works great on Heroku. Rubinius, on the other hand, doesn't. It was incredibly slow, the app took more than a minute to boot up which caused Heroku to think it crashed. However, you can still use Puma with MRI 1.9.3 or something else and it'll still work great.

jrochkind1 13y ago

I attempted to do some generic benchmarking of just this question and found Puma way outperformed Unicorn under heavy load if your request processing are largely io-bound. Didn't get much attention when I tried to post it to HN when I did it a couple months ago: https://github.com/jrochkind/fake_work_app/blob/master/README.md https://github.com/jrochkind/fake_work_app/blob/master/READM...

tinco 13y ago
Your application is not fair at all. It looks like your sleep mimics I/O boundedness, but in reality it does nothing but set an arbitrary request length. In a real world application I/O requests contend for resources with eachother. Sleep calls don't contend with each other at all. This means that application servers can just keep stacking concurrent requests with perfect performance. If there was actually I/O being done that took 250ms, then you would find that it was the real bottleneck, and it would be the dominant factor in your benchmark, with the differences between app servers all but disappearing.
- FooBarWidget 13y ago
  That benchmark is fair if you consider the context: I/O bound workoads. On those kinds of workoads, your app spends a large amount of time waiting for I/O. While waiting, it does not use any CPU. This is very well simulated using sleep calls.
- jrochkind1 13y ago
  It is not clear to me that my application fails at realistically simulating I/O (I agree with FooBarWidget below) (and I'm not sure where the word 'fair' comes from or what it means here, that's your word not mine), but I tried to provide as much code and information about exactly what I tested so you could decide for yourself, so fair enough. (I agree that with real I/O, there wouldn't be _exactly_ (eg) 50ms of waiting in every request; it would differ from request to request, and might slow down under heavier load. That's true, but I don't think it matters for what I was trying to test -- I was not trying to test how well a given rdbms can handle load, for instance; I was trying to standardize that, to test the app's performance. My simulation assumes an (eg) 50ms _average_ iowait time, by just assigning every request 50ms of wait. Or instead of 50ms, whatever average iowait you wanted to posit and test, and I tested a few. It is indeed a simulation, but I think it captures the significant things for what we wanted to measure). However! I would very much welcome seeing the results from a test done by you (or anyone else), with an application that simulates (or actually does) I/O in a way that you believe is more realistic and leads to more instructive results. It certainly would be interesting to see if it led to different results than my tests, or not. I suspect it would not (because I think my app is realistically simulating the parts that matter, naturally! but I've been wrong before!) Feel free to fork my code to do so, if my code is a useful starting point for you. Please do share your results! Of course, you could also run tests with some actual real world app as well, instead of an app simulating a real world load in a standardized way like I used, and see what those results look like. I'd certainly welcome seeing that too! There's certainly plenty of different sorts of tests that could be done. I'd love it if people started doing some of them and sharing their results. I tried to contribute my best effort to that.