4 ms·

This sounds really cool but one thing that this blog post didn't mention is that there is memory constraint on each dyno. So, this model might work if you have

by siong1987 14y ago

This sounds really cool but one thing that this blog post didn't mention is that there is memory constraint on each dyno. So, this model might work if you have a Rails[1] app that doesn't use up a lot of memory. And, since Unicorn forks as worker processes, so, 2 unicorn workers will use up double the memory of a typical Rails app. If you have a really huge Rails app, you could still end up using only one unicorn worker for your Rails app, which is the same as just using any webserver on one dyno. One interesting thing about the post is the mention of Puma[2] as the webserver for Rails app. Puma is thread based, so, its memory footprint is significantly less than unicorn. So, in theory, you should be able to handle more concurrent requests(more threads) per dyno with Puma. Btw, don't just blindly follow the post. Make sure you test your app and see whether your app use up more memory each dyno allows. If your dyno runs out of memory, Heroku will just silently drop all your requests. So, you might end up having a worse problem. [1]: In general, Rails uses up quite a lot of memory on cold start. [2]: http://puma.io/ http://puma.io/

tenderlove 14y ago
I'd like to figure out where the memory consumption is. I feel like we could do a better job reducing the cold start memory footprint. I've done some stuff with routes, but I feel there are other wins we could get. The good news is we have `ObjectSpace.memsize_of_all`: irb(main):001:0> require 'objspace' => true irb(main):002:0> ObjectSpace.memsize_of_all(String) => 169318 irb(main):003:0> x = 'x' * 1024; nil => nil irb(main):004:0> ObjectSpace.memsize_of_all(String) => 171817 irb(main):005:0> What's cool is that in MRI we can get size of the instruction sequences too: irb(main):005:0> ObjectSpace.memsize_of_all(RubyVM::InstructionSequence) => 1600800 irb(main):006:0> 100.times { |i| eval("def x_#{i}; end") } => 100 irb(main):007:0> ObjectSpace.memsize_of_all(RubyVM::InstructionSequence) => 1711904 Also we can see the delta between methods defined with eval vs define_method: irb(main):011:0> ObjectSpace.memsize_of_all(RubyVM::InstructionSequence) => 1717984 irb(main):012:0> 100.times { |i| define_method("x_#{i}") { } } => 100 irb(main):013:0> ObjectSpace.memsize_of_all(RubyVM::InstructionSequence) => 1722416 :-) EDIT: stop saying "cool" so much. :-/
- oelmekki 14y ago
  I'm totally not in memory optimization, but I suppose an app memory size is directly proportional to its required gems' size ? Or is it in the contrary quite insignificant in runtime memory ?
  - tenderlove 14y ago
    Ya, it's definitely related to the number of files required. There are other factors though, like whether methods are defined via eval vs define_method, string allocations from stuff like database tables / column names.
btilly 14y ago
And, since Unicorn forks as worker processes, so, 2 unicorn workers will use up double the memory of a typical Rails app. Thanks to copy on write, the amount of memory needed for 2 processes is strictly less than 2. Just make sure to preload as much data as you can before you fork. And in many use cases memory slowly comes unstuck (change anything, and a whole page of data comes unshared) so it can make sense to kill workers every so often and let them refork with everything properly shared.
- pixelcort 14y ago
  Will Unicorn automatically kill older forked processes periodically and fork off new ones, or is that something that would need to be done separately?
  - swampthing 14y ago
    One of the comments on the Heroku article mentions this gem, which does just that: https://github.com/kzk/unicorn-worker-killer https://github.com/kzk/unicorn-worker-killer
    - kawsper 14y ago
      Regarding memory, I think you can do something like this in your Unicorn config: after_fork do |server, worker| size_in_bytes = 60010241024 # megabytes Process.setrlimit(Process::RLIMIT_AS, size_in_bytes)
  - kawsper 14y ago
    I think Unicorn kills off old workers and spawns new regularly.
  - lucaspiller 14y ago
    We haven't had any issues with memory leaks since we've been using Unicorn and Rails 3. Back in the old days of Mongrel and Rails 2 we had to restart the processes periodically because of this, but we haven't seen any issues with this setup. Here are the stats: www-data 21714 0.0 5.6 102928 87164 ? Sl Feb26 0:30 unicorn_rails master -c production.conf.rb -E production -D www-data 21762 0.2 8.5 150672 131996 ? Sl Feb26 1:58 unicorn_rails worker[0] -c production.conf.rb -E production -D www-data 21768 0.2 8.5 152036 132352 ? Sl Feb26 1:58 unicorn_rails worker[1] -c production.conf.rb -E production -D www-data 21774 0.2 8.1 145248 125668 ? Sl Feb26 1:56 unicorn_rails worker[2] -c production.conf.rb -E production -D www-data 21780 0.2 8.3 148792 129736 ? Sl Feb26 2:01 unicorn_rails worker[3] -c production.conf.rb -E production -D www-data 21786 0.1 8.3 148384 129260 ? Sl Feb26 1:48 unicorn_rails worker[4] -c production.conf.rb -E production -D www-data 21792 0.1 8.6 152156 133880 ? Sl Feb26 1:52 unicorn_rails worker[5] -c production.conf.rb -E production -D
oelmekki 14y ago
> So, this model might work if you have a Rails[1] app that doesn't use up a lot of memory. I can confirm that. In my previous company, we had a lot of H12 (request timeout) on heroku. We thought this should be our codebase fault, without being able to isolate why some usually fast requests were sometime so long to complete. As a workaround, we decided to temporary switch to unicorn to add more concurrency to our dynos. H12 went down, but we were flooded by R14 (memory exceeded). Even with two workers, it was too much. Granted, our app was eating too much memory and we were refactoring it to split it into several distinct apps. We also had the problem that heroku router seemed to send requests to dyno while master unicorn process was starting (and thus, could not process it yet), which was a major problem since we used hirefire, which keep starting new dynos(and thus, new unicorn masters).

tenderlove 14y ago

I'd like to figure out where the memory consumption is. I feel like we could do a better job reducing the cold start memory footprint. I've done some stuff with routes, but I feel there are other wins we could get. The good news is we have `ObjectSpace.memsize_of_all`: irb(main):001:0> require 'objspace' => true irb(main):002:0> ObjectSpace.memsize_of_all(String) => 169318 irb(main):003:0> x = 'x' * 1024; nil => nil irb(main):004:0> ObjectSpace.memsize_of_all(String) => 171817 irb(main):005:0> What's cool is that in MRI we can get size of the instruction sequences too: irb(main):005:0> ObjectSpace.memsize_of_all(RubyVM::InstructionSequence) => 1600800 irb(main):006:0> 100.times { |i| eval("def x_#{i}; end") } => 100 irb(main):007:0> ObjectSpace.memsize_of_all(RubyVM::InstructionSequence) => 1711904 Also we can see the delta between methods defined with eval vs define_method: irb(main):011:0> ObjectSpace.memsize_of_all(RubyVM::InstructionSequence) => 1717984 irb(main):012:0> 100.times { |i| define_method("x_#{i}") { } } => 100 irb(main):013:0> ObjectSpace.memsize_of_all(RubyVM::InstructionSequence) => 1722416 :-) EDIT: stop saying "cool" so much. :-/

oelmekki 14y ago
I'm totally not in memory optimization, but I suppose an app memory size is directly proportional to its required gems' size ? Or is it in the contrary quite insignificant in runtime memory ?
- tenderlove 14y ago
  Ya, it's definitely related to the number of files required. There are other factors though, like whether methods are defined via eval vs define_method, string allocations from stuff like database tables / column names.

btilly 14y ago

And, since Unicorn forks as worker processes, so, 2 unicorn workers will use up double the memory of a typical Rails app. Thanks to copy on write, the amount of memory needed for 2 processes is strictly less than 2. Just make sure to preload as much data as you can before you fork. And in many use cases memory slowly comes unstuck (change anything, and a whole page of data comes unshared) so it can make sense to kill workers every so often and let them refork with everything properly shared.

pixelcort 14y ago
Will Unicorn automatically kill older forked processes periodically and fork off new ones, or is that something that would need to be done separately?
- swampthing 14y ago
  One of the comments on the Heroku article mentions this gem, which does just that: https://github.com/kzk/unicorn-worker-killer https://github.com/kzk/unicorn-worker-killer
  - kawsper 14y ago
    Regarding memory, I think you can do something like this in your Unicorn config: after_fork do |server, worker| size_in_bytes = 60010241024 # megabytes Process.setrlimit(Process::RLIMIT_AS, size_in_bytes)
- kawsper 14y ago
  I think Unicorn kills off old workers and spawns new regularly.
- lucaspiller 14y ago
  We haven't had any issues with memory leaks since we've been using Unicorn and Rails 3. Back in the old days of Mongrel and Rails 2 we had to restart the processes periodically because of this, but we haven't seen any issues with this setup. Here are the stats: www-data 21714 0.0 5.6 102928 87164 ? Sl Feb26 0:30 unicorn_rails master -c production.conf.rb -E production -D www-data 21762 0.2 8.5 150672 131996 ? Sl Feb26 1:58 unicorn_rails worker[0] -c production.conf.rb -E production -D www-data 21768 0.2 8.5 152036 132352 ? Sl Feb26 1:58 unicorn_rails worker[1] -c production.conf.rb -E production -D www-data 21774 0.2 8.1 145248 125668 ? Sl Feb26 1:56 unicorn_rails worker[2] -c production.conf.rb -E production -D www-data 21780 0.2 8.3 148792 129736 ? Sl Feb26 2:01 unicorn_rails worker[3] -c production.conf.rb -E production -D www-data 21786 0.1 8.3 148384 129260 ? Sl Feb26 1:48 unicorn_rails worker[4] -c production.conf.rb -E production -D www-data 21792 0.1 8.6 152156 133880 ? Sl Feb26 1:52 unicorn_rails worker[5] -c production.conf.rb -E production -D

oelmekki 14y ago

> So, this model might work if you have a Rails[1] app that doesn't use up a lot of memory. I can confirm that. In my previous company, we had a lot of H12 (request timeout) on heroku. We thought this should be our codebase fault, without being able to isolate why some usually fast requests were sometime so long to complete. As a workaround, we decided to temporary switch to unicorn to add more concurrency to our dynos. H12 went down, but we were flooded by R14 (memory exceeded). Even with two workers, it was too much. Granted, our app was eating too much memory and we were refactoring it to split it into several distinct apps. We also had the problem that heroku router seemed to send requests to dyno while master unicorn process was starting (and thus, could not process it yet), which was a major problem since we used hirefire, which keep starting new dynos(and thus, new unicorn masters).