4 ms·

Load Test Rails4 app ruby 2.1 MRI Vs Rubinius 2.0

27 points by resca79 13y ago

harlock81 13y ago
very interesting, thanks!
JimmaDaRustla 13y ago
So, MRI is way faster than Rubinius in this case? Interesting...
- resca79 13y ago
  yes... and I repeat it and sometimes in this case is 2x faster
- mrinterweb 13y ago
  Not necessarily. No warmup time was allowed for rubinius's cache. Also we don't know how concurrency was being handled. Rubinius should be able to utilize multiple cores. More information about the hardware, rails app, way that the puma server is being started, warmup time, duration of request time spent on the database, etc should be made available before any remote conclusion like which one is faster can be made.
  - JimmaDaRustla 13y ago
    That was my concern - Rubunius' selling point is concurrency, so I'm sure there is an approach to implementation, which is why I've never used it.
- mrinterweb 13y ago
  Not necessarily. No warmup time was allowed for rubinius's cache. Also we don't know how concurrency was being handled. Rubinius should be able to utilize multiple cores. More information about the hardware, rails app, way that the puma server is being started, warmup time, duration of request time spent on the database, etc should be made available before any remote conclusion like which one is faster can be made.
- warmwaffles 13y ago
  No. Rubinius' JIT has not had time to warm up. These tests are a little unreliable. I can't find the article off hand, but the way these tests were done aren't exactly telling me anything. Power of ten tests don't do much other than show me that the tester is naive.
cpuguy83 13y ago
This is in no way a good test. Yes, Rubinius takes longer to load than MRI. It'a also JITed, so it needs time to warm up to start screaming. Also, this is in dev mode... did you disable Rack::Lock? MRI can only handle 1 request at a time, Rubinius can handle whatever your concurrency level is (assuming you've removed Rack::Lock from your environment)
- resca79 13y ago
  Rails 4 is thread safe and I used puma as server Also the app is in production mode and the db pool is 16
  - timdorr 13y ago
    Can you post a link to the app? Have you considered running the TechEmpower benchmark app? https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks
  - fbernier 13y ago
    Rails 4.0.0 application starting in development on http://0.0.0.0:3000 http://0.0.0.0:3000 This most likely means Rack::Lock is used in your test.
    - resca79 13y ago
      I used the following command rails s puma -e production
    - rubiquity 13y ago
      Rack::Lock was removed as a default in Rails 4.
- timdorr 13y ago
  > This is in no way a good test. Yes, Rubinius takes longer to load than MRI. It'a also JITed, so it needs time to warm up to start screaming. This is a fair point. You should run ab multiple times over a longer duration of time. 5 seconds seems like a short time to declare an absolute winner.
ksec 13y ago
Gist title is Ruby 2.0 MRI vs Rubinius 2.0 While HN Title is Ruby 2.1 So which is correct?
- resca79 13y ago
  sorry.. I have updated the test with ruby-head Ruby 2.1 MRI 2.1 Rubinius 2.0
jacques_chester 13y ago
Reliable benchmarking is hard. There are actual journals about measuring computer system performance. Some questions: * How is startup time controlled for? * Why only one run? * Why so few requests? * Why no warmup period? * Why this particular configuration? * What is the OS, hardware configuration? * Why is it running in a development environment instead of a test or prod environment?
- resca79 13y ago
  Prod. The app is a very simple rails4 standard app iMac Intel Core 2 Duo 3,06 GHz cpu 1
  - jacques_chester 13y ago
    => Rails 4.0.0 application starting in development on http://0.0.0.0:3000
    - resca79 13y ago
      .... update the gits .. thanks!
- ebonfortress 13y ago
  I would really like to see some articles about proper benchmarking procedures at 1000+ requests per second. Are there any?
  - jacques_chester 13y ago
    The ACM's SIGMETRICS[1] are the main group publishing this kind of research. Some of it gets very specialised -- just picking a title from the most recent Performance Evaluation Review[2], I get "YOUQMON: a system for on-line monitoring of YouTube QoE in operational 3G networks". However, being the ACM, it's basically inaccessible unless you sell a kidney. An organisation with a slightly more industrial focus is the Computer Measurement Group[3]. Like the ACM ... bring money. [1] http://www.sigmetrics.org/ http://www.sigmetrics.org/ [2] http://dl.acm.org/citation.cfm?id=2518025&CFID=252561427&CFTOKEN=88550832 http://dl.acm.org/citation.cfm?id=2518025&CFID=252561427&CFT... [3] http://www.cmg.org/ http://www.cmg.org/
warmwaffles 13y ago
I would also love to see an application in production measured with NewRelic or something else and compare the results. A good way to test this would be to spin up three servers. * 1 load balancer that plays round robin on the requests. * 1 server running MRI * 1 server running Rubinius Each with NewRelic installed and compare the A/B test that is taking place for a period of about a week of heavy usage. Obviously the gems that the rubinius application would have would be different than what the MRI gems would be. But the core of the application would remain the same. This would give me and others better insight as to how the two interpreters are running in a production.
bhauer 13y ago
Although you may be running a single-threaded test intentionally, I generally recommend using a multi-threaded load simulation tool versus Apache's ab tool [1]. My preference is wrk [2], especially since it provides for time-limited tests rather than request-limited. Another alternative is WeigHTTP [3], which is basically a multi-threaded replacement for ab. [1] https://news.ycombinator.com/item?id=6211883 https://news.ycombinator.com/item?id=6211883 [2] https://github.com/wg/wrk https://github.com/wg/wrk [3] https://github.com/lighttpd/weighttp https://github.com/lighttpd/weighttp

27 points by resca79 13y ago

harlock81 13y ago

very interesting, thanks!

JimmaDaRustla 13y ago

So, MRI is way faster than Rubinius in this case? Interesting...

resca79 13y ago
yes... and I repeat it and sometimes in this case is 2x faster
mrinterweb 13y ago
Not necessarily. No warmup time was allowed for rubinius's cache. Also we don't know how concurrency was being handled. Rubinius should be able to utilize multiple cores. More information about the hardware, rails app, way that the puma server is being started, warmup time, duration of request time spent on the database, etc should be made available before any remote conclusion like which one is faster can be made.
- JimmaDaRustla 13y ago
  That was my concern - Rubunius' selling point is concurrency, so I'm sure there is an approach to implementation, which is why I've never used it.
mrinterweb 13y ago
Not necessarily. No warmup time was allowed for rubinius's cache. Also we don't know how concurrency was being handled. Rubinius should be able to utilize multiple cores. More information about the hardware, rails app, way that the puma server is being started, warmup time, duration of request time spent on the database, etc should be made available before any remote conclusion like which one is faster can be made.
warmwaffles 13y ago
No. Rubinius' JIT has not had time to warm up. These tests are a little unreliable. I can't find the article off hand, but the way these tests were done aren't exactly telling me anything. Power of ten tests don't do much other than show me that the tester is naive.

cpuguy83 13y ago

This is in no way a good test. Yes, Rubinius takes longer to load than MRI. It'a also JITed, so it needs time to warm up to start screaming. Also, this is in dev mode... did you disable Rack::Lock? MRI can only handle 1 request at a time, Rubinius can handle whatever your concurrency level is (assuming you've removed Rack::Lock from your environment)

resca79 13y ago
Rails 4 is thread safe and I used puma as server Also the app is in production mode and the db pool is 16
- timdorr 13y ago
  Can you post a link to the app? Have you considered running the TechEmpower benchmark app? https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks
- fbernier 13y ago
  Rails 4.0.0 application starting in development on http://0.0.0.0:3000 http://0.0.0.0:3000 This most likely means Rack::Lock is used in your test.
  - resca79 13y ago
    I used the following command rails s puma -e production
  - rubiquity 13y ago
    Rack::Lock was removed as a default in Rails 4.
timdorr 13y ago
> This is in no way a good test. Yes, Rubinius takes longer to load than MRI. It'a also JITed, so it needs time to warm up to start screaming. This is a fair point. You should run ab multiple times over a longer duration of time. 5 seconds seems like a short time to declare an absolute winner.

ksec 13y ago

Gist title is Ruby 2.0 MRI vs Rubinius 2.0 While HN Title is Ruby 2.1 So which is correct?

resca79 13y ago
sorry.. I have updated the test with ruby-head Ruby 2.1 MRI 2.1 Rubinius 2.0

jacques_chester 13y ago

Reliable benchmarking is hard. There are actual journals about measuring computer system performance. Some questions: * How is startup time controlled for? * Why only one run? * Why so few requests? * Why no warmup period? * Why this particular configuration? * What is the OS, hardware configuration? * Why is it running in a development environment instead of a test or prod environment?

resca79 13y ago
Prod. The app is a very simple rails4 standard app iMac Intel Core 2 Duo 3,06 GHz cpu 1
- jacques_chester 13y ago
  => Rails 4.0.0 application starting in development on http://0.0.0.0:3000
  - resca79 13y ago
    .... update the gits .. thanks!
ebonfortress 13y ago
I would really like to see some articles about proper benchmarking procedures at 1000+ requests per second. Are there any?
- jacques_chester 13y ago
  The ACM's SIGMETRICS[1] are the main group publishing this kind of research. Some of it gets very specialised -- just picking a title from the most recent Performance Evaluation Review[2], I get "YOUQMON: a system for on-line monitoring of YouTube QoE in operational 3G networks". However, being the ACM, it's basically inaccessible unless you sell a kidney. An organisation with a slightly more industrial focus is the Computer Measurement Group[3]. Like the ACM ... bring money. [1] http://www.sigmetrics.org/ http://www.sigmetrics.org/ [2] http://dl.acm.org/citation.cfm?id=2518025&CFID=252561427&CFTOKEN=88550832 http://dl.acm.org/citation.cfm?id=2518025&CFID=252561427&CFT... [3] http://www.cmg.org/ http://www.cmg.org/

warmwaffles 13y ago

I would also love to see an application in production measured with NewRelic or something else and compare the results. A good way to test this would be to spin up three servers. * 1 load balancer that plays round robin on the requests. * 1 server running MRI * 1 server running Rubinius Each with NewRelic installed and compare the A/B test that is taking place for a period of about a week of heavy usage. Obviously the gems that the rubinius application would have would be different than what the MRI gems would be. But the core of the application would remain the same. This would give me and others better insight as to how the two interpreters are running in a production.

bhauer 13y ago

Although you may be running a single-threaded test intentionally, I generally recommend using a multi-threaded load simulation tool versus Apache's ab tool [1]. My preference is wrk [2], especially since it provides for time-limited tests rather than request-limited. Another alternative is WeigHTTP [3], which is basically a multi-threaded replacement for ab. [1] https://news.ycombinator.com/item?id=6211883 https://news.ycombinator.com/item?id=6211883 [2] https://github.com/wg/wrk https://github.com/wg/wrk [3] https://github.com/lighttpd/weighttp https://github.com/lighttpd/weighttp