11 ms·

Is Ruby Fast Yet?

132 points by j_s 13y ago

austinpray 13y ago
>Hardware: Mid-2007 MacBook Pro Wouldn't it make more sense to do this on generic server hardware with a LTS Linux OS? Once that vintage and customized MacBook bites the dust the website pretty much dies as well.
- Tloewald 13y ago
  There's nothing magical about a Linux server. If you're arguing an LTS edition will run on newer hardware, well sure but the benchmarks will be just as incomparable.
  - awinder 13y ago
    You're right that theres nothing magical, but there is a clear advantage to faster disks, a more modern & faster CPU, newer & faster ram (this is running DDR2 in the benchmark), etc. I'd be curious why this particular machine was used for benchmarking, and if the same benchmarking method yielded different results on EC2 or something like that.
    - rschmitty 13y ago
      You should check out http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/ where they test on hardware and in the cloud
  - asveikau 13y ago
    What are the odds that the Linux kernel is faster than OS X for server workloads? I don't have any data to back this up but I've heard awful things about their filesystem in particular. (Supposedly they have a global filesystem lock.)
RyanZAG 13y ago
Not sure how comparing different versions of rails shows if its 'fast' yet. I'd think setting up some request target and seeing how far RoR is from that target with each version would be better. Or comparing RoR to other platforms is also a good bet, eg http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/ I just don't really see how this answers the question other than 'ehh I think it looks a bit faster than it was awhile ago?' EDIT: Also interestingly, JRuby seems to be slightly better on the small instance EC2 test at techempower than regular Ruby (small instance should be similar to a laptop?). Yet in these graphs it's much slower. Anybody know the reason?
- jarrett 13y ago
  Amazon says their "m1.small" instance has 1.7 GiB of memory. That's a lot less than most current laptops. Maybe JRuby uses more memory in total, but can handle more concurrent requests per MB of memory used? That could explain the discrepancy if it's true, though it's just speculation on my part. Another theory is that the benchmarks are too application-specific. Rails is a complex framework with which you can build many different kinds of applications, each with its own performance characteristics. Conceivably, application A could perform better on JRuby, while application B could perform better on MRI. Depending on which application you use for your benchmark, either interpreter could appear faster. This calls into question the utility of benchmarking an entire framework. It might be better to do micro-benchmarks, with each one testing a specific scenario. Scenarios could include, for example: A request that loads 1,000 database records into memory; a request for a static image; a request that queries a many-to-many relationship; a request that saves an uploaded file to disk.
  - lukeschlather 13y ago
    The thing is, those are all I/O bound operations and in the case of the databases most of the code you're benchmarking is native extensions, so I would expect the difference to be negligible. Testing whole Rails apps really does seem like the best way to get good data. The interesting speedups are likely going to be in places like string matching / string copying and other places that don't involve network access.
matthuggins 13y ago
Wouldn't it make more sense to test various features of Ruby for comparison rather than comparing an entire web framework? Given that, the URL should be israilsfastyet.com, or more appropriately howdoesrailsbehaveonvariousrubyversions.com
- mikesilvis 13y ago
  Agreed. This is much more of a test of the rails framework then ruby
  - jrochkind1 13y ago
    Well, it's a comparison of different ruby versions, using Rails. That's pretty useful to people who use Rails, which is a lot of people. Of course, it's not a real Rails app presumably, but just a sample/dummy one. So you may or may not see comparable speed-up with a real app. But it's probably suggestive of the upper limit of improvement you can get by switching ruby versions. "Will I get a performance improvement from my Rails app by upgrading ruby" is a real question a lot of people will have. Micro-benchmarks of ruby wouldn't answer it as well as the OP does.
  - jeltz 13y ago
    Not really since it does not compare it to another other benchmarks. This is a test of how the performance of MRI ruby has changed over time.
- regularfry 13y ago
  If you test small features, you get knocked for concentrating on microbenchmarks which don't have any real-world meaning. If you test whole applications or frameworks, you get criticised for benchmarking that rather than the language implementation.
  - Liru 13y ago
    The criticism is with the name, not the features tested.
mpweiher 13y ago
tl;dr: no.
- michaelhoffman 13y ago
  I was expecting the web page to just consist of "No."
  - mpweiher 13y ago
    Yep, that was my expectation as well :-)
  - theandrewbailey 13y ago
    ...just like https://isitchristmas.com/ https://isitchristmas.com/ on days other than Christmas.
yuvipanda 13y ago
5s startup time for a web framework seems rather high. I wonder how well it compares to similar frameworks in other languages (Django? ASP.NET MVC?)
- x1024 13y ago
  Eh... no it doesn't.
  - briantakita 13y ago
    It is when your development process involves rapidly iterating the code by running the process many times (i.e. TDD). Anecdotally, when the feedback loop is tighter, I'm much happier. Developing (especially refactoring) in Rails feels like watching paint dry, since it's soooooo slooooow. Every time the process needs to restart, I'm left hanging........waiting......for..............the.............. ..........process.......................to.................... ...................................................s.......... ................t.............................a............... ..............r............................................... ..tttt........... Also the benchmark is the best case. In real world projects, I've seen Rails take over 60 seconds to start (it took a few days of development effort to bring it down to ~20 seconds). Every time you rerun tests, run a rake task, start or restart your server, etc. It's like the heartbeat of the development process grinds to a halt when you add more features to your Rails server. And no, tools like Spork are not the answer, since you still have to balance the amount that is resident in memory to fork with coverage of what you can change. It also opens up a fractal of complexity.
    - tawm 13y ago
      Oh god. Please edit your comment so it doesn't break the whole layout.
      - briantakita 13y ago
        No can do. It got downvoted and HN does not allow me to edit negative karma posts. So please upvote my previous post so I can fix the layout of these comments :-) I'm assuming the negative votes are due to the formatting and not the message. I hope you, dear reader, don't miss the irony of this situation... If it's downvotes because of the message, well I feel bad for the downvoters and wish them much luck with using their framework.
Jgrubb 13y ago
JRuby question -- Why is it that much slower? I'd have thought that part of the point was to take advantage of the JVM's built in optimization.
- mml 13y ago
  My question as well. Every perf benchmark I've seen of jruby over the years, it's much faster than MRI (given time to boot & JIT anyway). I think this guy's numbers are very suspect.
  - exabrial 13y ago
    I was about to say the same thing... I thought JRuby was king for long running Ruby apps (like a Rails app) for many years now... seems very suspect.
  - jshen 13y ago
    I've benchmarked MRI vs jruby a number of times over the years on real apps and MRI usually cones out faster on average with worse numbers at 95th and 98th percentiles due to GC performance. I never took the time to figure out why it is faster, but I assume that it is due to C libraries being faster for json serialization and my apps doing primarily json serialization after fetching from the database. Edit: I should test the performance of the json libs, which would be easy to do
- sluukkonen 13y ago
  Looks like it's running with the threadsafe mode off. That would probably give JRuby an edge, as it's able to take advantage of greater thread parallelism than MRI. No idea if he's letting the JVM warm up enough either.
  - jrochkind1 13y ago
    Depending on how you are deploying the ruby app. As far as I know, Passenger Enterprise and Puma are the only multi-platform ruby rack app servers that support multi-threaded concurrent request dispatch. There may be other jruby/java-only deployment scenarios that support concurrent request dispatch. Without concurrent request dispatch, jruby's hypothetical greater thread parallelism is unlikely to effect requests-per-second benchmarks. Also, the benchmarking test would have to be willing to issue requests concurrently!
timmorgan 13y ago
I'm surprised at how negative the comments are here. Graphing how different Ruby versions/implementations perform running Rails is very helpful! Glad to see that MRI continues to get faster with every version. Merry Christmas!
- jarito 13y ago
  The negativity comes from the fact that this benchmark is useless and, at worst, misleading. True benchmarks are hard to get right. Running a bunch of stuff on a laptop isn't sufficient evidence of anything. Kudos to making the benchmark opensource at least, but even from a quick peak, I wouldn't trust this at all. As others have mentioned, JRuby shouldn't be this slow. Odds are the benchmark is throwing away the JIT gains or the configuration is poor. Add that to the fact that there is no discussion of what a 'rails request' is or what version of Java is being used and there seems to be little to learn from this research.
  - jrochkind1 13y ago
    I'm not sure I have ever seen a benchmark on HN that wasn't called misleading and useless in the comments. But I agree it is hard to get benchmarks right -- I'm not even sure there is a 'right', as a benchmark will usually be based on certain assumptions that only sometimes apply. (Unless you are benchmarking your own actual production code, in which case you know it's assumptions are valid -- for your own code). So I don't think that's a reason not to share benchmarks. But I _do_ think it's a reason to be really transparent about exactly how the benchmarks were done. Ideally all code should be available on github. Then people can examine your assumptions (including ones you knew you were making and ones you didn't) and modelling, as well as fork or pull request to do benchmarks in some other way to better model the scenarios they care about. Although also in my experience, you get LOTS of people telling you your benchmark is wrong because of X, Y or Z -- sometimes they are correct that it doesn't accurately capture something they are interested in, sometimes they are just confused -- but virtually nobody actually interested in writing code via fork or pull request to capture what they are interested in. Once you put out a benchmark, everyone in the peanut gallery thinks they are entitled to keep demanding you change it in various ways for them.) No matter how much time you spend on it and thought you put into it, publishing a benchmark on HN is probably the easiest way to have lots of people telling you you're an idiot who did things wrong. It almost seems like HN collectively wants to discourage people from running benchmarks -- or from sharing what they found publicly. But it's because doing good benchmarking is so hard -- well, that's exactly we need more people working on it, and sharing how they did it so others can improve upon it, etc, right?
bri3d 13y ago
Without knowing what runtimes and parameters are used it's really hard to tell what's going on in this benchmark; JRuby can see 100% and sometimes more improvement on many workloads by tweaking JVM options. It looks like JRuby is just run under the system JRE (which is what?) with no additional options; that's going to be slow for most workloads. It'd be nice to see a lot more documentation around the runtime configurations and what the Rails app itself is doing.
- hornetblack 13y ago
  Down the bottom he states that it is running on Osx. The fan is premptively spun up, which gives Jruby a speed boost. It's probably going to run even faster with proper cooling.
- hornetblack 13y ago
  Down the bottom he states that it is running on Osx. The fan is premptively spun up, which gives Jruby a speed boost. It's probably going to run even faster with proper cooling.
- hornetblack 13y ago
  Down the bottom he states that it is running on Osx. The fan is premptively spun up, which gives Jruby a speed boost. It's probably going to run even faster with proper cooling.
Freaky 13y ago
> The benchmark premptively runs the fans and pauses between runs in order to cool off. These measures give JRuby at least a 25% speed gain. However, JRuby still might be running slower than it would on a box with appropriate cooling. Right. Your computer can't be trusted to not throttle its performance due to crap cooling, and you're trying to benchmark using it?
- miahi 13y ago
  The choice of hardware is quite strange. Should be fun to check the results on normal hardware though.
deleted 13y ago
[deleted]
gesman 13y ago
Compare to what? And the answer is no, and unlikely to be any time soon: http://www.techempower.com/benchmarks/#section=data-r8&hw=i7&test=query http://www.techempower.com/benchmarks/#section=data-r8&hw=i7...
astrodust 13y ago
There's great data here that's obscured by choosing terrible colours for the graph and showing things that are pretty much irrelevant. A summary chart giving relevant percentage differences would be a lot better than this line chart.
phaedryx 13y ago
I was expecting Ruby benchmarks, not Rails benchmarks.
w0rd-driven 13y ago
There is one sentence missing from each graph that would've made this a lot easier to grok: "x is better" where x is more requests, faster startup time, or less memory usage. I prefer graphs that lead me to what I should expect and that sentence would go a long way to eliminate the "mental math" I had to do to understand each one. On another note, the 2 year graphs are particularly interesting. They're more consistent than not but those inconsistencies seem a little awkward.
- brianhempel 13y ago
  The variance is sometimes due to RVM troubles. The benchmark is partly a test of the entire Ruby ecosystem. If RVM is bugging, then people are installing sub-optimal implementations. For example: https://github.com/wayneeseguin/rvm/issues/1418 https://github.com/wayneeseguin/rvm/issues/1418
lowglow 13y ago
[serious] I'm building http://techendo.co/ http://techendo.co/ and it's running rails on heroku. I was wondering if there was some easy ways to get some better performance out of it immediately now that we're picking up traffic. If anyone could give me some pointers or link me to a page I'd appreciate it.
- RyanZAG 13y ago
  Going off the techempower benchmarks, looks like you should be able to get some speedup by switching to TorqueBox. No guarantees, but if you've got the time and need the performance it could be worth testing. Here seems to be a decent guide you could follow: https://gist.github.com/bbrowning/4296297 https://gist.github.com/bbrowning/4296297
  - lowglow 13y ago
    does it matter that I'm not using JRuby?
    - RyanZAG 13y ago
      You'd have to convert your gems over - you need jruby gems rather than standard ones, but apparently most gems are ported. You can read up here for more: https://devcenter.heroku.com/articles/moving-an-existing-rails-app-to-run-on-jruby https://devcenter.heroku.com/articles/moving-an-existing-rai... http://stackoverflow.com/questions/151595/jruby-on-rails-vs-ruby-on-rails-whats-difference http://stackoverflow.com/questions/151595/jruby-on-rails-vs-... So it's not just a 5 second thing, but if you have a bit of time and need performance it's something you can try.
      - lowglow 13y ago
        hm. I guess I'm not ready to switch to jruby.
- awj 13y ago
  That's an incredibly broad question, so it's pretty hard to answer. Here's some general ones: * More instances will give you more requests/second, but only so long as secondary system (database, search server, whatever) can keep up with the load * N+1 queries will kill your performance. They can add a lot of time to processing requests. * Everything in the most used parts of your site should hit an index on the database. If it's not I can almost guarantee it will be slow. * Look for work that you can move "offline". As long as they go out within a minute nobody will notice that you aren't sending emails at the exact time they are being generated. Move that work to a background queue so your web server can just add a queue entry and get on to the next request. * Cache anything you reasonably can. It's a lot cheaper (i.e. faster) to read from a cache than to do a database lookup. It's cheaper to read from a cache than to render an ERB template. Caching can introduce its own difficulties, but the performance gains can be worth it. Without knowing some specifics of your implementation I really can't give you better advice. Depending on the specific libraries you use it might not even be possible to move to JRuby due to support for C extensions. You might get some performance benefits out of it, but I'd expect it to only buy you some time to execute on some of the above improvements.
saalweachter 13y ago
If I understand this correctly, this is running periodically on someone's Macbook; people are complaining that's a terrible platform for a benchmark. It should be possible (if you care enough) to re-run this from the source control history on platforms of your choosing, across whatever date ranges you wish. Anyone really, really care enough?
polysics 13y ago
As a non-Rails Rubyist, I will have to run benchmarks myself as usual, because people seem to be still equating Ruby to Rails.
noelwelsh 13y ago
If you're going to call it "Is Ruby Fast Yet", you should really be comparing Ruby implementations to implementations of other programming languages. Asked this way, I believe the answer to the question is a definite "no". As a small example, I benchmarked a real Scala web app running on worse hardware (a 1GB Linux VM) at over 650 requests/s. There was no degradation in load at this rate of requests, but it was 10x the load we saw in production so I stopped there. The benchmark of Rails here goes to ~120 requests/s.
- warmwaffles 13y ago
  Well there is a JRuby and Rubinius comparison. Those are different implementations of the the interpreter. Ruby will never be faster than C, but this site is aiming to show the improvements over time.
- brianhempel 13y ago
  Comparing across languages was originally part of the long-term goal. However, I haven't yet come across any good, multi-language, real-world benchmark.
lazugod 13y ago
Are web fonts fast yet?
bernardom 13y ago
I was expecting a one-time-use site such as http://www.whatwouldmyharvardgrade.be/ http://www.whatwouldmyharvardgrade.be/
davidw 13y ago
I don't like those charts at all. I want to see the progressive Ruby versions on the X axis, I think. The dates are pretty meaningless. If it's the same code, it should run in the same number of cycles; anything else is just noise. That said, +1 for having a go at it! Also, yes, Ruby is pretty slow. But it's fast for getting something up and running quickly. If you get wildly successful, you can always swap it out for something else if you need to.
sams99 13y ago
FYI we are also working on a similar project see: http://samsaffron.com/archive/2013/12/11/call-to-action-long-running-ruby-benchmark http://samsaffron.com/archive/2013/12/11/call-to-action-long... and http://community.miniprofiler.com/t/is-ruby-fast-yet/197 http://community.miniprofiler.com/t/is-ruby-fast-yet/197
kbar13 13y ago
benchmarking a web framework on an old macbook pro?
shocks 13y ago
Yes, if we're talking about development time! :P
spoiler 13y ago
We can at least see MRI has gotten faster with each new version, but we knew that already. Also, I'm a bit surprised that this doesn't include more implementations of Ruby. Where's Rubinius? IronRuby? REE? I hope the list gets updated. Furthermore, the benchmark itself could be better; I won't say I am disappointed, but I am not too thrilled either.
munificent 13y ago
I've been hacking on a little scripting language of my own lately[1]. To get a handle on its perf, I have a few small benchmarks[2] I run that compare it against Lua (5.1.5), Python (2.7.5), and Ruby (2.0.0p247). (All of those and my language are bytecode-compiled, dynamically typed languages, hence that comparison.) I happen to have Ruby 2.0 installed, and I was surprised to see it handily beat the other languages. I had expected Ruby to be the slowest of the pack, but they've apparently made big strides in 2.0. On the off chance that anyone is curious, here's the results of running them on my machine: binary_trees - wren 3531 0.28s binary_trees - lua 1366 0.73s 258.40% binary_trees - python 1420 0.70s 248.64% binary_trees - ruby 3170 0.32s 111.39% fib - wren 2688 0.37s fib - lua 2998 0.33s 89.65% fib - python 1390 0.72s 193.38% fib - ruby 3645 0.27s 73.74% for - wren 9092 0.11s for - lua 9387 0.11s 96.86% for - python 3171 0.32s 286.74% for - ruby 8319 0.12s 109.30% method_call - wren 4703 0.21s method_call - lua 1780 0.56s 264.20% method_call - python 806 1.24s 583.33% method_call - ruby 3408 0.29s 138.02% Those numbers are score, time, and percentage. Score is basically 1/time, just to yield a number where bigger is better. Time is best of ten runs. Percentage is how my language's score compares to that one. So, for example, the 264.20% lua method_call score means my language is about 2.6x faster than Lua on a benchmark designed to stress dynamic method call performance. [1] https://github.com/munificent/wren https://github.com/munificent/wren [2] https://github.com/munificent/wren/tree/master/benchmark https://github.com/munificent/wren/tree/master/benchmark
- StefanKarpinski 13y ago
  Oh, man, this reminds me of the pain of writing even the most trivial microbenchmarks in seven or eight languages. How about throwing in some of the fast scripting languages for healthy competition? E.g. JavaScript, LuaJIT and Julia. Feel free to borrow (MIT licensed, but I'm willing to relicense whatever I can): https://github.com/JuliaLang/julia/tree/master/test/perf/micro https://github.com/JuliaLang/julia/tree/master/test/perf/mic...
  - munificent 13y ago
fantnn 13y ago
Was expecting one of those generic yes/no sites, a la http://istherejava0day.com/ http://istherejava0day.com/
wadus1 13y ago
this is incomplete and then bullshit, it's comparing a single-core with a multicore running a single-threaded process.

132 points by j_s 13y ago

>Hardware: Mid-2007 MacBook Pro Wouldn't it make more sense to do this on generic server hardware with a LTS Linux OS? Once that vintage and customized MacBook bites the dust the website pretty much dies as well.

Tloewald 13y ago
There's nothing magical about a Linux server. If you're arguing an LTS edition will run on newer hardware, well sure but the benchmarks will be just as incomparable.
- awinder 13y ago
  You're right that theres nothing magical, but there is a clear advantage to faster disks, a more modern & faster CPU, newer & faster ram (this is running DDR2 in the benchmark), etc. I'd be curious why this particular machine was used for benchmarking, and if the same benchmarking method yielded different results on EC2 or something like that.
  - rschmitty 13y ago
    You should check out http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/ where they test on hardware and in the cloud
- asveikau 13y ago
  What are the odds that the Linux kernel is faster than OS X for server workloads? I don't have any data to back this up but I've heard awful things about their filesystem in particular. (Supposedly they have a global filesystem lock.)

RyanZAG 13y ago

Not sure how comparing different versions of rails shows if its 'fast' yet. I'd think setting up some request target and seeing how far RoR is from that target with each version would be better. Or comparing RoR to other platforms is also a good bet, eg http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/ I just don't really see how this answers the question other than 'ehh I think it looks a bit faster than it was awhile ago?' EDIT: Also interestingly, JRuby seems to be slightly better on the small instance EC2 test at techempower than regular Ruby (small instance should be similar to a laptop?). Yet in these graphs it's much slower. Anybody know the reason?

jarrett 13y ago
Amazon says their "m1.small" instance has 1.7 GiB of memory. That's a lot less than most current laptops. Maybe JRuby uses more memory in total, but can handle more concurrent requests per MB of memory used? That could explain the discrepancy if it's true, though it's just speculation on my part. Another theory is that the benchmarks are too application-specific. Rails is a complex framework with which you can build many different kinds of applications, each with its own performance characteristics. Conceivably, application A could perform better on JRuby, while application B could perform better on MRI. Depending on which application you use for your benchmark, either interpreter could appear faster. This calls into question the utility of benchmarking an entire framework. It might be better to do micro-benchmarks, with each one testing a specific scenario. Scenarios could include, for example: A request that loads 1,000 database records into memory; a request for a static image; a request that queries a many-to-many relationship; a request that saves an uploaded file to disk.
- lukeschlather 13y ago
  The thing is, those are all I/O bound operations and in the case of the databases most of the code you're benchmarking is native extensions, so I would expect the difference to be negligible. Testing whole Rails apps really does seem like the best way to get good data. The interesting speedups are likely going to be in places like string matching / string copying and other places that don't involve network access.

matthuggins 13y ago

Wouldn't it make more sense to test various features of Ruby for comparison rather than comparing an entire web framework? Given that, the URL should be israilsfastyet.com, or more appropriately howdoesrailsbehaveonvariousrubyversions.com

mikesilvis 13y ago
Agreed. This is much more of a test of the rails framework then ruby
- jrochkind1 13y ago
  Well, it's a comparison of different ruby versions, using Rails. That's pretty useful to people who use Rails, which is a lot of people. Of course, it's not a real Rails app presumably, but just a sample/dummy one. So you may or may not see comparable speed-up with a real app. But it's probably suggestive of the upper limit of improvement you can get by switching ruby versions. "Will I get a performance improvement from my Rails app by upgrading ruby" is a real question a lot of people will have. Micro-benchmarks of ruby wouldn't answer it as well as the OP does.
- jeltz 13y ago
  Not really since it does not compare it to another other benchmarks. This is a test of how the performance of MRI ruby has changed over time.
regularfry 13y ago
If you test small features, you get knocked for concentrating on microbenchmarks which don't have any real-world meaning. If you test whole applications or frameworks, you get criticised for benchmarking that rather than the language implementation.
- Liru 13y ago
  The criticism is with the name, not the features tested.

mpweiher 13y ago

tl;dr: no.

michaelhoffman 13y ago
I was expecting the web page to just consist of "No."
- mpweiher 13y ago
  Yep, that was my expectation as well :-)
- theandrewbailey 13y ago
  ...just like https://isitchristmas.com/ https://isitchristmas.com/ on days other than Christmas.

yuvipanda 13y ago

5s startup time for a web framework seems rather high. I wonder how well it compares to similar frameworks in other languages (Django? ASP.NET MVC?)

x1024 13y ago
Eh... no it doesn't.
- briantakita 13y ago
  It is when your development process involves rapidly iterating the code by running the process many times (i.e. TDD). Anecdotally, when the feedback loop is tighter, I'm much happier. Developing (especially refactoring) in Rails feels like watching paint dry, since it's soooooo slooooow. Every time the process needs to restart, I'm left hanging........waiting......for..............the.............. ..........process.......................to.................... ...................................................s.......... ................t.............................a............... ..............r............................................... ..tttt........... Also the benchmark is the best case. In real world projects, I've seen Rails take over 60 seconds to start (it took a few days of development effort to bring it down to ~20 seconds). Every time you rerun tests, run a rake task, start or restart your server, etc. It's like the heartbeat of the development process grinds to a halt when you add more features to your Rails server. And no, tools like Spork are not the answer, since you still have to balance the amount that is resident in memory to fork with coverage of what you can change. It also opens up a fractal of complexity.
  - tawm 13y ago
    Oh god. Please edit your comment so it doesn't break the whole layout.
    - briantakita 13y ago
      No can do. It got downvoted and HN does not allow me to edit negative karma posts. So please upvote my previous post so I can fix the layout of these comments :-) I'm assuming the negative votes are due to the formatting and not the message. I hope you, dear reader, don't miss the irony of this situation... If it's downvotes because of the message, well I feel bad for the downvoters and wish them much luck with using their framework.

Jgrubb 13y ago

JRuby question -- Why is it that much slower? I'd have thought that part of the point was to take advantage of the JVM's built in optimization.

mml 13y ago
My question as well. Every perf benchmark I've seen of jruby over the years, it's much faster than MRI (given time to boot & JIT anyway). I think this guy's numbers are very suspect.
- exabrial 13y ago
  I was about to say the same thing... I thought JRuby was king for long running Ruby apps (like a Rails app) for many years now... seems very suspect.
- jshen 13y ago
  I've benchmarked MRI vs jruby a number of times over the years on real apps and MRI usually cones out faster on average with worse numbers at 95th and 98th percentiles due to GC performance. I never took the time to figure out why it is faster, but I assume that it is due to C libraries being faster for json serialization and my apps doing primarily json serialization after fetching from the database. Edit: I should test the performance of the json libs, which would be easy to do
sluukkonen 13y ago
Looks like it's running with the threadsafe mode off. That would probably give JRuby an edge, as it's able to take advantage of greater thread parallelism than MRI. No idea if he's letting the JVM warm up enough either.
- jrochkind1 13y ago
  Depending on how you are deploying the ruby app. As far as I know, Passenger Enterprise and Puma are the only multi-platform ruby rack app servers that support multi-threaded concurrent request dispatch. There may be other jruby/java-only deployment scenarios that support concurrent request dispatch. Without concurrent request dispatch, jruby's hypothetical greater thread parallelism is unlikely to effect requests-per-second benchmarks. Also, the benchmarking test would have to be willing to issue requests concurrently!

timmorgan 13y ago

I'm surprised at how negative the comments are here. Graphing how different Ruby versions/implementations perform running Rails is very helpful! Glad to see that MRI continues to get faster with every version. Merry Christmas!

jarito 13y ago
The negativity comes from the fact that this benchmark is useless and, at worst, misleading. True benchmarks are hard to get right. Running a bunch of stuff on a laptop isn't sufficient evidence of anything. Kudos to making the benchmark opensource at least, but even from a quick peak, I wouldn't trust this at all. As others have mentioned, JRuby shouldn't be this slow. Odds are the benchmark is throwing away the JIT gains or the configuration is poor. Add that to the fact that there is no discussion of what a 'rails request' is or what version of Java is being used and there seems to be little to learn from this research.
- jrochkind1 13y ago
  I'm not sure I have ever seen a benchmark on HN that wasn't called misleading and useless in the comments. But I agree it is hard to get benchmarks right -- I'm not even sure there is a 'right', as a benchmark will usually be based on certain assumptions that only sometimes apply. (Unless you are benchmarking your own actual production code, in which case you know it's assumptions are valid -- for your own code). So I don't think that's a reason not to share benchmarks. But I _do_ think it's a reason to be really transparent about exactly how the benchmarks were done. Ideally all code should be available on github. Then people can examine your assumptions (including ones you knew you were making and ones you didn't) and modelling, as well as fork or pull request to do benchmarks in some other way to better model the scenarios they care about. Although also in my experience, you get LOTS of people telling you your benchmark is wrong because of X, Y or Z -- sometimes they are correct that it doesn't accurately capture something they are interested in, sometimes they are just confused -- but virtually nobody actually interested in writing code via fork or pull request to capture what they are interested in. Once you put out a benchmark, everyone in the peanut gallery thinks they are entitled to keep demanding you change it in various ways for them.) No matter how much time you spend on it and thought you put into it, publishing a benchmark on HN is probably the easiest way to have lots of people telling you you're an idiot who did things wrong. It almost seems like HN collectively wants to discourage people from running benchmarks -- or from sharing what they found publicly. But it's because doing good benchmarking is so hard -- well, that's exactly we need more people working on it, and sharing how they did it so others can improve upon it, etc, right?

bri3d 13y ago

Without knowing what runtimes and parameters are used it's really hard to tell what's going on in this benchmark; JRuby can see 100% and sometimes more improvement on many workloads by tweaking JVM options. It looks like JRuby is just run under the system JRE (which is what?) with no additional options; that's going to be slow for most workloads. It'd be nice to see a lot more documentation around the runtime configurations and what the Rails app itself is doing.

hornetblack 13y ago
Down the bottom he states that it is running on Osx. The fan is premptively spun up, which gives Jruby a speed boost. It's probably going to run even faster with proper cooling.
hornetblack 13y ago
Down the bottom he states that it is running on Osx. The fan is premptively spun up, which gives Jruby a speed boost. It's probably going to run even faster with proper cooling.
hornetblack 13y ago
Down the bottom he states that it is running on Osx. The fan is premptively spun up, which gives Jruby a speed boost. It's probably going to run even faster with proper cooling.

Freaky 13y ago

> The benchmark premptively runs the fans and pauses between runs in order to cool off. These measures give JRuby at least a 25% speed gain. However, JRuby still might be running slower than it would on a box with appropriate cooling. Right. Your computer can't be trusted to not throttle its performance due to crap cooling, and you're trying to benchmark using it?

miahi 13y ago
The choice of hardware is quite strange. Should be fun to check the results on normal hardware though.

deleted 13y ago

[deleted]

gesman 13y ago

Compare to what? And the answer is no, and unlikely to be any time soon: http://www.techempower.com/benchmarks/#section=data-r8&hw=i7&test=query http://www.techempower.com/benchmarks/#section=data-r8&hw=i7...

astrodust 13y ago

There's great data here that's obscured by choosing terrible colours for the graph and showing things that are pretty much irrelevant. A summary chart giving relevant percentage differences would be a lot better than this line chart.

phaedryx 13y ago

I was expecting Ruby benchmarks, not Rails benchmarks.

w0rd-driven 13y ago

There is one sentence missing from each graph that would've made this a lot easier to grok: "x is better" where x is more requests, faster startup time, or less memory usage. I prefer graphs that lead me to what I should expect and that sentence would go a long way to eliminate the "mental math" I had to do to understand each one. On another note, the 2 year graphs are particularly interesting. They're more consistent than not but those inconsistencies seem a little awkward.

brianhempel 13y ago
The variance is sometimes due to RVM troubles. The benchmark is partly a test of the entire Ruby ecosystem. If RVM is bugging, then people are installing sub-optimal implementations. For example: https://github.com/wayneeseguin/rvm/issues/1418 https://github.com/wayneeseguin/rvm/issues/1418

lowglow 13y ago

[serious] I'm building http://techendo.co/ http://techendo.co/ and it's running rails on heroku. I was wondering if there was some easy ways to get some better performance out of it immediately now that we're picking up traffic. If anyone could give me some pointers or link me to a page I'd appreciate it.

RyanZAG 13y ago
Going off the techempower benchmarks, looks like you should be able to get some speedup by switching to TorqueBox. No guarantees, but if you've got the time and need the performance it could be worth testing. Here seems to be a decent guide you could follow: https://gist.github.com/bbrowning/4296297 https://gist.github.com/bbrowning/4296297
- lowglow 13y ago
  does it matter that I'm not using JRuby?
  - RyanZAG 13y ago
    You'd have to convert your gems over - you need jruby gems rather than standard ones, but apparently most gems are ported. You can read up here for more: https://devcenter.heroku.com/articles/moving-an-existing-rails-app-to-run-on-jruby https://devcenter.heroku.com/articles/moving-an-existing-rai... http://stackoverflow.com/questions/151595/jruby-on-rails-vs-ruby-on-rails-whats-difference http://stackoverflow.com/questions/151595/jruby-on-rails-vs-... So it's not just a 5 second thing, but if you have a bit of time and need performance it's something you can try.
    - lowglow 13y ago
      hm. I guess I'm not ready to switch to jruby.
awj 13y ago
That's an incredibly broad question, so it's pretty hard to answer. Here's some general ones: * More instances will give you more requests/second, but only so long as secondary system (database, search server, whatever) can keep up with the load * N+1 queries will kill your performance. They can add a lot of time to processing requests. * Everything in the most used parts of your site should hit an index on the database. If it's not I can almost guarantee it will be slow. * Look for work that you can move "offline". As long as they go out within a minute nobody will notice that you aren't sending emails at the exact time they are being generated. Move that work to a background queue so your web server can just add a queue entry and get on to the next request. * Cache anything you reasonably can. It's a lot cheaper (i.e. faster) to read from a cache than to do a database lookup. It's cheaper to read from a cache than to render an ERB template. Caching can introduce its own difficulties, but the performance gains can be worth it. Without knowing some specifics of your implementation I really can't give you better advice. Depending on the specific libraries you use it might not even be possible to move to JRuby due to support for C extensions. You might get some performance benefits out of it, but I'd expect it to only buy you some time to execute on some of the above improvements.

saalweachter 13y ago

If I understand this correctly, this is running periodically on someone's Macbook; people are complaining that's a terrible platform for a benchmark. It should be possible (if you care enough) to re-run this from the source control history on platforms of your choosing, across whatever date ranges you wish. Anyone really, really care enough?

polysics 13y ago

As a non-Rails Rubyist, I will have to run benchmarks myself as usual, because people seem to be still equating Ruby to Rails.

noelwelsh 13y ago

If you're going to call it "Is Ruby Fast Yet", you should really be comparing Ruby implementations to implementations of other programming languages. Asked this way, I believe the answer to the question is a definite "no". As a small example, I benchmarked a real Scala web app running on worse hardware (a 1GB Linux VM) at over 650 requests/s. There was no degradation in load at this rate of requests, but it was 10x the load we saw in production so I stopped there. The benchmark of Rails here goes to ~120 requests/s.

warmwaffles 13y ago
Well there is a JRuby and Rubinius comparison. Those are different implementations of the the interpreter. Ruby will never be faster than C, but this site is aiming to show the improvements over time.
brianhempel 13y ago
Comparing across languages was originally part of the long-term goal. However, I haven't yet come across any good, multi-language, real-world benchmark.

lazugod 13y ago

Are web fonts fast yet?

bernardom 13y ago

I was expecting a one-time-use site such as http://www.whatwouldmyharvardgrade.be/ http://www.whatwouldmyharvardgrade.be/

davidw 13y ago

I don't like those charts at all. I want to see the progressive Ruby versions on the X axis, I think. The dates are pretty meaningless. If it's the same code, it should run in the same number of cycles; anything else is just noise. That said, +1 for having a go at it! Also, yes, Ruby is pretty slow. But it's fast for getting something up and running quickly. If you get wildly successful, you can always swap it out for something else if you need to.

sams99 13y ago

FYI we are also working on a similar project see: http://samsaffron.com/archive/2013/12/11/call-to-action-long-running-ruby-benchmark http://samsaffron.com/archive/2013/12/11/call-to-action-long... and http://community.miniprofiler.com/t/is-ruby-fast-yet/197 http://community.miniprofiler.com/t/is-ruby-fast-yet/197

kbar13 13y ago

benchmarking a web framework on an old macbook pro?

shocks 13y ago

Yes, if we're talking about development time! :P

spoiler 13y ago

We can at least see MRI has gotten faster with each new version, but we knew that already. Also, I'm a bit surprised that this doesn't include more implementations of Ruby. Where's Rubinius? IronRuby? REE? I hope the list gets updated. Furthermore, the benchmark itself could be better; I won't say I am disappointed, but I am not too thrilled either.

munificent 13y ago

I've been hacking on a little scripting language of my own lately[1]. To get a handle on its perf, I have a few small benchmarks[2] I run that compare it against Lua (5.1.5), Python (2.7.5), and Ruby (2.0.0p247). (All of those and my language are bytecode-compiled, dynamically typed languages, hence that comparison.) I happen to have Ruby 2.0 installed, and I was surprised to see it handily beat the other languages. I had expected Ruby to be the slowest of the pack, but they've apparently made big strides in 2.0. On the off chance that anyone is curious, here's the results of running them on my machine: binary_trees - wren 3531 0.28s binary_trees - lua 1366 0.73s 258.40% binary_trees - python 1420 0.70s 248.64% binary_trees - ruby 3170 0.32s 111.39% fib - wren 2688 0.37s fib - lua 2998 0.33s 89.65% fib - python 1390 0.72s 193.38% fib - ruby 3645 0.27s 73.74% for - wren 9092 0.11s for - lua 9387 0.11s 96.86% for - python 3171 0.32s 286.74% for - ruby 8319 0.12s 109.30% method_call - wren 4703 0.21s method_call - lua 1780 0.56s 264.20% method_call - python 806 1.24s 583.33% method_call - ruby 3408 0.29s 138.02% Those numbers are score, time, and percentage. Score is basically 1/time, just to yield a number where bigger is better. Time is best of ten runs. Percentage is how my language's score compares to that one. So, for example, the 264.20% lua method_call score means my language is about 2.6x faster than Lua on a benchmark designed to stress dynamic method call performance. [1] https://github.com/munificent/wren https://github.com/munificent/wren [2] https://github.com/munificent/wren/tree/master/benchmark https://github.com/munificent/wren/tree/master/benchmark

StefanKarpinski 13y ago
Oh, man, this reminds me of the pain of writing even the most trivial microbenchmarks in seven or eight languages. How about throwing in some of the fast scripting languages for healthy competition? E.g. JavaScript, LuaJIT and Julia. Feel free to borrow (MIT licensed, but I'm willing to relicense whatever I can): https://github.com/JuliaLang/julia/tree/master/test/perf/micro https://github.com/JuliaLang/julia/tree/master/test/perf/mic...
- munificent 13y ago

fantnn 13y ago

Was expecting one of those generic yes/no sites, a la http://istherejava0day.com/ http://istherejava0day.com/

wadus1 13y ago

this is incomplete and then bullshit, it's comparing a single-core with a multicore running a single-threaded process.