4 ms·

* As a very crude overview of performance, JRuby+Truffle with the latest development version of GraalVM across 63 benchmarks is over 31x faster than MRI 2.3.0,

* As a very crude overview of performance, JRuby+Truffle with the latest development version of GraalVM across 63 benchmarks is over 31x faster than MRI 2.3.0, over 32x faster than Rubinius 2.5.8, over 17x faster than Topaz (Ruby using PyPy technology) and over 24x faster than JRuby 9.0.4.0 with invokedynamic * I haven't looked at the benchmarks, but are they synthetic or are they actually relevant to typical rails usage?

chrisseaton 11y ago
We've focused on benchmarks that do metaprogramming and use lots of small intermediate hashes and arrays as we think this is relevant for Rails. See our Acid Test benchmark for an exteme example http://chrisseaton.com/rubytruffle/pushing-pixels/ http://chrisseaton.com/rubytruffle/pushing-pixels/ Evan Phoenix recently said at RubyKaigi that a weaker version of this benchmark is the standard that Ruby 3 should be aiming at.
- sandGorgon 11y ago
  Your help page states that this does not support gems yet - so I guess a running Rails app has not been tested yet? Would love to try it out if you did.
  - nirvdrum 11y ago
    Correct. We do not currently support RubyGems. It's something we're working towards, but it's a fairly complex project and not having OpenSSL support currently limits the ability to install gems quite a bit. However, we do ship with a tool that will install gems using JRuby without Truffle and then running your script with a the $LOAD_PATH set up appropriately. Please see: https://github.com/jruby/jruby/blob/master/lib/ruby/truffle/jruby%2Btruffle/README.md https://github.com/jruby/jruby/blob/master/lib/ruby/truffle/... Re: Rails ... there's still a fair bit of work involved there. We've been working on passing all the ActiveSupport tests (we're currently at 99%), as that's a core dependency. We haven't looked much into the other gems. Things like ActiveRecord simply won't work since we currently don't run C or Java extensions. I think it's a bit more likely we'll start with a custom driver of some sort with an ActiveModel front-end. I strongly suspect the asset pipeline and Spring will present problems, as well. The rest of Rails should pull together somewhat quickly and that's a big goal for us in 2016.
- igouy 11y ago
  >> … almost 6x faster … over 50x faster … We think this shows how can be optimise more effectively on more realistic Ruby code than synthetic benchmarks.<< Might the difference simply be test coverage? -- The other Ruby implementations have been testing performance on those same synthetic benchmarks, and have already taken the opportunity to improve performance for those cases. -- The other Ruby implementations have not been testing performance in other cases, and still have considerable opportunity to improve performance for those cases.
  - chrisseaton 11y ago
    I think the difference is that the synthetic benchmarks are generally written in a way that is as tight as possible, avoids allocation and abstraction, and they certainly don't use metaprogramming. That stuff is easier for everyone to optimise. Real Ruby code uses a lot of abstraction, allocates objects constantly, and uses metaprogramming. Optimising these aspects of Ruby is much more complex and doing it well requires some optimisations such as partial escape analysis and powerful allocation removal that we have and JRuby and Rubinius do not. My favourite example is this code from PSD.rb that implements a clamp routine. It does it by creating an array, sorting and finding the middle value. You wouldn't normally find code like this in a synthetic benchmark, but you would in real code. def clamp(value, min, max) [value, min, max].sort[1] end In JRuby and Rubinius that code really will allocate an array, sort it using some library routine, and then index it. In JRuby+Truffle we compile that method to effectively: def clamp(value, min, max) (value > max) ? max : ((value < min) ? min : value); end There's a massive massive difference between those two. One allocates objects on the heap, passes them into the runtime, runs a general purpose sort routine etc etc etc, thousands of machine instructions, and the other is a just couple of assembly instructions. When you run this code as a benchmark, we're over 300x faster than Rubinius' LLVM-based JIT. Of course we still support if someone has redefined Array#sort or something like that, and you could still find that Array instance using ObjectSpace if you wanted to, using deoptimisation.
    - igouy 11y ago
      >>… that we have and JRuby and Rubinius do not.<< Do JRuby and Rubinius even have performance tests that cover those aspects of Ruby? (I don't track Ruby implementation, I don't know the answer.)

chrisseaton 11y ago

We've focused on benchmarks that do metaprogramming and use lots of small intermediate hashes and arrays as we think this is relevant for Rails. See our Acid Test benchmark for an exteme example http://chrisseaton.com/rubytruffle/pushing-pixels/ http://chrisseaton.com/rubytruffle/pushing-pixels/ Evan Phoenix recently said at RubyKaigi that a weaker version of this benchmark is the standard that Ruby 3 should be aiming at.

sandGorgon 11y ago
Your help page states that this does not support gems yet - so I guess a running Rails app has not been tested yet? Would love to try it out if you did.
- nirvdrum 11y ago
  Correct. We do not currently support RubyGems. It's something we're working towards, but it's a fairly complex project and not having OpenSSL support currently limits the ability to install gems quite a bit. However, we do ship with a tool that will install gems using JRuby without Truffle and then running your script with a the $LOAD_PATH set up appropriately. Please see: https://github.com/jruby/jruby/blob/master/lib/ruby/truffle/jruby%2Btruffle/README.md https://github.com/jruby/jruby/blob/master/lib/ruby/truffle/... Re: Rails ... there's still a fair bit of work involved there. We've been working on passing all the ActiveSupport tests (we're currently at 99%), as that's a core dependency. We haven't looked much into the other gems. Things like ActiveRecord simply won't work since we currently don't run C or Java extensions. I think it's a bit more likely we'll start with a custom driver of some sort with an ActiveModel front-end. I strongly suspect the asset pipeline and Spring will present problems, as well. The rest of Rails should pull together somewhat quickly and that's a big goal for us in 2016.
igouy 11y ago
>> … almost 6x faster … over 50x faster … We think this shows how can be optimise more effectively on more realistic Ruby code than synthetic benchmarks.<< Might the difference simply be test coverage? -- The other Ruby implementations have been testing performance on those same synthetic benchmarks, and have already taken the opportunity to improve performance for those cases. -- The other Ruby implementations have not been testing performance in other cases, and still have considerable opportunity to improve performance for those cases.
- chrisseaton 11y ago
  I think the difference is that the synthetic benchmarks are generally written in a way that is as tight as possible, avoids allocation and abstraction, and they certainly don't use metaprogramming. That stuff is easier for everyone to optimise. Real Ruby code uses a lot of abstraction, allocates objects constantly, and uses metaprogramming. Optimising these aspects of Ruby is much more complex and doing it well requires some optimisations such as partial escape analysis and powerful allocation removal that we have and JRuby and Rubinius do not. My favourite example is this code from PSD.rb that implements a clamp routine. It does it by creating an array, sorting and finding the middle value. You wouldn't normally find code like this in a synthetic benchmark, but you would in real code. def clamp(value, min, max) [value, min, max].sort[1] end In JRuby and Rubinius that code really will allocate an array, sort it using some library routine, and then index it. In JRuby+Truffle we compile that method to effectively: def clamp(value, min, max) (value > max) ? max : ((value < min) ? min : value); end There's a massive massive difference between those two. One allocates objects on the heap, passes them into the runtime, runs a general purpose sort routine etc etc etc, thousands of machine instructions, and the other is a just couple of assembly instructions. When you run this code as a benchmark, we're over 300x faster than Rubinius' LLVM-based JIT. Of course we still support if someone has redefined Array#sort or something like that, and you could still find that Array instance using ObjectSpace if you wanted to, using deoptimisation.
  - igouy 11y ago
    >>… that we have and JRuby and Rubinius do not.<< Do JRuby and Rubinius even have performance tests that cover those aspects of Ruby? (I don't track Ruby implementation, I don't know the answer.)