4 ms·

In case someone is interested in the (JIT) performance of 3.0 RC 1 compared to version 2.7.2, here are some measurement results based on the are-we-fast-yet ben

by Rochus 6y ago

In case someone is interested in the (JIT) performance of 3.0 RC 1 compared to version 2.7.2, here are some measurement results based on the are-we-fast-yet benchmark suite: http://software.rochus-keller.ch/are-we-fast-yet_crystal_ruby_lua_node_i386_results_2020-12-20.pdf http://software.rochus-keller.ch/are-we-fast-yet_crystal_rub... and http://software.rochus-keller.ch/are-we-fast-yet_crystal_ruby_lua_node_i386_results_2020-12-10.pdf http://software.rochus-keller.ch/are-we-fast-yet_crystal_rub.... I see only a slight speedup of the 3.0 (JIT) compared to 2.7 (less than 15%). See here for more information about the benchmark suite: https://stefan-marr.de/papers/dls-marr-et-al-cross-language-compiler-benchmarking-are-we-fast-yet/ https://stefan-marr.de/papers/dls-marr-et-al-cross-language-... and https://github.com/smarr/are-we-fast-yet https://github.com/smarr/are-we-fast-yet. In contrast to other benchmarks the focus is on ideomatic language use and representative results (i.e. not just some random micro benchmarks) suited for inter-language comparisons.

desireco42 6y ago
What I got from this is how fast Crystal is :).
- vinceguidry 6y ago
  If you want to split the difference there's compiled mruby. http://mruby.org/docs/articles/executing-ruby-code-with-mruby.html http://mruby.org/docs/articles/executing-ruby-code-with-mrub...
- Rochus 6y ago
  Crystal is a very interesting technology indeed and with my 2020-12-10 and 2020-12-15 measurements I was mainly interested how fast the compiled binary and the compiler work in comparison. With the 2020-12-20 measurement though I was interested whether MRI Ruby 3.0 was indeed significantly faster than 2.7 which doesn't seem to be the case. It's also interesting to note that even if MRI Ruby was tree times faster it still would be half as fast as LuaJIT (considering that LuaJIT would be even faster if the tracing JIT would also support closures).
oblio 6y ago
15% is a decent speedup.
- Rochus 6y ago
  When implementing a JIT I would hope for a much higher speed up than 15%. In comparison: LuaJIT with the JIT switched on runs about five times faster than the most recent PUC Lua interpreter, see http://software.rochus-keller.ch/are-we-fast-yet_lua_results_2020-10-12.pdf http://software.rochus-keller.ch/are-we-fast-yet_lua_results.... Compared to that 15% is nothing.
  - oblio 6y ago
    True, but I would imagine that a lot more time has been invested in LuaJIT, since this is just the first release of the Ruby JIT. Plus isn't Lua better suited for optimization, from the get-go?
    - Rochus 6y ago
      I have no information on how much time was invested in the implementation of either of the VMs. Lua has many similarities both with Smalltalk and Ruby. It has also a very powerful closure concept, which unfortunately is not yet supported by the tracing JIT.
  - mercer 6y ago
    The JIT efforts for Elixir/Erlang apparently offer 30% speed up, worst case, and while I'm way out of my depth here, apparently it's a relatively 'naive' implementation. I'm very curious why there isn't as much of an improvement for Ruby. Is it something to do with the extreme dynamic nature of the language?
    - Rochus 6y ago
      > Is it something to do with the extreme dynamic nature of the language? Smalltalk/SOM, JavaScript and Lua share the "extreme dynamic nature" but still have much faster JIT implementations. From my reports you can see that V8 (Node) is about ten times and LuaJIT about five times faster than MRI Ruby 3 JIT. Here are some other performance results using the same benchmark suite: https://github.com/smarr/are-we-fast-yet/blob/master/docs/performance.md https://github.com/smarr/are-we-fast-yet/blob/master/docs/pe.... Also TruffleRuby is reported to perform much better than MRI Ruby JIT (see e.g. https://pragtob.wordpress.com/2020/08/24/the-great-rubykon-benchmark-2020-cruby-vs-jruby-vs-truffleruby/ https://pragtob.wordpress.com/2020/08/24/the-great-rubykon-b...).
Rochus 6y ago
Meanwhile I was able to run the benchmark on MRI Ruby 2.1.0 and compare with 3.0.0 RC1; here are the results: http://software.rochus-keller.ch/are-we-fast-yet_crystal_ruby_2_3_lua_node_i386_results_2020-12-23.pdf http://software.rochus-keller.ch/are-we-fast-yet_crystal_rub... Ruby 3.0 RC1 without JIT is factor 1.2 faster than Ruby 2.1.0; Ruby 3.0 with JIT is factor 1.4 faster than Ruby 2.1.0. So definitely less than factor 3. Unfortunately the benchmark didn't work with Ruby 2.0.

by Rochus 6y ago

desireco42 6y ago

What I got from this is how fast Crystal is :).

vinceguidry 6y ago
If you want to split the difference there's compiled mruby. http://mruby.org/docs/articles/executing-ruby-code-with-mruby.html http://mruby.org/docs/articles/executing-ruby-code-with-mrub...
Rochus 6y ago
Crystal is a very interesting technology indeed and with my 2020-12-10 and 2020-12-15 measurements I was mainly interested how fast the compiled binary and the compiler work in comparison. With the 2020-12-20 measurement though I was interested whether MRI Ruby 3.0 was indeed significantly faster than 2.7 which doesn't seem to be the case. It's also interesting to note that even if MRI Ruby was tree times faster it still would be half as fast as LuaJIT (considering that LuaJIT would be even faster if the tracing JIT would also support closures).

oblio 6y ago

15% is a decent speedup.

Rochus 6y ago
When implementing a JIT I would hope for a much higher speed up than 15%. In comparison: LuaJIT with the JIT switched on runs about five times faster than the most recent PUC Lua interpreter, see http://software.rochus-keller.ch/are-we-fast-yet_lua_results_2020-10-12.pdf http://software.rochus-keller.ch/are-we-fast-yet_lua_results.... Compared to that 15% is nothing.
- oblio 6y ago
  True, but I would imagine that a lot more time has been invested in LuaJIT, since this is just the first release of the Ruby JIT. Plus isn't Lua better suited for optimization, from the get-go?
  - Rochus 6y ago
    I have no information on how much time was invested in the implementation of either of the VMs. Lua has many similarities both with Smalltalk and Ruby. It has also a very powerful closure concept, which unfortunately is not yet supported by the tracing JIT.
- mercer 6y ago
  The JIT efforts for Elixir/Erlang apparently offer 30% speed up, worst case, and while I'm way out of my depth here, apparently it's a relatively 'naive' implementation. I'm very curious why there isn't as much of an improvement for Ruby. Is it something to do with the extreme dynamic nature of the language?
  - Rochus 6y ago
    > Is it something to do with the extreme dynamic nature of the language? Smalltalk/SOM, JavaScript and Lua share the "extreme dynamic nature" but still have much faster JIT implementations. From my reports you can see that V8 (Node) is about ten times and LuaJIT about five times faster than MRI Ruby 3 JIT. Here are some other performance results using the same benchmark suite: https://github.com/smarr/are-we-fast-yet/blob/master/docs/performance.md https://github.com/smarr/are-we-fast-yet/blob/master/docs/pe.... Also TruffleRuby is reported to perform much better than MRI Ruby JIT (see e.g. https://pragtob.wordpress.com/2020/08/24/the-great-rubykon-benchmark-2020-cruby-vs-jruby-vs-truffleruby/ https://pragtob.wordpress.com/2020/08/24/the-great-rubykon-b...).

Rochus 6y ago

Meanwhile I was able to run the benchmark on MRI Ruby 2.1.0 and compare with 3.0.0 RC1; here are the results: http://software.rochus-keller.ch/are-we-fast-yet_crystal_ruby_2_3_lua_node_i386_results_2020-12-23.pdf http://software.rochus-keller.ch/are-we-fast-yet_crystal_rub... Ruby 3.0 RC1 without JIT is factor 1.2 faster than Ruby 2.1.0; Ruby 3.0 with JIT is factor 1.4 faster than Ruby 2.1.0. So definitely less than factor 3. Unfortunately the benchmark didn't work with Ruby 2.0.