5 ms·

So with the promised 3x speed improvement (hooray!), where does Ruby stand with regard to performance, compared to say Python? According to [this benchmark][1],

So with the promised 3x speed improvement (hooray!), where does Ruby stand with regard to performance, compared to say Python? According to [this benchmark][1], it seems (the 3.0.0-preview1 version was) on average about 2x slower than Python. Are there other benchmarks to compare with? Looking into it, this other benchmark [2] of 3.0.0-preview1 vs Ruby 2.7 actually concludes there's no real performance improvement there. So I guess an additional question is, is there actually a 3x speed increase in realistic benchmarks? [1] https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/yarv-python3.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... [2] https://www.fastruby.io/blog/rails/performance/ruby/hows-the-performance-of-ruby-3.0.0-preview1.html https://www.fastruby.io/blog/rails/performance/ruby/hows-the...

pansa2 6y ago
> where does Ruby stand with regard to performance, compared to say Python? In my experience Ruby has been faster than Python for a long time - ever since its performance improved significantly going from 1.8 to 1.9, and Python’s got worse going from 2.x to 3.x.
- thirdusername 6y ago
  Python 3.6 was pretty much on par with 2.7 and that was 5 years ago. There doesn't seem to be any meaningful difference between python and ruby these days at all https://benchmarksgame-team.pages.debian.net/benchmarksgame/index.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
Rochus 6y ago
I did some measurements recently with 3.0.0 RC1 compared to 2.1.0 based on https://github.com/smarr/are-we-fast-yet https://github.com/smarr/are-we-fast-yet. Here are the results: http://software.rochus-keller.ch/are-we-fast-yet_crystal_ruby_2_3_lua_node_i386_results_2020-12-23.pdf http://software.rochus-keller.ch/are-we-fast-yet_crystal_rub.... Ruby 3.0 without JIT is factor 1.2 faster than Ruby 2.1.0; Ruby 3.0 with JIT is only factor 1.4 faster than Ruby 2.1.0. Crystal is about factor 20 faster.
- eregon 6y ago
  Interesting results, thanks for sharing, that matches my expectations. OptCarrot is probably one of the rare cases where Ruby 3.0+JIT is 3x Ruby 2.0. On most benchmarks, the gains are much smaller. I'll try to run these benchmarks on TruffleRuby (and maybe JRuby) too, would be an interesting comparison. BTW, is there any reason you used 2.1.0 and not 2.0.0? Maybe some issue with 2.0.0?
  - Rochus 6y ago
    > OptCarrot is probably one of the rare cases where Ruby 3.0+JIT is 3x Ruby 2.0 The Are-we-fast-yet suite is optimized for inter-language comparisons and sets the focus on ideomatic language use and representative algorithms. There is a peer-reviewed scientific publication; here is a preprint: https://stefan-marr.de/papers/dls-marr-et-al-cross-language-compiler-benchmarking-are-we-fast-yet/ https://stefan-marr.de/papers/dls-marr-et-al-cross-language-... > is there any reason you used 2.1.0 and not 2.0.0? The 2.0 version produced errors like "uninitialized constant Process::CLOCK_MONOTONIC (NameError)"; from that I concluded that it is not compatible. I still had to apply a fix (https://bugs.ruby-lang.org/issues/9578 https://bugs.ruby-lang.org/issues/9578) to make Ruby 2.1.0 compile on my test machine. Here is a report version where I compare with Ruby 2.7 instead of 2.1: http://software.rochus-keller.ch/are-we-fast-yet_crystal_ruby_lua_node_i386_results_2020-12-20.pdf http://software.rochus-keller.ch/are-we-fast-yet_crystal_rub...
    - jashmatthews 6y ago
      > The Are-we-fast-yet suite is optimized for inter-language comparisons and sets the focus on ideomatic language use and representative algorithms. There is a peer-reviewed scientific publication; here is a preprint: https://stefan-marr.de/papers/dls-marr-et-al-cross-language- https://stefan-marr.de/papers/dls-marr-et-al-cross-language-... Benoit (eregon) is the second author of that paper.
      - Rochus 6y ago
        Well, apparently he does not want to be recognized; without your hint I would not have figured it out (there is also no indication in the profile). At least I don't have to convince him of the benchmark's quality. EDIT: surprised to learn that his group is responsible for the GraalVM and even located in Zurich (https://labs.oracle.com/pls/apex/f?p=LABS:project_details:0:47 https://labs.oracle.com/pls/apex/f?p=LABS:project_details:0:...) which is not far from the place I live; it's a small world ;-)
fouc 6y ago
the Ruby 2.7 benchmarks [0] seem to shows that Ruby is slightly faster than Python 3, or at least fairly competitive. Though things like Python 3 going from 18.45 seconds to 2.67 seconds on the regex-redux test [1][2] tends to make me question the validity/usefulness of these benchmarks. (inasmuch - how much of it is testing language performance vs bypassing it?) [0] https://web.archive.org/web/20200301005643/https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/yarv-python3.html https://web.archive.org/web/20200301005643/https://benchmark... [1] https://web.archive.org/web/20191225153009/https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/regexredux.html https://web.archive.org/web/20191225153009/https://benchmark... [2] https://web.archive.org/web/20200604134902/https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/regexredux.html https://web.archive.org/web/20200604134902/https://benchmark...
- p_l 6y ago
  Ruby version of regex-redux uses built-in regex support, and I'm not even sure if it uses it the most performant way. Significant portion of the Python test case is FFI code to directly link PCRE bypassing even normal Python extension support. I guess I should augment some of the SBCL benchmarks with custom assembly then...
  - igouy 6y ago
    > …SBCL benchmarks with custom assembly then… How could that possibly effect the performance of the Ruby regex-redux? :-) "Wanted — Ruby PCRE2 regex-redux program" https://salsa.debian.org/benchmarksgame-team/benchmarksgame/-/issues/374 https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...
    - p_l 6y ago
      It wouldn't help ruby, but I'd rather write common lisp ;)
      - igouy 6y ago
        Fair enough :-) (Although they wouldn't be accepted — the option of using those widely libraries is only for pi-digits and regex-redux; and not "custom assembly".)
joshspankit 6y ago
If you’re interested in speed, check this repo: even among Ruby idioms there are big speed differences. https://github.com/JuanitoFatas/fast-ruby https://github.com/JuanitoFatas/fast-ruby
peteforde 6y ago
The promised gains were (basically) achieved: 1. on the optcarrot benchmark 2. between Ruby 2.0.0 and Ruby 3.0.0 The speed difference between 2.7 and 3.0 is completely moot.
igouy 6y ago
> 2x slower than Python Several multi-core Ruby programs failed when run with 3.0.0-preview1 --jit Which is why you see Python3 programs using 4 cores for spectral-norm and binary-trees and fannkuch-redux — when the corresponding Ruby program is only using one core. https://benchmarksgame-team.pages.debian.net/benchmarksgame/measurements/yarv.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... It’s multi-core age today. Concurrency is very important. With Ractor, along with Async Fiber, Ruby will be a real concurrent language. — Matz We need new Ruby programs for the benchmarks game, to show that new concurrent language.
ziotom78 6y ago
The few times I tested, Ruby 2 was ~2x faster than Python 3 in tasks involving iterations on long lists.

pansa2 6y ago

> where does Ruby stand with regard to performance, compared to say Python? In my experience Ruby has been faster than Python for a long time - ever since its performance improved significantly going from 1.8 to 1.9, and Python’s got worse going from 2.x to 3.x.

thirdusername 6y ago
Python 3.6 was pretty much on par with 2.7 and that was 5 years ago. There doesn't seem to be any meaningful difference between python and ruby these days at all https://benchmarksgame-team.pages.debian.net/benchmarksgame/index.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Rochus 6y ago

I did some measurements recently with 3.0.0 RC1 compared to 2.1.0 based on https://github.com/smarr/are-we-fast-yet https://github.com/smarr/are-we-fast-yet. Here are the results: http://software.rochus-keller.ch/are-we-fast-yet_crystal_ruby_2_3_lua_node_i386_results_2020-12-23.pdf http://software.rochus-keller.ch/are-we-fast-yet_crystal_rub.... Ruby 3.0 without JIT is factor 1.2 faster than Ruby 2.1.0; Ruby 3.0 with JIT is only factor 1.4 faster than Ruby 2.1.0. Crystal is about factor 20 faster.

eregon 6y ago
Interesting results, thanks for sharing, that matches my expectations. OptCarrot is probably one of the rare cases where Ruby 3.0+JIT is 3x Ruby 2.0. On most benchmarks, the gains are much smaller. I'll try to run these benchmarks on TruffleRuby (and maybe JRuby) too, would be an interesting comparison. BTW, is there any reason you used 2.1.0 and not 2.0.0? Maybe some issue with 2.0.0?
- Rochus 6y ago
  > OptCarrot is probably one of the rare cases where Ruby 3.0+JIT is 3x Ruby 2.0 The Are-we-fast-yet suite is optimized for inter-language comparisons and sets the focus on ideomatic language use and representative algorithms. There is a peer-reviewed scientific publication; here is a preprint: https://stefan-marr.de/papers/dls-marr-et-al-cross-language-compiler-benchmarking-are-we-fast-yet/ https://stefan-marr.de/papers/dls-marr-et-al-cross-language-... > is there any reason you used 2.1.0 and not 2.0.0? The 2.0 version produced errors like "uninitialized constant Process::CLOCK_MONOTONIC (NameError)"; from that I concluded that it is not compatible. I still had to apply a fix (https://bugs.ruby-lang.org/issues/9578 https://bugs.ruby-lang.org/issues/9578) to make Ruby 2.1.0 compile on my test machine. Here is a report version where I compare with Ruby 2.7 instead of 2.1: http://software.rochus-keller.ch/are-we-fast-yet_crystal_ruby_lua_node_i386_results_2020-12-20.pdf http://software.rochus-keller.ch/are-we-fast-yet_crystal_rub...
  - jashmatthews 6y ago
    > The Are-we-fast-yet suite is optimized for inter-language comparisons and sets the focus on ideomatic language use and representative algorithms. There is a peer-reviewed scientific publication; here is a preprint: https://stefan-marr.de/papers/dls-marr-et-al-cross-language- https://stefan-marr.de/papers/dls-marr-et-al-cross-language-... Benoit (eregon) is the second author of that paper.
    - Rochus 6y ago
      Well, apparently he does not want to be recognized; without your hint I would not have figured it out (there is also no indication in the profile). At least I don't have to convince him of the benchmark's quality. EDIT: surprised to learn that his group is responsible for the GraalVM and even located in Zurich (https://labs.oracle.com/pls/apex/f?p=LABS:project_details:0:47 https://labs.oracle.com/pls/apex/f?p=LABS:project_details:0:...) which is not far from the place I live; it's a small world ;-)

fouc 6y ago

the Ruby 2.7 benchmarks [0] seem to shows that Ruby is slightly faster than Python 3, or at least fairly competitive. Though things like Python 3 going from 18.45 seconds to 2.67 seconds on the regex-redux test [1][2] tends to make me question the validity/usefulness of these benchmarks. (inasmuch - how much of it is testing language performance vs bypassing it?) [0] https://web.archive.org/web/20200301005643/https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/yarv-python3.html https://web.archive.org/web/20200301005643/https://benchmark... [1] https://web.archive.org/web/20191225153009/https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/regexredux.html https://web.archive.org/web/20191225153009/https://benchmark... [2] https://web.archive.org/web/20200604134902/https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/regexredux.html https://web.archive.org/web/20200604134902/https://benchmark...

p_l 6y ago
Ruby version of regex-redux uses built-in regex support, and I'm not even sure if it uses it the most performant way. Significant portion of the Python test case is FFI code to directly link PCRE bypassing even normal Python extension support. I guess I should augment some of the SBCL benchmarks with custom assembly then...
- igouy 6y ago
  > …SBCL benchmarks with custom assembly then… How could that possibly effect the performance of the Ruby regex-redux? :-) "Wanted — Ruby PCRE2 regex-redux program" https://salsa.debian.org/benchmarksgame-team/benchmarksgame/-/issues/374 https://salsa.debian.org/benchmarksgame-team/benchmarksgame/...
  - p_l 6y ago
    It wouldn't help ruby, but I'd rather write common lisp ;)
    - igouy 6y ago
      Fair enough :-) (Although they wouldn't be accepted — the option of using those widely libraries is only for pi-digits and regex-redux; and not "custom assembly".)

joshspankit 6y ago

If you’re interested in speed, check this repo: even among Ruby idioms there are big speed differences. https://github.com/JuanitoFatas/fast-ruby https://github.com/JuanitoFatas/fast-ruby

peteforde 6y ago

The promised gains were (basically) achieved: 1. on the optcarrot benchmark 2. between Ruby 2.0.0 and Ruby 3.0.0 The speed difference between 2.7 and 3.0 is completely moot.

igouy 6y ago

> 2x slower than Python Several multi-core Ruby programs failed when run with 3.0.0-preview1 --jit Which is why you see Python3 programs using 4 cores for spectral-norm and binary-trees and fannkuch-redux — when the corresponding Ruby program is only using one core. https://benchmarksgame-team.pages.debian.net/benchmarksgame/measurements/yarv.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... It’s multi-core age today. Concurrency is very important. With Ractor, along with Async Fiber, Ruby will be a real concurrent language. — Matz We need new Ruby programs for the benchmarks game, to show that new concurrent language.

ziotom78 6y ago

The few times I tested, Ruby 2 was ~2x faster than Python 3 in tasks involving iterations on long lists.