19 ms·

Benchmarking CRuby, MJIT, YJIT, JRuby, and TruffleRuby

91 points by shelajev 5y ago

sparker72678 5y ago
I keep forgetting just how fast TruffleRuby is.
- karmicthreat 5y ago
  What is the downside to TruffleRuby? Memory consumption since its JVM based?
  - kenhwang 5y ago
    Very subtle differences in behavior. It's fine if you start writing your project targeting TruffleRuby, but its a nightmare porting a massive existing ruby codebase.
    - eregon 5y ago
      Compatibility is certainly an issue for massive codebases, OTOH I think over time TruffleRuby is getting closer and closer to CRuby behavior to the point it would be fairly rare to find a compatibility issue. Do you have personal experience trying to run such a codebase on TruffleRuby?
      - kenhwang 5y ago
        We noticed many numerical operations sometimes didn't return the same class as cruby (Fixnum/Bignum/BigDecimal), sometimes resulting in wildly different calculation results. Numerical to string formatting is a bit different too. Which makes the above problem even worse, compounded by how frequently ruby web code goes string to numbers to string to numbers. Regexes behave differently if they worked at all.
        eregon 5y ago
        When did you try? Fixnum/Bignum are gone since many years (before 1.0RC1 which was April 2018, it's all Integer now). So I guess many years ago, and back then TruffleRuby was basically implementing Ruby features, had fairly limited compatibility and likely could not run Rails, very different from today. There were compatibility issues with BigDecimal, but TruffleRuby now uses the C extension, hence it should be exactly the same behavior as CRuby. > Numerical to string formatting is a bit different too. AFAIK that was fixed years ago if you mean float formatting. > Regexes behave differently if they worked at all. TruffleRuby always used Joni, which is literally a translation of CRuby's Regexp engine to Java (by the JRuby team), so that is very surprising and I have a really hard time to believe it. At least "Regexes behave differently if they worked at all" seems harsh and highly inaccurate to me. There likely were a couple Regexp issues but the generalization seems wildly exaggerated.
klelatti 5y ago
The Truffleruby results look amazing but leave me with the question why we don’t see stories of implementation at major Ruby / Rails sites. Too soon, licensing or not yet 100% compatible?
- headius 5y ago
  JRuby is in wide deployment at hundreds or thousands of businesses across every industry. We generally run real-world apps much faster than CRuby (lower latency, better use of CPU and memory) and are especially good at big data and high concurrency applications. JRuby remains the only alternative Ruby to see large-scale production deployment.
- slx26 5y ago
  Windows support?
- Mikeb85 5y ago
  Pretty sure Shopify uses it for some apps.
  - klelatti 5y ago
    Thanks - yes of course Chris Seaton works at Shopify. Still a bit puzzled why we haven't seen any headlines about major savings.
- ch4s3 5y ago
  It isn't 100% MRI 3.0 compatible[0] and wasn't able to run Rails at first. It's also not always faster for Rails which occupies a lot of Ruby mindshare. [0] https://github.com/oracle/truffleruby#current-status https://github.com/oracle/truffleruby#current-status
  - klelatti 5y ago
    Is the Rails performance something to do with the amount of meta programming going on - I can imagine that being hard to optimise!
Mikeb85 5y ago
I use a benchmark that's pretty rough on languages, basically lots of maths and loops. Python 3 and Ruby 3 (with no jit) both take about 90 seconds on my computer. Ruby 3.1 with --YJIT takes about 35 seconds. TruffleRuby takes 9 seconds (!!) JS (Chrome) takes 8 seconds. C++ takes just over a second and C is about a second even.
- igouy 5y ago
  Please use something "basically lots of maths and loops" where we can see the source code (like the blog post does). n-body https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbody.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/nbody-node-6.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... spectral-norm https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/spectralnorm.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/spectralnorm-node-6.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

91 points by shelajev 5y ago

sparker72678 5y ago

I keep forgetting just how fast TruffleRuby is.

karmicthreat 5y ago
What is the downside to TruffleRuby? Memory consumption since its JVM based?
- kenhwang 5y ago
  Very subtle differences in behavior. It's fine if you start writing your project targeting TruffleRuby, but its a nightmare porting a massive existing ruby codebase.
  - eregon 5y ago
    Compatibility is certainly an issue for massive codebases, OTOH I think over time TruffleRuby is getting closer and closer to CRuby behavior to the point it would be fairly rare to find a compatibility issue. Do you have personal experience trying to run such a codebase on TruffleRuby?
    - kenhwang 5y ago
      We noticed many numerical operations sometimes didn't return the same class as cruby (Fixnum/Bignum/BigDecimal), sometimes resulting in wildly different calculation results. Numerical to string formatting is a bit different too. Which makes the above problem even worse, compounded by how frequently ruby web code goes string to numbers to string to numbers. Regexes behave differently if they worked at all.
      - eregon 5y ago
        When did you try? Fixnum/Bignum are gone since many years (before 1.0RC1 which was April 2018, it's all Integer now). So I guess many years ago, and back then TruffleRuby was basically implementing Ruby features, had fairly limited compatibility and likely could not run Rails, very different from today. There were compatibility issues with BigDecimal, but TruffleRuby now uses the C extension, hence it should be exactly the same behavior as CRuby. > Numerical to string formatting is a bit different too. AFAIK that was fixed years ago if you mean float formatting. > Regexes behave differently if they worked at all. TruffleRuby always used Joni, which is literally a translation of CRuby's Regexp engine to Java (by the JRuby team), so that is very surprising and I have a really hard time to believe it. At least "Regexes behave differently if they worked at all" seems harsh and highly inaccurate to me. There likely were a couple Regexp issues but the generalization seems wildly exaggerated.

klelatti 5y ago

The Truffleruby results look amazing but leave me with the question why we don’t see stories of implementation at major Ruby / Rails sites. Too soon, licensing or not yet 100% compatible?

headius 5y ago
JRuby is in wide deployment at hundreds or thousands of businesses across every industry. We generally run real-world apps much faster than CRuby (lower latency, better use of CPU and memory) and are especially good at big data and high concurrency applications. JRuby remains the only alternative Ruby to see large-scale production deployment.
slx26 5y ago
Windows support?
Mikeb85 5y ago
Pretty sure Shopify uses it for some apps.
- klelatti 5y ago
  Thanks - yes of course Chris Seaton works at Shopify. Still a bit puzzled why we haven't seen any headlines about major savings.
ch4s3 5y ago
It isn't 100% MRI 3.0 compatible[0] and wasn't able to run Rails at first. It's also not always faster for Rails which occupies a lot of Ruby mindshare. [0] https://github.com/oracle/truffleruby#current-status https://github.com/oracle/truffleruby#current-status
- klelatti 5y ago
  Is the Rails performance something to do with the amount of meta programming going on - I can imagine that being hard to optimise!

Mikeb85 5y ago

I use a benchmark that's pretty rough on languages, basically lots of maths and loops. Python 3 and Ruby 3 (with no jit) both take about 90 seconds on my computer. Ruby 3.1 with --YJIT takes about 35 seconds. TruffleRuby takes 9 seconds (!!) JS (Chrome) takes 8 seconds. C++ takes just over a second and C is about a second even.

igouy 5y ago
Please use something "basically lots of maths and loops" where we can see the source code (like the blog post does). n-body https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/nbody.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/nbody-node-6.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... spectral-norm https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/spectralnorm.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/... https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/spectralnorm-node-6.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...