5 ms·

The Truffleruby results look amazing but leave me with the question why we don’t see stories of implementation at major Ruby / Rails sites. Too soon, licensing

The Truffleruby results look amazing but leave me with the question why we don’t see stories of implementation at major Ruby / Rails sites. Too soon, licensing or not yet 100% compatible?

headius 5y ago
JRuby is in wide deployment at hundreds or thousands of businesses across every industry. We generally run real-world apps much faster than CRuby (lower latency, better use of CPU and memory) and are especially good at big data and high concurrency applications. JRuby remains the only alternative Ruby to see large-scale production deployment.
slx26 5y ago
Windows support?
Mikeb85 5y ago
Pretty sure Shopify uses it for some apps.
- klelatti 5y ago
  Thanks - yes of course Chris Seaton works at Shopify. Still a bit puzzled why we haven't seen any headlines about major savings.
ch4s3 5y ago
It isn't 100% MRI 3.0 compatible[0] and wasn't able to run Rails at first. It's also not always faster for Rails which occupies a lot of Ruby mindshare. [0] https://github.com/oracle/truffleruby#current-status https://github.com/oracle/truffleruby#current-status
- klelatti 5y ago
  Is the Rails performance something to do with the amount of meta programming going on - I can imagine that being hard to optimise!
  - joelbluminator 5y ago
    Well YJIT succeeded to improve Rails by 30% in a year , so it IS possible. https://speed.yjit.org/ https://speed.yjit.org/
    - eregon 5y ago
      It's the same benchmark in the blog post, `railsbench`. So TruffleRuby already speeds up some Rails apps like that one but not every Rails app/program. (TruffleRuby 3.27x, YJIT 1.33x on railsbench)
  - ch4s3 5y ago
    I'm not sure. If Chris Seaton pops into the thread, you should ask him.
  - eregon 5y ago
    Actually TruffleRuby optimizes metaprogramming more than any other Ruby implementation. Rails is not the issue, big codebases is the issue: more code = more chances to use something which is not so fast on TruffleRuby yet = more chance to become a bottleneck and degrade the overall performance.
    - klelatti 5y ago
      Thanks for the thoughtful and insightful responses. Understand that a large code base is hard - your point here though presumably means that some things in truffleruby are significantly slower than CRuby to cancel out the gains. Or have I misunderstood?!
      - eregon 5y ago
        Yes, some things are slower. Most of them I'd say are unexpected performance bugs. I'd think most are easy to fix once investigated, but some can be hard to fix (recent example, `File.read` is quite fast on CRuby). Some things are expected to be slower, for instance constantly redefining (monkey-patching) methods or constants is slower on TruffleRuby, but that's typically because the program is broken and so it'd be slow on CRuby as well.
x3n0ph3n3 5y ago
I just ran a comparison of a benchmarking script we use for our own product and saw wildly different results. There's no way I'd consider migrating to truffle today: # With MRI ruby-2.6.6 Measuring [loading ruby dependencies] 3.560960 2.507366 6.075755 ( 6.154898) Measuring [loading configuration yaml] 0.823779 0.085605 0.909384 ( 0.931334) Measuring [creating objects from YAML and performing filtering] 0.499576 0.042023 0.541599 ( 0.552427) Measuring [object validation] 2.362139 0.226133 2.588272 ( 2.674958) Total time: 7.246786 2.861230 10.115445 ( 10.314025) # With truffleruby-21.3.0 Measuring [loading ruby dependencies] 91.818473 5.598149 97.427109 ( 36.676471) Measuring [loading configuration yaml] 28.032630 0.811577 28.844207 ( 8.616437) Measuring [creating objects from YAML and performing filtering] 14.952781 0.479387 15.432168 ( 4.634803) Measuring [object validation] 86.747057 2.788266 89.535323 ( 28.324909) Total time: 221.595872 9.680172 231.286531 ( 78.264226)
- eregon 5y ago
  Startup is likely to be worse, because it typically runs a lot of different code for not long and the JIT might not have enough time to optimize much of that. OTOH the JIT needs to learn what the program is doing, i.e., profile it in a sense, and that has a cost on interpreter speed. Still these numbers are worse than expected, especially [object validation], so if there is a way to reproduce it'd be great if you can open an issue about it. Persisting the JITed code is what we think can solve the slower startup entirely: https://www.graalvm.org/graalvm-as-a-platform/language-implementation-framework/AOTOverview/ https://www.graalvm.org/graalvm-as-a-platform/language-imple... Also other things mentioned in https://eregon.me/blog/2022/01/06/benchmarking-cruby-mjit-yjit-jruby-truffleruby.html#warmup https://eregon.me/blog/2022/01/06/benchmarking-cruby-mjit-yj...
pmahoney 5y ago
Many years ago (2011 or so?), I worked on a Rails app that used JRuby. The driving force was a small part of the app needing a Java library. I think we did see some performance benefit, or at least benefit from using multi-threaded web servers instead of process-per-request servers (like Unicorn). However, we also ran into many problems that felt like we were the first to encounter them (few or no similar reports on project bug lists or StackOverflow etc.) which is never fun if you're under any kind of pressure to deliver. In hindsight, if I had something small and reasonably isolated (a microservice?), it could make sense to use JRuby or Truffle (if you need some JVM lib; if it's purely for performance reasons I'd just grab a different language for that microservice). But for a larger app that pays the bills, I'd stick with the well-trod path (at the time that was MRI, Unicorn, single-threaded).
- headius 5y ago
  Glad to hear you got some benefit from JRuby! A lot has changed since 2011, including much better compatibility and many little performance improvements. I'm not sure we were even on the current JRuby runtime (register-based IR + JVM JIT) at that time, which has improved perf and compat tremendously. If you ever have any issues with JRuby again, please let us know. We spend most of our time supporting users and want them to have a good experience.
pizza234 5y ago
Explanation from Stripe¹: > Nearly all of Stripe’s codebase is implemented in Ruby running on the default Ruby VM (YARV). Not only did we not need Java VM-level interoperability, choosing either alternative Ruby implementation would have made for a difficult migration path. Stripe relies heavily on gems with native extensions, and as you can imagine, a multi-million line Ruby codebase over time starts to depend on Ruby-the-implementation, not just Ruby-the-language. ¹: https://sorbet.org/blog/2021/07/30/open-sourcing-sorbet-compiler https://sorbet.org/blog/2021/07/30/open-sourcing-sorbet-comp...
- eregon 5y ago
  That second sentence seems a fair assessment. TruffleRuby does support many native (C/C++) extensions so that's rarely an issue. But indeed in such a large codebase it's likely to depend unexpectedly on CRuby-specific behavior. And while that can be fixed it takes some effort either in TruffleRuby (to match CRuby) or in the app (e.g. to avoid relying on `RubyVM`).
eregon 5y ago
Compatibility is one, it's hard to be 100% compatible with CRuby, and large codebases tend to sometimes depend unintentionally on weird behavior or even bugs in CRuby. Keeping up compatibility (while keeping things efficient) is a lot of work, TruffleRuby tries to reduce that by reusing as much as possible existing code, including reusing C extensions shipped with CRuby.

The Truffleruby results look amazing but leave me with the question why we don’t see stories of implementation at major Ruby / Rails sites. Too soon, licensing or not yet 100% compatible?

headius 5y ago

JRuby is in wide deployment at hundreds or thousands of businesses across every industry. We generally run real-world apps much faster than CRuby (lower latency, better use of CPU and memory) and are especially good at big data and high concurrency applications. JRuby remains the only alternative Ruby to see large-scale production deployment.

slx26 5y ago

Windows support?

Mikeb85 5y ago

Pretty sure Shopify uses it for some apps.

klelatti 5y ago
Thanks - yes of course Chris Seaton works at Shopify. Still a bit puzzled why we haven't seen any headlines about major savings.

ch4s3 5y ago

It isn't 100% MRI 3.0 compatible[0] and wasn't able to run Rails at first. It's also not always faster for Rails which occupies a lot of Ruby mindshare. [0] https://github.com/oracle/truffleruby#current-status https://github.com/oracle/truffleruby#current-status

klelatti 5y ago
Is the Rails performance something to do with the amount of meta programming going on - I can imagine that being hard to optimise!
- joelbluminator 5y ago
  Well YJIT succeeded to improve Rails by 30% in a year , so it IS possible. https://speed.yjit.org/ https://speed.yjit.org/
  - eregon 5y ago
    It's the same benchmark in the blog post, `railsbench`. So TruffleRuby already speeds up some Rails apps like that one but not every Rails app/program. (TruffleRuby 3.27x, YJIT 1.33x on railsbench)
- ch4s3 5y ago
  I'm not sure. If Chris Seaton pops into the thread, you should ask him.
- eregon 5y ago
  Actually TruffleRuby optimizes metaprogramming more than any other Ruby implementation. Rails is not the issue, big codebases is the issue: more code = more chances to use something which is not so fast on TruffleRuby yet = more chance to become a bottleneck and degrade the overall performance.
  - klelatti 5y ago
    Thanks for the thoughtful and insightful responses. Understand that a large code base is hard - your point here though presumably means that some things in truffleruby are significantly slower than CRuby to cancel out the gains. Or have I misunderstood?!
    - eregon 5y ago
      Yes, some things are slower. Most of them I'd say are unexpected performance bugs. I'd think most are easy to fix once investigated, but some can be hard to fix (recent example, `File.read` is quite fast on CRuby). Some things are expected to be slower, for instance constantly redefining (monkey-patching) methods or constants is slower on TruffleRuby, but that's typically because the program is broken and so it'd be slow on CRuby as well.

x3n0ph3n3 5y ago

I just ran a comparison of a benchmarking script we use for our own product and saw wildly different results. There's no way I'd consider migrating to truffle today: # With MRI ruby-2.6.6 Measuring [loading ruby dependencies] 3.560960 2.507366 6.075755 ( 6.154898) Measuring [loading configuration yaml] 0.823779 0.085605 0.909384 ( 0.931334) Measuring [creating objects from YAML and performing filtering] 0.499576 0.042023 0.541599 ( 0.552427) Measuring [object validation] 2.362139 0.226133 2.588272 ( 2.674958) Total time: 7.246786 2.861230 10.115445 ( 10.314025) # With truffleruby-21.3.0 Measuring [loading ruby dependencies] 91.818473 5.598149 97.427109 ( 36.676471) Measuring [loading configuration yaml] 28.032630 0.811577 28.844207 ( 8.616437) Measuring [creating objects from YAML and performing filtering] 14.952781 0.479387 15.432168 ( 4.634803) Measuring [object validation] 86.747057 2.788266 89.535323 ( 28.324909) Total time: 221.595872 9.680172 231.286531 ( 78.264226)

eregon 5y ago
Startup is likely to be worse, because it typically runs a lot of different code for not long and the JIT might not have enough time to optimize much of that. OTOH the JIT needs to learn what the program is doing, i.e., profile it in a sense, and that has a cost on interpreter speed. Still these numbers are worse than expected, especially [object validation], so if there is a way to reproduce it'd be great if you can open an issue about it. Persisting the JITed code is what we think can solve the slower startup entirely: https://www.graalvm.org/graalvm-as-a-platform/language-implementation-framework/AOTOverview/ https://www.graalvm.org/graalvm-as-a-platform/language-imple... Also other things mentioned in https://eregon.me/blog/2022/01/06/benchmarking-cruby-mjit-yjit-jruby-truffleruby.html#warmup https://eregon.me/blog/2022/01/06/benchmarking-cruby-mjit-yj...

pmahoney 5y ago

Many years ago (2011 or so?), I worked on a Rails app that used JRuby. The driving force was a small part of the app needing a Java library. I think we did see some performance benefit, or at least benefit from using multi-threaded web servers instead of process-per-request servers (like Unicorn). However, we also ran into many problems that felt like we were the first to encounter them (few or no similar reports on project bug lists or StackOverflow etc.) which is never fun if you're under any kind of pressure to deliver. In hindsight, if I had something small and reasonably isolated (a microservice?), it could make sense to use JRuby or Truffle (if you need some JVM lib; if it's purely for performance reasons I'd just grab a different language for that microservice). But for a larger app that pays the bills, I'd stick with the well-trod path (at the time that was MRI, Unicorn, single-threaded).

headius 5y ago
Glad to hear you got some benefit from JRuby! A lot has changed since 2011, including much better compatibility and many little performance improvements. I'm not sure we were even on the current JRuby runtime (register-based IR + JVM JIT) at that time, which has improved perf and compat tremendously. If you ever have any issues with JRuby again, please let us know. We spend most of our time supporting users and want them to have a good experience.

pizza234 5y ago

Explanation from Stripe¹: > Nearly all of Stripe’s codebase is implemented in Ruby running on the default Ruby VM (YARV). Not only did we not need Java VM-level interoperability, choosing either alternative Ruby implementation would have made for a difficult migration path. Stripe relies heavily on gems with native extensions, and as you can imagine, a multi-million line Ruby codebase over time starts to depend on Ruby-the-implementation, not just Ruby-the-language. ¹: https://sorbet.org/blog/2021/07/30/open-sourcing-sorbet-compiler https://sorbet.org/blog/2021/07/30/open-sourcing-sorbet-comp...

eregon 5y ago
That second sentence seems a fair assessment. TruffleRuby does support many native (C/C++) extensions so that's rarely an issue. But indeed in such a large codebase it's likely to depend unexpectedly on CRuby-specific behavior. And while that can be fixed it takes some effort either in TruffleRuby (to match CRuby) or in the app (e.g. to avoid relying on `RubyVM`).

eregon 5y ago

Compatibility is one, it's hard to be 100% compatible with CRuby, and large codebases tend to sometimes depend unintentionally on weird behavior or even bugs in CRuby. Keeping up compatibility (while keeping things efficient) is a lot of work, TruffleRuby tries to reduce that by reusing as much as possible existing code, including reusing C extensions shipped with CRuby.