3 ms·

The author pans meta-tracing and shows a graph of TruffleRuby having really crazy runtime behavior. However, it looks extremely suspicious - like the kind of th

The author pans meta-tracing and shows a graph of TruffleRuby having really crazy runtime behavior. However, it looks extremely suspicious - like the kind of thing you'd see if there was a memory leak and the garbage collector was running constantly. Most people seem really excited about the results coming out of Graal and TruffleRuby seems to have some really impressive results in general, so the graph is surprising. It's also missing a bunch of details, so it's hard to speak to its voracity - for example, what are the different versions of the runtimes, what flags were used, on what hardware? As a counter example, there's a different (admittedly old) result where TruffleRuby beats cruby on the same benchmark by 4.38x. https://eregon.me/blog/2022/01/06/benchmarking-cruby-mjit-yjit-jruby-truffleruby.html https://eregon.me/blog/2022/01/06/benchmarking-cruby-mjit-yj...

jamii 2y ago
> it looks extremely suspicious It matches my anecdotal experiences with graal. It's a huge complex machine with a lot of surface area for weird performance bugs. > TruffleRuby beats cruby on the same benchmark by 4.38x That benchmark only looks at peak performance, which TR unarguably dominates at, and ignores other important considerations like warmup time. Here's another railsbench (https://i.imgur.com/IzMRKdQ.png https://i.imgur.com/IzMRKdQ.png) from the same paper (https://dl.acm.org/doi/pdf/10.1145/3617651.3622982 https://dl.acm.org/doi/pdf/10.1145/3617651.3622982). TR achieves the best peak performance, but only after 3 minutes of warmup and it starts out 17x slower than cruby. If every time you deploy new code your latency spikes by 17x for 3 minutes, that's gonna be painful. I don't know of anyone that is using TR in production. Shopify were doing some work on it for a while, but they've since developed and shipped yjit so I don't know if they still have any investment in TR.

jamii 2y ago

> it looks extremely suspicious It matches my anecdotal experiences with graal. It's a huge complex machine with a lot of surface area for weird performance bugs. > TruffleRuby beats cruby on the same benchmark by 4.38x That benchmark only looks at peak performance, which TR unarguably dominates at, and ignores other important considerations like warmup time. Here's another railsbench (https://i.imgur.com/IzMRKdQ.png https://i.imgur.com/IzMRKdQ.png) from the same paper (https://dl.acm.org/doi/pdf/10.1145/3617651.3622982 https://dl.acm.org/doi/pdf/10.1145/3617651.3622982). TR achieves the best peak performance, but only after 3 minutes of warmup and it starts out 17x slower than cruby. If every time you deploy new code your latency spikes by 17x for 3 minutes, that's gonna be painful. I don't know of anyone that is using TR in production. Shopify were doing some work on it for a while, but they've since developed and shipped yjit so I don't know if they still have any investment in TR.