6 ms·

Pretty cool. Tbh, the benchmark result improvements are not as big as I'd expect (ie, the biggest improvement is "only" 41% compared to non-JITted code) but spe

by WJW 5y ago

Pretty cool. Tbh, the benchmark result improvements are not as big as I'd expect (ie, the biggest improvement is "only" 41% compared to non-JITted code) but speedups are speedups and this is pretty nice considering you can get it for (almost) free with a single command line flag.

kawsper 5y ago
Maxime explains in the video[0] that they have other optimizations planned but haven't implemented them yet. It will be very interesting to see the impact :) [0] https://www.youtube.com/watch?v=PBVLf3yfMs8 https://www.youtube.com/watch?v=PBVLf3yfMs8
jaynetics 5y ago
biggest improvement is 68% for optcarrot (ruby-based NES emulator). not bad at all!
- m12k 5y ago
  I imagine an emulator will have hot paths in tight loops in a way that web apps don't, so it makes sense that it sees the biggest benefit from jit'ing
- pjmlp 5y ago
  JRuby also runs it quite fast, although I think does require some warmup measured in minutes, until it is almost C like.
  - lf-non 5y ago
    We used to run a production service written in JRuby until a year or so ago (at which point it was rewritten in kotlin). It was a minimal pure ruby service deployed as a tomcat servlet. Despite not using any heavy weight framework (eg. rails etc.) its performance (after warmup) was nowhere close to "almost C like". JVM gives you true multithreading etc. but claiming that JRuby performance is C like is just misleading.
    - pjmlp 5y ago
      Well, https://news.ycombinator.com/item?id=13056334 https://news.ycombinator.com/item?id=13056334
      - byroot 5y ago
        That's TruffleRuby, not JRuby. And even then, I haven't benchmarked myself, but I wouldn't be surprised most NES emulators written in C or other native languages out there would do way more than 150FPS. Edit: and it doesn't really matter either way because while useful, optcarrot is a very different workload from what Ruby is generally used for.
        nitrogen 5y ago
        very different workload from what Ruby is generally used for. Ruby is generally used for a lot of things, including scientific computing. I'm using it to process sound and make videos about sound, for example. Audio processing isn't too far off from the type of hot loop an emulator would have, so I hope to see benefits from YJIT in my own work, and will also try TruffleRuby once I've finished my current round of video editing.
        pjmlp 5y ago
        Yeah, I got that wrong. Who cares if they do more than 150 FPS, when humans cannot see more than 60 FPS, unless we are talking about VR, which is nonsense for a NES emulator anyway. Trying to win the benchmarks game makes people lose sight from what actually matters. SPA would never had taken off if it wasn't for the work done in JavaScript JITs. Similarly, Ruby JITs can open doors for just being the language used for Rails applications.
        byroot 5y ago
        > Who cares if they do more than 150 FPS I don't, but you said "almost C like".
        5y ago
  - nobleach 5y ago
    Back in the day, I used JRuby because it was the only way I was able to run Ruby on Windows 2003 servers that I was forced to use while working for the Federal Government. (Yes, Thin and Mongrel existed, but their perf on Windows was REALLY bad) Was it fast? No. Was it "good enough?" yep. It never fell over.
- munificent 5y ago
  Even that's not very large. Moving from a bytecode VM to a JIT is a big jump in codebase complexity that all future maintainers and contributors to Ruby will have to take on. The set of people who can contribute to a JIT is much smaller than to a bytecode VM. I'd guess one or two orders of magnitude. I can't see that codebase complexity being worthwhile unless the JIT at least gives you something like 2-3x better perf across the board. In Java and JS VMs, you do see an improvement of that scale.
  - joelbluminator 5y ago
    That's an interesting take. It seems like Matz and the rest of Ruby core don't see this as such a big deal though as this is going to get merged. I can see the added complexity but this can just be another piece of Ruby core maintained by a few experts, no? Seems like most Ruby contributors specialize anyway.
rattray 5y ago
Interesting; Stripe claims 22-170% speedups with its Sorbet-based compiler for Ruby: https://twitter.com/patrickc/status/1410269843585069056?s=19 https://twitter.com/patrickc/status/1410269843585069056?s=19 That may be based on wildly different benchmarks, but still sounds much more compelling than 7-40%. I wonder what drives the difference...
- nerdponx 5y ago
  I would guess that the difference is static type information. Similar to how PyPy is faster than CPython, but Cython (and probably Nuitka and MyPyC) is faster than PyPt.
  - formerly_proven 5y ago
    JITs don't need static type information, because they have that information available to them anyway. Cython is not a good example for performance comparisons, because while you can compile Python code unchanged with it, the performance gains are very modest, as Cython still uses the full, unabbreviated Python data model; it only cuts out the overhead of the CPython bytecode VM: everything else is still there, and that's what dominates. Cython also allows you to write C-with-a-Python-syntax, which then has the performance profile you'd expect from C - but you're not writing Python any more.
    - nerdponx 5y ago
      That's a good point. As I hinted in my post, it might be more instructive to see if programs compiled with MyPyC or Nuitka run faster than PyPy or GraalPython.
    - Zababa 5y ago
      > JITs don't need static type information, because they have that information available to them anyway. I've heard that V8 can specialize functions but will sometimes unspecialize them if you use them with unexpected types. Using static typing can push your code into patterns that "helps" the JIT, even if it doesn't use directly the type information that you wrote.
ksec 5y ago
There are plenty of Ruby JIT in the past which shows order of magnitude improvement in micro-benchmarks but doesn't even run rails. ( I think to the point people gave up making JIT for Ruby ) So running these complex code, not only just those heavy benchmarks like railsbench but also their production code all while passing their test is pretty damn impressive if you ask me. And as pointed out both inside the rubykagi talk and twitter, there are lots more optimisation in the pipeline. At the scale of Shopify, those improvement could easily be million dollar saving per year.

by WJW 5y ago

kawsper 5y ago

Maxime explains in the video[0] that they have other optimizations planned but haven't implemented them yet. It will be very interesting to see the impact :) [0] https://www.youtube.com/watch?v=PBVLf3yfMs8 https://www.youtube.com/watch?v=PBVLf3yfMs8

jaynetics 5y ago

biggest improvement is 68% for optcarrot (ruby-based NES emulator). not bad at all!

m12k 5y ago
I imagine an emulator will have hot paths in tight loops in a way that web apps don't, so it makes sense that it sees the biggest benefit from jit'ing
pjmlp 5y ago
JRuby also runs it quite fast, although I think does require some warmup measured in minutes, until it is almost C like.
- lf-non 5y ago
  We used to run a production service written in JRuby until a year or so ago (at which point it was rewritten in kotlin). It was a minimal pure ruby service deployed as a tomcat servlet. Despite not using any heavy weight framework (eg. rails etc.) its performance (after warmup) was nowhere close to "almost C like". JVM gives you true multithreading etc. but claiming that JRuby performance is C like is just misleading.
  - pjmlp 5y ago
    Well, https://news.ycombinator.com/item?id=13056334 https://news.ycombinator.com/item?id=13056334
    - byroot 5y ago
      That's TruffleRuby, not JRuby. And even then, I haven't benchmarked myself, but I wouldn't be surprised most NES emulators written in C or other native languages out there would do way more than 150FPS. Edit: and it doesn't really matter either way because while useful, optcarrot is a very different workload from what Ruby is generally used for.
      - nitrogen 5y ago
        very different workload from what Ruby is generally used for. Ruby is generally used for a lot of things, including scientific computing. I'm using it to process sound and make videos about sound, for example. Audio processing isn't too far off from the type of hot loop an emulator would have, so I hope to see benefits from YJIT in my own work, and will also try TruffleRuby once I've finished my current round of video editing.
      - pjmlp 5y ago
        Yeah, I got that wrong. Who cares if they do more than 150 FPS, when humans cannot see more than 60 FPS, unless we are talking about VR, which is nonsense for a NES emulator anyway. Trying to win the benchmarks game makes people lose sight from what actually matters. SPA would never had taken off if it wasn't for the work done in JavaScript JITs. Similarly, Ruby JITs can open doors for just being the language used for Rails applications.
        byroot 5y ago
        > Who cares if they do more than 150 FPS I don't, but you said "almost C like".
        5y ago
- nobleach 5y ago
  Back in the day, I used JRuby because it was the only way I was able to run Ruby on Windows 2003 servers that I was forced to use while working for the Federal Government. (Yes, Thin and Mongrel existed, but their perf on Windows was REALLY bad) Was it fast? No. Was it "good enough?" yep. It never fell over.
munificent 5y ago
Even that's not very large. Moving from a bytecode VM to a JIT is a big jump in codebase complexity that all future maintainers and contributors to Ruby will have to take on. The set of people who can contribute to a JIT is much smaller than to a bytecode VM. I'd guess one or two orders of magnitude. I can't see that codebase complexity being worthwhile unless the JIT at least gives you something like 2-3x better perf across the board. In Java and JS VMs, you do see an improvement of that scale.
- joelbluminator 5y ago
  That's an interesting take. It seems like Matz and the rest of Ruby core don't see this as such a big deal though as this is going to get merged. I can see the added complexity but this can just be another piece of Ruby core maintained by a few experts, no? Seems like most Ruby contributors specialize anyway.

rattray 5y ago

Interesting; Stripe claims 22-170% speedups with its Sorbet-based compiler for Ruby: https://twitter.com/patrickc/status/1410269843585069056?s=19 https://twitter.com/patrickc/status/1410269843585069056?s=19 That may be based on wildly different benchmarks, but still sounds much more compelling than 7-40%. I wonder what drives the difference...

nerdponx 5y ago
I would guess that the difference is static type information. Similar to how PyPy is faster than CPython, but Cython (and probably Nuitka and MyPyC) is faster than PyPt.
- formerly_proven 5y ago
  JITs don't need static type information, because they have that information available to them anyway. Cython is not a good example for performance comparisons, because while you can compile Python code unchanged with it, the performance gains are very modest, as Cython still uses the full, unabbreviated Python data model; it only cuts out the overhead of the CPython bytecode VM: everything else is still there, and that's what dominates. Cython also allows you to write C-with-a-Python-syntax, which then has the performance profile you'd expect from C - but you're not writing Python any more.
  - nerdponx 5y ago
    That's a good point. As I hinted in my post, it might be more instructive to see if programs compiled with MyPyC or Nuitka run faster than PyPy or GraalPython.
  - Zababa 5y ago
    > JITs don't need static type information, because they have that information available to them anyway. I've heard that V8 can specialize functions but will sometimes unspecialize them if you use them with unexpected types. Using static typing can push your code into patterns that "helps" the JIT, even if it doesn't use directly the type information that you wrote.

ksec 5y ago

There are plenty of Ruby JIT in the past which shows order of magnitude improvement in micro-benchmarks but doesn't even run rails. ( I think to the point people gave up making JIT for Ruby ) So running these complex code, not only just those heavy benchmarks like railsbench but also their production code all while passing their test is pretty damn impressive if you ask me. And as pointed out both inside the rubykagi talk and twitter, there are lots more optimisation in the pipeline. At the scale of Shopify, those improvement could easily be million dollar saving per year.