5 ms·
Ruby: Porting YJIT to Rust
- __s 5y agoLooks like they aren't intending to use Cranelift. Which makes sense for an initial port that already has its own codegen. Personally found Cranelift a joy to use. Would be nice if Cranelift had a WASM backend to could compile a JIT to WASM to JIT WASM in WASM (this cyclic feature would be useful for creating JITs for dynamic languages in WASM rather than compiling the C interpreters to WASM)
- woodruffw 5y agoIt's unfortunate, but cranelift might still be considered too unstable for "primetime": my understanding as an outsider is that they're looking to stabilize YJIT sooner rather than later, to prevent it from being a perpetually unstable/experimental feature like MJIT ended up being. (Also, the last time I checked, cranelift emitted extremely suboptimal code. That's not their fault given the design constraints (it's a new project and they're aiming to beat LLVM's SelectionDAG in codegen performance), but it's probably not ideal given Ruby's long term performance goals. I'm also not sure how easy it would be to retrofit the LBBV[1] architecture of YJIT onto cranelift.) [1]: https://dl.acm.org/doi/10.1145/3486606.3486781 https://dl.acm.org/doi/10.1145/3486606.3486781
- taldridge 5y agoWorking on it! https://GitHub.com/teymour-aldridge/cranelift_codegen_wasm https://GitHub.com/teymour-aldridge/cranelift_codegen_wasm
- Arcuru 5y ago> Rust as a programming language community has a great deal of enthusiasm behind it. This could translate to more enthusiasm for YJIT and for Ruby as well. It's refreshing to see a project explicitly call this out as a benefit for adopting Rust. Love it or hate it, the RIIR (Rewrite It In Rust) crowd certainly has a lot of momentum behind it.
- dkga 5y agoYes! Love the acronym, too.
- gk1256 5y ago> it's refreshing to see a project explicitly call this out as a benefit for adopting Rust It seems to be a proposal. The proposal was made by a user who was registered on 09/27/2021. The text you quoted was included on his proposal. The project itself is not explicitly call that one out.
- ksec 5y ago> The proposal was made by a user who was registered on 09/27/2021. maximecb is the lead of YJIT project in Shopify.
- pygy_ 5y agoFor context, YJIT was developed at Shopify, based Maxime's thesis (see lazy basic block versioning). It has been merged into Ruby core, and she now has commit rights to Ruby.
- megumax 5y agoThe platform `problem` shouldn't be a problem, because you wouldn't use ruby outside of Linux, Windows, Mac, FreeBSD, OpenBSD and Rust has support for more than that[1]. Backwards compatibility isn't a concert either, Rust promised and kept it for some years (after 1.6). The real problem is whether or not the expense of porting it is justified. Taking a look at the git repository of it (before it was merged into upstream), I understand why they want to rewrite it in Rust, the repository looks like a complete mess. I don't know if they want to use an existing JIT codegen like LLVM or Cranelift, or their own.
- byroot 5y ago> you wouldn't use ruby outside of Linux, Windows, Mac, FreeBSD, OpenBSD and Rust has support for more than that[1]. Ruby has support for plenty of "exotic" systems like HP UX, AIX etc. It's unclear how many people actually use it, but it's there. > I don't know if they want to use an existing JIT codegen like LLVM or Cranelift, or their own. Their own.
- petercooper 5y agoFun bit of trivia.. Ruby only (officially) dropped Atari ST support in 2016: https://github.com/ruby/ruby/blob/c5eb24349a4535948514fe765c3ddb0628d81004/doc/ChangeLog-2.4.0#L4155 https://github.com/ruby/ruby/blob/c5eb24349a4535948514fe765c... .. I've never heard of anyone actually using Ruby on an Atari ST though(!)
- pjmlp 5y agoAt least in what concerns AIX, IBM has migrated their compilers on top of LLVM, so some support is there. No idea about HP-UX, it is impossible to find anything now, apparently HPE has buried almost everything about it. Completely unrelated to how good it used to be about 20 years ago.
- byroot 5y agoI just gave a couple examples I had in mind because they came up recently. The point was that the parent's assertion: > you wouldn't use ruby outside of Linux, Windows, Mac, FreeBSD, OpenBSD Isn't quite true.
- baggy_trough 5y agoIt's a little disturbing that this YJIT was accepted into ruby 3.1 near the end of its cycle, just a couple months ago, and now it's being rewritten in a new language. Kind of a smell if you ask me.
- baggy_trough 5y ago"We are currently still early in our investigation of the Rust port. I am curious to know if you would also be open to YJIT being written in a language like Zig, or is that too new/experimental?" sounds half-baked...
- viraptor 5y agoThey posted the reasoning - C is limiting them from finishing the job cleanly for more platforms. It sounds like a reasonable choice to move to something leaner before yjit grows too large and the sunk cost issue starts being raised.
- maxime_cb 5y agoYeah to be honest, C was never our favorite pick to begin with. We initially wrote YJIT in C99 because that was what the rest of the CRuby codebase was written in. However, JIT compilers are very complex pieces of software. They need to do things like symbolic transformations on graphs, and deal with multiple different kinds of intermediate representations. Doing that in C, you get really bogged down with the details of manual memory management and rolling your own data structures. You can get all kinds of subtle bugs if you aren't careful.
- baggy_trough 5y agoIf you know you're going to rewrite the thing in a new language, then why push to include it in ruby now?
- maxime_cb 5y agoWe didn't push to upstream now, we were invited to upstream in CRuby and we accepted the invitation. As for the Rust rewrite, we just started to work on the prototype, and no decision had been made. The ticket linked in this thread is me essentially asking the Ruby core devs for permission to do so. We wanted to give everyone an opportunity to chime in. Not sure what to tell you. Things don't aways happen in the most optimal order in the real world, sometimes plans change, but I'm fairly confident that we can make this work well.
- ksec 5y agoIt is interesting the discussion of Rust and Zig, which is the opposite spectrum of programming. It make sense if Ruby Cores are interested to develop / move to a new Ruby VM towards Rust. Not surprising considering Love2Code ( along with many other Ruby VM dev ) have complained over the years for CRuby's complexity. The Ruby Spec is a fairly recent thing done mostly by JRuby and TruffleRuby folks ( Correct me if am wrong ). If not, Zig on the other hand just works a lot better with old C code. One could argue Rust is Progressive and Zig is conservative. Unfortunately it is still way too early for Zig. I also wonder what is happening to TruffleRuby inside Shopify. And where is TenderJIT :)
- shellac 5y agoRuby spec has been going for a long time. It came from rubinius originally.
- pjmlp 5y agoTo me it looks like looking for new toys. My C complaints are well known, yet I don't see the need to make the build system even more complex, dropping platform support, for the benefit of yet another language, regardless of Rust or Zig winning in the end. Then they shouldn't have merged the JIT to start with and just do their own Ruby implementation instead.
- byroot 5y ago> just do their own Ruby implementation instead "just"... Several other Ruby JIT projects (Rubinius, JRuby, TruffleRuby, etc) have poured tons of resources, and while they perform very well, the adoption is still very low because keeping up with the main implementation takes tons of effort. On the other hand YJIT managed to speedup real world codebases with pretty much 100% compatibility in under a year of development by a fairly small team. Shopify already invest quite a lot in TruffleRuby, that's the new implementation moonshot you call for, YJIT is the more grounded, shorter term hedge.
- pjmlp 5y agoI am fully aware of it, and spending now several months rewriting YJIT into Rust/Zig is supposed to get back that money (developer hours x $ per hour) exactly how?
- faitswulff 5y agoIn related news, I've also been following this PR to add Rust support to Ruby for gem native extensions: https://github.com/rubygems/rubygems/pull/5175 https://github.com/rubygems/rubygems/pull/5175 It seems to have hit a minor speedbump dealing with support for Windows.
- Rochus 5y agoWhy don't they use Crystal? Wouldn't it support the listed value propositions too? And the syntax is close to Ruby.
- Gigachad 5y agoIs Crystal still useful anymore? Now that Ruby has type checking / optional types.
- Rochus 5y agoI think so. It has a similar syntax to Ruby, is statically typed with type inference, and has a much higher performance. I don't think it intends to nor can it replace Ruby. But it's still an attractive alternative for many projects.
- antifa 5y agoCrystal has a full, real type system (probably on par with Rust), and compiles into a fast binary.
- ezrast 5y agoI like Crystal, but it doesn't support the last three value propositions: it supports fewer platforms than Rust (in particular, no Windows outside of WSL) and has less community support/enthusiasm. It also has an obligate runtime, so linking to Crystal code from non-Crystal programs is problematic.
- sandstrom 5y agoFor anyone interested in Ruby and Rust, check out Artichoke. https://github.com/artichoke/artichoke https://github.com/artichoke/artichoke It’s ruby, implemented in rust. Early days, but very interesting. (I have no affiliation or involvement with it, just a fan)
- p4lindromica 5y agoHi, Artichoke author here. I feel very blessed to have Artichoke be mentioned in this ticket on Ruby's bug tracker. If y'all are interested to follow along, @artichokeruby on Twitter is the best spot to do it. https://twitter.com/artichokeruby https://twitter.com/artichokeruby
- nusaru 5y agoIt's worth noting that their backend (VM and parser) currently uses MRuby, which is written in C, so not yet quite a "real" implementation in my view.
- lopopolo 5y agoThe mruby VM as used in Artichoke uses Artichoke's Rust implementations of `String`, `Symbol`, and `Array` (and many other core classes, these ones just happen to be used by the VM itself). The symbol table (which is an essential part of how classes and methods and `send` are implemented) is implemented using the `intaglio` crate which is part of the Artichoke project. These implemented-in-Rust bits expose themselves to the remaining C with `extern "C"` APIs written in Rust that maintain ABI compat with the mruby code they replace. The mruby VM is getting eaten by the Rust parts, so I don't think it's quite fair to say that Artichoke uses mruby as an off the shelf component. To fully move beyond mruby though to a native Rust VM will take time.
- komuW 5y ago"it's OK for me to use Zig to implement YJIT, as long as foreseeable portability problems (mentioned above) are addressed. Besides that, I prefer Zig to Rust, as a long-term C programmer, but all up to you." - Matz (Yukihiro Matsumoto) reference: https://bugs.ruby-lang.org/issues/18481#note-17 https://bugs.ruby-lang.org/issues/18481#note-17