17 ms·
Thanks for sharing, author here! (AMA?) I had a lot of fun working on this project, having implemented enough of the VM to run both Elixir and IEx before I sto
by archseer 6y ago
Thanks for sharing, author here! (AMA?)
I had a lot of fun working on this project, having implemented enough of the VM to run both Elixir and IEx before I stopped.
Ultimately development stalled since I couldn't get any community interest (I was hoping to give an ElixirConf talk but wasn't accepted either). Was hoping to raise some interest and find some contributors in similar vein to https://github.com/RustPython/RustPython https://github.com/RustPython/RustPython
Nowadays I write a lot less Elixir and a lot more Rust.
- hopia 6y agoA new Erlang VM with just replicated functionality is a fairly hard sell to the Erlang/Elixir community, who brag with the industrial track record of the BEAM. I believe you'd get much more interest if there was some ambitious new promise for this new VM, such as 10x sequential performance etc.
- themgt 6y agoIf the VM is in Rust could it be compiled to WASM?
- rkangel 6y agoYou wouldn't want to, for various reasons. See this blog post about Lumen and the decision decisions: https://tylerscript.dev/bringing-the-beam-to-webassembly-with-lumen https://tylerscript.dev/bringing-the-beam-to-webassembly-wit...
- hobofan 6y agoYou wouldn't want to _right now_. However for almost all points there is a solution underway/in planning. In a year or two it might be feasible. There would however be other limitations, like filesystem APIs etc. not being available in the browser that a lot of frameworks in BEAM languages expect, that would severely limit the usefulness, though I guess that applies to either implementation strategy.
- dnautics 6y agoI think the point though is that architecturally there are performance hits if you don't respect the fact that WASM has a different architecture than what the BEAM expects (harvard vs von neumann IIRC), so you may NEVER want to if you get it right in the first place.
- wahern 6y agoThe Harvard-Von Neumann dichotomy is wrong. C also works perfectly fine on Harvard architectures--it's why function pointers in C are special, aren't guaranteed to be convertible to a void pointer, and why uintptr_t is optional. POSIX adds these additional guarantees to support dlsym, which returns function addresses as void pointers. The problems with compiling other languages to Web Assembly are primarily 1) lack of goto and 2) inability to instantiate and jump between different stacks. These limitations are especially problematic for languages like Erlang/BEAM and Go because Web Assembly-based VM implementations require an extra level of indirection in order to implement some of their core language semantics, resulting in quite slow performance compared to even a pure, strictly compliant C implementation (and presuming the WASM VM itself adds no overheard, which is not actually the case). WASM excluded goto support because it was argued that the relooper algorithm required to translate goto constructs to structured WASM statements was sufficiently capable to cover the vast majority of existing code. And they provided evidence to back up that claim. The flaw in that reasoning is that language implementations and similar niche cases have special needs that application code rarely requires, and in that space constructs like goto are crucial to both simplicity of implementation and performance; the inadequacies of relooper become the norm rather than the exception.
- dnautics 6y agothanks for the clarification!
- rkangel 6y agoI can't imagine a situation in which it would be acceptable to download an entire (WASM compiled) copy of the BEAM from every new website I visit. Full AOT compilation with dead code elimination seems like a much better fit for minimising that bundle size.
- archseer 6y agoYeah I definitely didn't see it being production ready any time soon, but I thought it was an interesting project for people that wanted to learn BEAM internals. That's how I got started with it at least, I had problems trying to contribute BEAM just because of the sheer size of the codebase and lacking the domain specific knowledge. I do think that having alternative implementations is good for experimentation though, similar to how Ruby was improved upon ideas from JRuby and Rubinius, even if most users never used those two directly.
- bsder 6y ago> That's how I got started with it at least, I had problems trying to contribute BEAM just because of the sheer size of the codebase and lacking the domain specific knowledge. And the fact that a LOT of BEAM was terribly undocumented. I'm impressed you got anywhere, personally. The last time I looked at BEAM to understand some weird behavior (that eventually did turn out to be a problem in Erlang/BEAM), it was completely impenetrable. Your implementation is probably useful as documentation, if nothing else.
- bitwalker 6y ago100%, when I started building Lumen, I spent an enormous amount of time working out how various parts of the BEAM runtime were implemented, and it was (and still is sometimes) a grind. That macro-heavy C code is just such a bear to read. I understand why it was written that way, but working with it is just unpleasant. Having Enigma, and other implementations like it, provides a huge value in terms of understanding how it all fits together - understanding the Enigma implementation and then going and trying to make sense of the BEAM would probably be a way better path than trying to dive into the BEAM straight away.
- dnautics 6y agoSomeday this is going to need to happen though. IMO, "the right way" to do this is via the strangler pattern: https://www.michielrook.nl/2016/11/strangler-pattern-practice/ https://www.michielrook.nl/2016/11/strangler-pattern-practic... Probably the language that is most poised to achieve this is Zig; it would be feasible to start by wrapping the entire BEAM in a zig compilation unit; which at the very least potentially offers an easier path to maintaining the codebase across multiple platforms. Followed by hodgepodge doing bits and pieces in zig, which could be achieved via straightforward transliteration at first. The very different mindset of the rust PL lends itself to total rewrites, which I don't think will sit well in the BEAM community. On the other hand erlang has tons of strange rewrites happening over its own internal ecosystem all the time (gen_fsm -> gen_statem, pg -> pg2 -> pg), etc.
- muizelaar 6y agoDo you know of any examples of this being done with Zig? I can think of a couple with Rust: - https://gitlab.gnome.org/GNOME/librsvg https://gitlab.gnome.org/GNOME/librsvg completed a migration to Rust. - https://github.com/RazrFalcon/rustybuzz https://github.com/RazrFalcon/rustybuzz and https://github.com/immunant/rexpat https://github.com/immunant/rexpat are making decent progress.
- dnautics 6y agoNo, because zig is still in 0.6.0, and the BDFL says "don't use this in prod yet"? Yeesh.
- catach 6y agoPerhaps then, for the sake of setting those expectations, the wording "the language that may be most poised to achieve this in the future is Zig" would be preferred?
- dnautics 6y agoLast I checked "poised to achieve this" implies the future. Moreover I have specific reasons to say that, namely that zig now ships with C compilation with transparent cross-compilation support so you can use it as a direct replacement for C and (I believe make), so the statement is based on current zig features whose importance I alluded to in the comment. Fwiw this is also a real problem as I am often hearing complaints about windows ABI support in elixir, anyways. Moreover, having written nifs, I have to say the c header for nifs is basically unintelligible without manually parsing the DEFINES because of windows cross-support needs. Please let me know if I'm using "poised" incorrectly.
- fortran77 6y agoWe do a lot of Erlang work here. The BEAM is so reliable that I wouldn't spend 1 minute looking at an experimental alternative.
- carapace 6y agoYeah, this. I'm just getting started with Erlang and I already feel like an idiot for not using it sooner. When I think of some of the things I've done to try to achieve what the BEAM does out-of-the-box...
- rhlsthrm 6y agoCan you give some examples? I've been getting more and more interested in Erlang.
- battery_cowboy 6y agoRPC is basically built in, so you'll probably never use REST internally. There's an in memory database built in (ETS) that will replace redis for most key value storage cases. There's easy recovery from crashes via supervision trees and associated features. You can do hot upgrades while your system is fully operational.
- dnautics 6y agoA few protips: hot reloads are outside of a few corner cases generally unnecessary these days. Be careful about erlang rpc if you expose it in any fashion over the network and follow the eef security working group guidelines re: erlang term format. I use Elixir's plug.crypto.non_executable_binary_to_term/2
- battery_cowboy 6y agoThanks, I knew both of those bits of info, but I didn't know about the guidelines and how to deal with the RPC issue, very helpful.
- conradfr 6y agoThere is some go projects like [1] that can connect to Erlang nodes and claim to be speedier. [1] https://github.com/halturin/ergo https://github.com/halturin/ergo
- jlg23 6y agoJust being able to amend job requirements with "or rust experience" is most probably worth it.
- d4mi3n 6y agoI love seeing new implementations of popular languages. Curious: Did implementing this in Rust expose any bad or interesting behavior when replicating the Erlang language spec (https://github.com/erlang/spec https://github.com/erlang/spec) or whatever reference implementation you were targeting?
- archseer 6y agoIf I remember correctly I found a few edge cases, but they weren't ever hit by OTP, just by partially implemented VMs like mine :) It was kind of interesting exploring the OTP internals, especially some of the parts that haven't changed in a long time. One example is the PAM: I think it stood for "patrick's abstract machine" and it would compile erlang terms into bytecode for pattern matches (intended for fast ETS lookups). It's all there in one file and it took a fair bit of digging to figure out how it works since it's been static for a long while and nothing on the internet really documented it.
- d4mi3n 6y agoHah! Great find! If this is still the case you should definitely consider contributing to the documentation of those files. Odds are they'll be used by the next person to try something similar. :)
- callamdelaney 6y agoThe pattern matching algorithm was originally based on the algorithm described in `The implementation of Functional Programming Languages`, the 1987 edition (there are two versions, one is more basic). Edit: this book is available for free here: https://www.microsoft.com/en-us/research/publication/the-implementation-of-functional-programming-languages https://www.microsoft.com/en-us/research/publication/the-imp...
- foota 6y agoI was really hoping this comment was going to be by patrick in that weird synchronicity that is demonstrated on hn :-)
- adamnemecek 6y agoDo you think it's reasonable to make a Rust library that allows you to do Erlang style binary matching? That's what I was originally looking for when I found this.
- swsieber 6y agoIf your okay with macros, probably nightly only, then it seems reasonable. There is also slice matching on stable, which let's you match on parts of slices: https://github.com/rust-lang/rust/pull/67712/ https://github.com/rust-lang/rust/pull/67712/ . It went out in 1.42. It has some stuff which makes binary stuff easier, but not by much. But perhaps someday you'll get native binary matching in the language that's closer to what Erlang offers. It made it in the 1.42 release.
- archseer 6y ago^ agree with this, slice_patterns fulfilled my needs on the byte level at least. Bit level you could extract some parts of https://github.com/archseer/enigma/blob/master/enigma/src/bitstring.rs https://github.com/archseer/enigma/blob/master/enigma/src/bi... and wrap them in a macro but it's a bit clunky. I tend to just do it by hand (match a { a if a ^ 0b1100_0000 => ... } etc)
- organicfigs 6y agoNice work! If I wanted to learn how to build VMs, how would I start? My experience is in backend development/distributed systems in Java and Go (so assume I know nothing outside of an introductory OS course)
- Aqua_Geek 6y agoThe "Crafting Interpreters" book by Bob Nystrom is probably a good way to dig in. It has a whole chunk on implementing a VM: http://craftinginterpreters.com/a-bytecode-virtual-machine.html http://craftinginterpreters.com/a-bytecode-virtual-machine.h...
- shijie 6y agoThank you for posting this! Thanks to you I'm digging in to this book right now and having a blast. The author is an engaging writer and it's been tremendous fun thus far.
- Aqua_Geek 6y agoHis book, "Game Programming Patterns," is great as well: http://gameprogrammingpatterns.com http://gameprogrammingpatterns.com
- organicfigs 6y agoThis is pretty engaging, thank you!
- tomp 6y agoHow did you implement the GC? Is it possible to implement an allocator + GC in Rust without hitting UB?
- jacquesm 6y agoWhy start if you have no intention to see it through? Talks and adoption by the community are a chicken and egg problem, if you don't believe enough in the project to give it staying power then the community is right to not adopt it: they already have a VM for BEAM and it works well. Without an additional selling point 'now in Rust' doesn't cut it.
- busterarm 6y agoI see this tendency a lot from the Rust community where there's a lot of "now in Rust" being built, expectations had and then hurt feelings when they're either ignored or shown the door. The community seems to think that "now in Rust" _is_ the selling point. Tools are just tools. What they don't realize is that they're often building solutions that are looking for problems, rather than solutions to solve problems. It's also vaguely cultish in the approach. It's a terrific language and there's a lot of learn from it, but I'd like to see it solve real world problems on its own versus try and screw itself into everyone else's.
- 59nadir 6y agoWhile it's undeniable the majority of things to come out of Rust are mostly superfluous rewrites of already solid projects (to your point about "now in Rust!" being the selling point) I think it's clear the author in this particular case was just looking into BEAM internals and started a fun project, so I don't think it applies here. In general, though, I think people ought to consider that if they are putting the language they wrote their project in in the marketing blurb for it (given a more serious project), maybe that indicates that the project itself is of little value to other people. "* Written in Rust" isn't a value proposition, it's just an implementation detail. Make real claims about zero crashes, zero leaks, something actually concrete and it can be scrutinized for real.
- chc 6y agoI think you're possibly making a faulty assumption here. You're right that "written in Rust" has no particular value to people who just want to use the thing, but that's not the only perspective people bring to open-source software. When somebody markets a project as "X written in Y," I generally assume they're marketing to people who might want to hack on it, and it is relevant from that perspective.
- songshuu 6y agoA VM is more than enough of a project, but are there any thoughts of a Phoenix port?
- ethelward 6y agoWhy would you want to port Phoenix? As long as the underlying VM follow the spec, Phoenix should be oblivious of on which it is running.
- pdimitar 6y agoHave you considered using C2Rust[0] where applicable instead? [0] https://c2rust.com/ https://c2rust.com/