5 ms·

> What does this mean? Erlang/Elixir are AOT-compiled languages.. With the BEAM compiler today (and by BEAM compiler, they are referring to the compiler provid

> What does this mean? Erlang/Elixir are AOT-compiled languages.. With the BEAM compiler today (and by BEAM compiler, they are referring to the compiler provided as part of the standard library shipped with the BEAM, it is just a way to clarify which Erlang compiler is referred to), Erlang sources are partially AOT-compiled to BEAM bytecode, then the code loader does additional compilation steps at runtime. Firefly AOT compiles Erlang to native code directly, and does not use a virtual machine at runtime. > So is the key benefit here that the compilation is faster because it's not running on the BEAM (good for e.g. CI); or is the key benefit that the resulting executable is faster because it's not running on the BEAM? There are a few benefits we hope to provide using this approach: 1. Compilation can be faster because the compiler is implemented in Rust, rather than implemented in Erlang and running on the BEAM. 2. We impose a restriction that hot code loading is not permitted, so we are able to do forms of optimization that the BEAM cannot do, as a result of having to pessimize in the presence of hot code loading. 3. Related to above, we can do whole program analysis, including optimizations that take advantage of unboxing terms and working with more machine-friendly types. 4. We can produce a single statically-linked executable that is as small as possible, in part due to being able to do more aggressive and precise dead-code elimination across both the application being compiled and the runtime it links to, because we know what must be available at runtime. On the other hand, if one were to ship the BEAM itself to run in browsers via Wasm (if it was modified in such a way as to be possible), you'd also need to ship large amounts of bytecode for most applications, regardless of whether much of that bytecode is actually needed at runtime. Deployment has always been a pain point of the BEAM (and I'm speaking as someone who built the primary release tools for the Elixir ecosystem), Firefly aims to make deployment as simple as Go/Rust/etc. 5. I'd like to be able to take advantage of the fact that we use MLIR behind the scenes to explore using Firefly as a natural pairing with Elixir applications using Nx, by being able to more seamlessly integrate regular Elixir code with Nx-managed functions. > How does a non-bytecode version of Erlang/Elixir abstract-machine semantics, achieve these same guarantees? Is there an explicit reduction-counter being carried around in the emitted native code? In short, yes, we implement preemptive-scheduling the same way the BEAM does, using compiler-injected yield points based on a few criteria (a reduction counter is just one, some others are garbage collection and blocking I/O). > And also, there is no mention of disadvantages/constraints of using this system. It's pretty clear that you wouldn't be able to do hot reloading or dynamic trace-point insertion without the BEAM there to intermediate it. That's fine for some use-cases, but they should explicitly mention the trade-offs and target audience. The readme of the project certainly does this, but I agree it would have been good to include in the blog post. In any case, Firefly is still in early stages, so this isn't something anyone is using today. When we reach the point where we feel it is production ready, I can assure you I will be writing up a very detailed analysis of what it is ideally suited for, and what it is not - like you, I feel it is critically important to be clear about that.

agent281 4y ago
Thank you for elaborating! I think that Erlang/Elixir could make amazing TUI apps if they were easier to distribute. The ability to push work to the edges of the system while keeping the UI responsive would be awesome. It sounds like static binaries provided by Firefly would make this a viable option.
- freedomben 4y ago
  Likewise. I still mainly use Ruby or Go for writing CLIs, but I would much, much rather use Elixir. escript is nice but still requires the target to have erlang installed. Being able to produce a statically-linked binary would be enough to just use elixir only, which is my dream.
  - bitwalker 4y ago
    This is one of the things I'm personally excited about using Firefly for, since I first started working professionally with Erlang/Elixir, I wanted the ability to use it for CLIs. You can be sure it will be a use case well supported :)
derefr 4y ago
> Compilation can be faster because the compiler is implemented in Rust, rather than implemented in Erlang and running on the BEAM. If this is a full-blown Erlang/Elixir (or only Elixir?) compiler, rather than a "BEAM IR to native" compiler, that sure sounds like a lot of independent work to maintain, especially without there being any sort of spec/standard defining what the behavior of such a compiler should be separate from what the reference compiler does, and what the resulting abstract machine should do separate from what the reference VM does. How do you plan on ensuring your compiler tracks updates introduced by major Erlang releases? Will there be features introduced in the Erlang runtime (I'm thinking things like Erlang 21's `atomics`) that won't be immediately available for FireFly-compiled programs, where FireFly will just choke on these? > We impose a restriction that hot code loading is not permitted, so we are able to do forms of optimization that the BEAM cannot do, as a result of having to pessimize in the presence of hot code loading. Hot code loading happens in places other than just relups, though, no? There are some Erlang/Elixir libraries which do crazy things at runtime — for example, protocol serialization libraries which, at runtime, discover remote schemas; dynamically generate code to ser/des values for those schema; compile that code to a module in memory; and then load the resulting module. And this is not discouraged/disincentivized in the Erlang ecosystem; the compiler infrastructure is usually considered to be "part of the standard library," for any application to use freely at runtime. So even if I don't do any of this in my own project, I would be worried that some transitive dependency of my project would be secretly doing this. Even if you have no plans to support hot code loading in the "remote calls can jump into the new version of a module" sense, do you think it would make sense for FireFly to ever support "module compilation+loading at runtime"? Maybe with the resulting modules being native DLLs? --- Tangent to that — and I know that this was probably nowhere on your mind when you were working on the project, but something interesting to consider: how hard would it be to convince the FireFly compiler to emit a C-ABI library that could be loaded into the BEAM as a NIF? I ask, because this would be a really interesting way of achieving the same sorts of speedups what HiPE does/did — but more explicitly, on a higher unit level, and (probably) better.

agent281 4y ago

Thank you for elaborating! I think that Erlang/Elixir could make amazing TUI apps if they were easier to distribute. The ability to push work to the edges of the system while keeping the UI responsive would be awesome. It sounds like static binaries provided by Firefly would make this a viable option.

freedomben 4y ago
Likewise. I still mainly use Ruby or Go for writing CLIs, but I would much, much rather use Elixir. escript is nice but still requires the target to have erlang installed. Being able to produce a statically-linked binary would be enough to just use elixir only, which is my dream.
- bitwalker 4y ago
  This is one of the things I'm personally excited about using Firefly for, since I first started working professionally with Erlang/Elixir, I wanted the ability to use it for CLIs. You can be sure it will be a use case well supported :)

derefr 4y ago

> Compilation can be faster because the compiler is implemented in Rust, rather than implemented in Erlang and running on the BEAM. If this is a full-blown Erlang/Elixir (or only Elixir?) compiler, rather than a "BEAM IR to native" compiler, that sure sounds like a lot of independent work to maintain, especially without there being any sort of spec/standard defining what the behavior of such a compiler should be separate from what the reference compiler does, and what the resulting abstract machine should do separate from what the reference VM does. How do you plan on ensuring your compiler tracks updates introduced by major Erlang releases? Will there be features introduced in the Erlang runtime (I'm thinking things like Erlang 21's `atomics`) that won't be immediately available for FireFly-compiled programs, where FireFly will just choke on these? > We impose a restriction that hot code loading is not permitted, so we are able to do forms of optimization that the BEAM cannot do, as a result of having to pessimize in the presence of hot code loading. Hot code loading happens in places other than just relups, though, no? There are some Erlang/Elixir libraries which do crazy things at runtime — for example, protocol serialization libraries which, at runtime, discover remote schemas; dynamically generate code to ser/des values for those schema; compile that code to a module in memory; and then load the resulting module. And this is not discouraged/disincentivized in the Erlang ecosystem; the compiler infrastructure is usually considered to be "part of the standard library," for any application to use freely at runtime. So even if I don't do any of this in my own project, I would be worried that some transitive dependency of my project would be secretly doing this. Even if you have no plans to support hot code loading in the "remote calls can jump into the new version of a module" sense, do you think it would make sense for FireFly to ever support "module compilation+loading at runtime"? Maybe with the resulting modules being native DLLs? --- Tangent to that — and I know that this was probably nowhere on your mind when you were working on the project, but something interesting to consider: how hard would it be to convince the FireFly compiler to emit a C-ABI library that could be loaded into the BEAM as a NIF? I ask, because this would be a really interesting way of achieving the same sorts of speedups what HiPE does/did — but more explicitly, on a higher unit level, and (probably) better.