10 ms·

Lumen: An alternative BEAM implementation, designed for WebAssembly

218 points by seeekr 7y ago

juhatl 7y ago
I watched the ElixirConf 2019 keynote [1] as soon as it came up. While I was left excited about Lumen in general, I wasn't entirely sure what kinds of things Lumen would be most useful for. Being able to ship a single binary (like with Go or Rust) sounds handy, but shipping a tarball hasn't been that much of a challenge either. Being able to run BEAM code in the browser also sounds handy, but due to size issues I probably wouldn't want to ship the entire BEAM to my users either, right? Can someone with a more thorough understanding of Lumen share some thoughts on this? [1] https://www.youtube.com/watch?v=uMgTIlgYB-U https://www.youtube.com/watch?v=uMgTIlgYB-U
- rehemiau 7y ago
  I think they mentioned this in the keynote, but bytecode size is on their radar. To allow dead code elimination, they are not planning to support hot upgrades. This means that you wouldn't be shipping the whole OTP - only the parts that you're using.
  - johnisgood 7y ago
    What would hot upgrades entail in the context of Web and for example Lumen? What would its benefits or drawbacks be? Would it be useful? Would it raise security concerns? Just want to play with the idea. :)
    - sansnomme 7y ago
      If Wasm does become as pervasive as JVM i.e. a universal VM of sorts, there will be plenty of IoT equipment using it since the trend now for anything "embedded" that doesn't demand ultra low power consumption is to use a full Linux system à la Raspberry Pi. A company optimizing for a tight memory/performance budget and whose purpose does not require high uptime across the entire fleet can simply use traditional blue-green deployments or something.
- 693471 7y ago
  You won't be shipping the entirety of BEAM. That's the point of this announcement and what makes this so exciting and plausible.
moomin 7y ago
So, an alternative BEAM implementation that behaves like a regular VM rather than BEAM? Not saying it’s useless, but it’s sure idiosyncratic.
lxrbst 7y ago
Cool idea, cool technology, cool project. I really just don't get the "why", even after watching the ElixirConf presentation [1] about it. Like what is the use-case for this, specifically? [1] https://www.youtube.com/watch?v=uMgTIlgYB-U https://www.youtube.com/watch?v=uMgTIlgYB-U
- nickjj 7y ago
  Same. There was a quick slide joking about how this solves the fully offline client side browser app problem but even Chris McCord said that in all his years of consulting, he's only gotten a handful of requests for these types of apps. I'm sure they have their reasons for taking on such a project but as an outsider I would have liked to see more announcements that apply to what the majority of people are using Elixir for today (web apps). There's Nerves too which I'm not into personally, but there were things at the conf for that so I'm sure that crowd was happy. For example, telemetry was announced over a year ago with a hint that it was being worked on next and will be implemented into Phoenix as a web dashboard in the future but there was no hint of that mentioned this year -- at least not in any of the Phoenix related talks and Chris' keynote. I also didn't see it in any of the other talk's titles. But telemetry is one of those things where if it existed in the form they talked about last year it would be one of the biggest killer features of any web framework available today (not a single one does what they proposed).
  - anthony_doan 7y ago
    Current momentum is in liveview. I'm actually eagerly waiting for that to get up to a good version after the early adopters.
    - nickjj 7y ago
      Yeah, LV is definitely worthy. I'm excited for it too. Once LV drops with debounce / file upload support that will be enough to get me to incorporate it into an existing project.
  - hauleth 7y ago
    > hint that it was being worked on next and will be implemented into Phoenix as a web dashboard in the future but there was no hint of that mentioned this year Currently there is EEF Observability WG that is working on integrating OpenTelemetry, so it will not be integrated into Phoenix, but will be more general and more cross-language solution for monitoring. Source: I am part of that WG.
pron 7y ago
While WASM makes for a convenient compilation target for languages like C/C++/Rust/Zig, isn't JS a better target for languages that can benefit from a JIT and/or a GC? You'd get better performance for less effort and a smaller deliverable. Why compile to WASM when JS seems, at least to me, an obviously superior target? What great advantage does WASM have that it makes it worthwhile to lose on three important fronts?
- anthony_doan 7y ago
  Wait what? WASM is a vm and JS is a language. Are you asking about the pro and con of a transpiler over a vm solution? For one thing BEAM is the Erlang VM which Elixir and many other languages run on. Why would anybody want to implement it in Javascript? BEAM does a lot of crazy things, including a preemptive scheduler. Also BEAM GC isn't some generic GC, there's tons of crazy work going on with it to work with the Actor concurrency model.
  - pron 7y ago
    > WASM is a vm and JS is a language. I'm talking about compiling to WASM vs. compiling to JS. Calling something a VM or a language is mostly a matter of convention. x86 machine code can also be viewed as a language, and compilers certainly don't care. A compiler translates code in one language into code in another (there is no precise definition for "transpile", which is why compiler people hate that word; colloquially "transpile" just means that the compilation's target language is considered to be at a similar abstraction level to the source). > Why would anybody want to implement it in Javascript? Why would anybody want to implement it in WASM? If it's so that it could run in the browser, then compiling BEAM bytecode to JS would achieve the same goal, be easier to write and would likely result in a smaller and faster deliverable. > Also BEAM GC isn't some generic GC, there's tons of crazy work going on with it to work with the Actor concurrency model. There's nothing crazy going on in BEAM. As far as VMs go, it's near the bottom in terms of sophistication.
    - ohyes 7y ago
      Exactly true! C, for example is a VM!
tiborsaas 7y ago
This is the gist of it: "The primary motivator for Lumen's development was the ability to compile Elixir applications that could target WebAssembly, enabling use of Elixir as a language for frontend development." I had to dig to lean what BEAM is :) "Bogumil Hausman next machine was called BEAM (Bogdan’s Erlang Abstract Machine). It was a hybrid machine that could execute both native code and threaded code with an interpreter. That allowed customers to compile their time-critial modules to native code and all other modules to threaded BEAM code. The threaded BEAM in itself was faster than JAM code." http://blog.erlang.org/beam-compiler-history/ http://blog.erlang.org/beam-compiler-history/ I now feel a little less bad that I couldn't figure out the acronym :)
- conradfr 7y ago
  Well if you've never worked with Erlang or Elixir it's quite normal to not know that.
- ilaksh 7y ago
  The readme should have a one sentence explanation of BEAM or at least mention "Erlang VM" in parenthesis.
bcardarella 7y ago
Hi I'm Brian, one of the member of the Lumen team. I'll try to address some of the questions I've seen in this thread: 1. Why? We've seen every growing complexity in building, maintaining, extending, and debugging client side applications. There are several reasons why this is but in our opinion how JS handles concurrency (the Event Loop) is a primary factor. Erlang's concurrency model is superior (also our opinion) in how a developer reasons about concurrency. We want to bring this power to building client side web apps. 2. I'm afraid of [it] being over-engineered, and hard to contribute to. Lumen itself is just the beginning of what we hope is an eco system of tooling and potential framework pieces for building web applications. When building Elixir or Erlang applications developers are not expected to be contributing back to the BEAM or to even understand how it works. Lumen will be in a similar position. When we finally get to `mix install lumen` you will likely only be including Lumen as a dependency in your apps. Higher level functionality will emerge that use Lumen to compile to WASM. However, we highly encourage people with experience in Rust, compiler design, runtime design, Erlang/BEAM, or just general curiosity to get involved in developing Lumen :) 3. even Chris McCord said that in all his years of consulting, he's only gotten a handful of requests for these types of apps Chris works here at DockYard, and this is because we don't put Chris on our PWA/SPA projects. While there is an obvious demand for LiveView to relieve people from needing a client side framework there are most definitely many clients and plenty of projects where client side apps make sense. 4.an alternative BEAM implementation that behaves like a regular VM rather than BEAM? Not entirely, although some concessions need to be made when working within the security model of the web. I would very much like it is the browser vendors would give up the argument that Web Workers fulfill the threading requirement of WASM as I personally feel that is disingenuous. I'm hoping that as Lumen develops and (hopefully) picks up steam/influence we can start to see some of these features land for real. 5. Due to size issues I probably wouldn't want to ship the entire BEAM to my users Correct, and we aren't advocating for this. Lumen is currently in a very early stage of development and we haven't even implemented any compilation optimizations like dead code elimination. There are many other WASM specific optimizations that could cut the footprint size by 50% and this is before we even gzip, which WASM is designed to respond well to despite being a binary. I'll keep a running list of responses in this thread if more questions come up.
- 7y ago
mrdoops 7y ago
My favorite part about the presentation was the correlations between the DOM tree and a supervision tree. If you've looked into the Scenic project (https://github.com/boydm/scenic https://github.com/boydm/scenic) you know how this Process & Supervision model shows real promise for client-side UI concurrency. As your states are changing you don't want one bad state to crash and cascade elsewhere - supervision trees lets you contain and recover from failures gracefully. I'll be honest I didn't think it would be possible to implement something like the BEAM's preemptive scheduler in WebAssembly until I saw the presentation. Turns out it's very doable with continuations. The timing should be right in the next couple years as both Lumen and some of the bigger components of WebAssembly maturity kick into gear. Similar high level language/runtime -> WebAssembly projects like Blazor(C#) are also pending features like more native control over the DOM without going through JS interfaces (see here: https://github.com/WebAssembly/interface-types/blob/master/proposals/interface-types/Explainer.md https://github.com/WebAssembly/interface-types/blob/master/p... for more technical details). Even if client-side UI frameworks don't pan out for Lumen, I can see Lumen being valuable for edge-computation especially where concurrency and fault-tolerance are important.
- bcardarella 7y ago
  Thanks, we are not entirely convinced if a 1:1 relationship of process to DOM element is going to be where we land ultimately but it most definitely is a fun and interesting experiment. And it does align well with many of the OTP concepts. It really comes down to performance numbers and further research to know what the best direction a rendering engine should take. Personally, unless a rendering engine in Lumen is doing something OTPish (i.e. supervisor trees) I don't see the reason why people would choose a Lumen solution over existing works. I think I mentioned it in the keynote but the performance is secondary to the architectural perspective that Lumen can bring to client side development. As Lumen continues to develop we'll continue to experiment with ideas and see what works best. I actually have a dbmon demo built but Hans and Luke were hand compiling the source and the DOM ast was just too much which is why the presentation had a very simple div/p tag. Once the compiler is ready for use I hope to write more about the dbmon demo and show of some crashing table rows. Should be cool!
artfulhippo 7y ago
This is some of the best news this summer! Anyone that wants to reduce the influence that Google and Facebook have on the Web should look into the Erlang/Elixir ecosystem.
- moosingin3space 7y ago
  I'm excited about this project too, but how does it "reduce the influence that Google and Facebook have on the Web"? It seems pretty orthogonal if you ask me.
  - bcardarella 7y ago
    Considering how much sway Google has over the WASM spec group I would say that Google still has considerable influence upon this project.
namelosw 7y ago
Interesting project. Although it's viable to transpile Erlang or Elixir to JavaScript, without the BEAM runtime everything seems pointless. However if an BEAM is viable we could do interesting things like React fiber equivalent could just be processes.

218 points by seeekr 7y ago

I watched the ElixirConf 2019 keynote [1] as soon as it came up. While I was left excited about Lumen in general, I wasn't entirely sure what kinds of things Lumen would be most useful for. Being able to ship a single binary (like with Go or Rust) sounds handy, but shipping a tarball hasn't been that much of a challenge either. Being able to run BEAM code in the browser also sounds handy, but due to size issues I probably wouldn't want to ship the entire BEAM to my users either, right? Can someone with a more thorough understanding of Lumen share some thoughts on this? [1] https://www.youtube.com/watch?v=uMgTIlgYB-U https://www.youtube.com/watch?v=uMgTIlgYB-U

rehemiau 7y ago
I think they mentioned this in the keynote, but bytecode size is on their radar. To allow dead code elimination, they are not planning to support hot upgrades. This means that you wouldn't be shipping the whole OTP - only the parts that you're using.
- johnisgood 7y ago
  What would hot upgrades entail in the context of Web and for example Lumen? What would its benefits or drawbacks be? Would it be useful? Would it raise security concerns? Just want to play with the idea. :)
  - sansnomme 7y ago
    If Wasm does become as pervasive as JVM i.e. a universal VM of sorts, there will be plenty of IoT equipment using it since the trend now for anything "embedded" that doesn't demand ultra low power consumption is to use a full Linux system à la Raspberry Pi. A company optimizing for a tight memory/performance budget and whose purpose does not require high uptime across the entire fleet can simply use traditional blue-green deployments or something.
693471 7y ago
You won't be shipping the entirety of BEAM. That's the point of this announcement and what makes this so exciting and plausible.

moomin 7y ago

So, an alternative BEAM implementation that behaves like a regular VM rather than BEAM? Not saying it’s useless, but it’s sure idiosyncratic.

lxrbst 7y ago

Cool idea, cool technology, cool project. I really just don't get the "why", even after watching the ElixirConf presentation [1] about it. Like what is the use-case for this, specifically? [1] https://www.youtube.com/watch?v=uMgTIlgYB-U https://www.youtube.com/watch?v=uMgTIlgYB-U

nickjj 7y ago
Same. There was a quick slide joking about how this solves the fully offline client side browser app problem but even Chris McCord said that in all his years of consulting, he's only gotten a handful of requests for these types of apps. I'm sure they have their reasons for taking on such a project but as an outsider I would have liked to see more announcements that apply to what the majority of people are using Elixir for today (web apps). There's Nerves too which I'm not into personally, but there were things at the conf for that so I'm sure that crowd was happy. For example, telemetry was announced over a year ago with a hint that it was being worked on next and will be implemented into Phoenix as a web dashboard in the future but there was no hint of that mentioned this year -- at least not in any of the Phoenix related talks and Chris' keynote. I also didn't see it in any of the other talk's titles. But telemetry is one of those things where if it existed in the form they talked about last year it would be one of the biggest killer features of any web framework available today (not a single one does what they proposed).
- anthony_doan 7y ago
  Current momentum is in liveview. I'm actually eagerly waiting for that to get up to a good version after the early adopters.
  - nickjj 7y ago
    Yeah, LV is definitely worthy. I'm excited for it too. Once LV drops with debounce / file upload support that will be enough to get me to incorporate it into an existing project.
- hauleth 7y ago
  > hint that it was being worked on next and will be implemented into Phoenix as a web dashboard in the future but there was no hint of that mentioned this year Currently there is EEF Observability WG that is working on integrating OpenTelemetry, so it will not be integrated into Phoenix, but will be more general and more cross-language solution for monitoring. Source: I am part of that WG.

pron 7y ago

While WASM makes for a convenient compilation target for languages like C/C++/Rust/Zig, isn't JS a better target for languages that can benefit from a JIT and/or a GC? You'd get better performance for less effort and a smaller deliverable. Why compile to WASM when JS seems, at least to me, an obviously superior target? What great advantage does WASM have that it makes it worthwhile to lose on three important fronts?

anthony_doan 7y ago
Wait what? WASM is a vm and JS is a language. Are you asking about the pro and con of a transpiler over a vm solution? For one thing BEAM is the Erlang VM which Elixir and many other languages run on. Why would anybody want to implement it in Javascript? BEAM does a lot of crazy things, including a preemptive scheduler. Also BEAM GC isn't some generic GC, there's tons of crazy work going on with it to work with the Actor concurrency model.
- pron 7y ago
  > WASM is a vm and JS is a language. I'm talking about compiling to WASM vs. compiling to JS. Calling something a VM or a language is mostly a matter of convention. x86 machine code can also be viewed as a language, and compilers certainly don't care. A compiler translates code in one language into code in another (there is no precise definition for "transpile", which is why compiler people hate that word; colloquially "transpile" just means that the compilation's target language is considered to be at a similar abstraction level to the source). > Why would anybody want to implement it in Javascript? Why would anybody want to implement it in WASM? If it's so that it could run in the browser, then compiling BEAM bytecode to JS would achieve the same goal, be easier to write and would likely result in a smaller and faster deliverable. > Also BEAM GC isn't some generic GC, there's tons of crazy work going on with it to work with the Actor concurrency model. There's nothing crazy going on in BEAM. As far as VMs go, it's near the bottom in terms of sophistication.
  - ohyes 7y ago
    Exactly true! C, for example is a VM!

tiborsaas 7y ago

This is the gist of it: "The primary motivator for Lumen's development was the ability to compile Elixir applications that could target WebAssembly, enabling use of Elixir as a language for frontend development." I had to dig to lean what BEAM is :) "Bogumil Hausman next machine was called BEAM (Bogdan’s Erlang Abstract Machine). It was a hybrid machine that could execute both native code and threaded code with an interpreter. That allowed customers to compile their time-critial modules to native code and all other modules to threaded BEAM code. The threaded BEAM in itself was faster than JAM code." http://blog.erlang.org/beam-compiler-history/ http://blog.erlang.org/beam-compiler-history/ I now feel a little less bad that I couldn't figure out the acronym :)

conradfr 7y ago
Well if you've never worked with Erlang or Elixir it's quite normal to not know that.
ilaksh 7y ago
The readme should have a one sentence explanation of BEAM or at least mention "Erlang VM" in parenthesis.

bcardarella 7y ago

Hi I'm Brian, one of the member of the Lumen team. I'll try to address some of the questions I've seen in this thread: 1. Why? We've seen every growing complexity in building, maintaining, extending, and debugging client side applications. There are several reasons why this is but in our opinion how JS handles concurrency (the Event Loop) is a primary factor. Erlang's concurrency model is superior (also our opinion) in how a developer reasons about concurrency. We want to bring this power to building client side web apps. 2. I'm afraid of [it] being over-engineered, and hard to contribute to. Lumen itself is just the beginning of what we hope is an eco system of tooling and potential framework pieces for building web applications. When building Elixir or Erlang applications developers are not expected to be contributing back to the BEAM or to even understand how it works. Lumen will be in a similar position. When we finally get to `mix install lumen` you will likely only be including Lumen as a dependency in your apps. Higher level functionality will emerge that use Lumen to compile to WASM. However, we highly encourage people with experience in Rust, compiler design, runtime design, Erlang/BEAM, or just general curiosity to get involved in developing Lumen :) 3. even Chris McCord said that in all his years of consulting, he's only gotten a handful of requests for these types of apps Chris works here at DockYard, and this is because we don't put Chris on our PWA/SPA projects. While there is an obvious demand for LiveView to relieve people from needing a client side framework there are most definitely many clients and plenty of projects where client side apps make sense. 4.an alternative BEAM implementation that behaves like a regular VM rather than BEAM? Not entirely, although some concessions need to be made when working within the security model of the web. I would very much like it is the browser vendors would give up the argument that Web Workers fulfill the threading requirement of WASM as I personally feel that is disingenuous. I'm hoping that as Lumen develops and (hopefully) picks up steam/influence we can start to see some of these features land for real. 5. Due to size issues I probably wouldn't want to ship the entire BEAM to my users Correct, and we aren't advocating for this. Lumen is currently in a very early stage of development and we haven't even implemented any compilation optimizations like dead code elimination. There are many other WASM specific optimizations that could cut the footprint size by 50% and this is before we even gzip, which WASM is designed to respond well to despite being a binary. I'll keep a running list of responses in this thread if more questions come up.

7y ago

mrdoops 7y ago

My favorite part about the presentation was the correlations between the DOM tree and a supervision tree. If you've looked into the Scenic project (https://github.com/boydm/scenic https://github.com/boydm/scenic) you know how this Process & Supervision model shows real promise for client-side UI concurrency. As your states are changing you don't want one bad state to crash and cascade elsewhere - supervision trees lets you contain and recover from failures gracefully. I'll be honest I didn't think it would be possible to implement something like the BEAM's preemptive scheduler in WebAssembly until I saw the presentation. Turns out it's very doable with continuations. The timing should be right in the next couple years as both Lumen and some of the bigger components of WebAssembly maturity kick into gear. Similar high level language/runtime -> WebAssembly projects like Blazor(C#) are also pending features like more native control over the DOM without going through JS interfaces (see here: https://github.com/WebAssembly/interface-types/blob/master/proposals/interface-types/Explainer.md https://github.com/WebAssembly/interface-types/blob/master/p... for more technical details). Even if client-side UI frameworks don't pan out for Lumen, I can see Lumen being valuable for edge-computation especially where concurrency and fault-tolerance are important.

bcardarella 7y ago
Thanks, we are not entirely convinced if a 1:1 relationship of process to DOM element is going to be where we land ultimately but it most definitely is a fun and interesting experiment. And it does align well with many of the OTP concepts. It really comes down to performance numbers and further research to know what the best direction a rendering engine should take. Personally, unless a rendering engine in Lumen is doing something OTPish (i.e. supervisor trees) I don't see the reason why people would choose a Lumen solution over existing works. I think I mentioned it in the keynote but the performance is secondary to the architectural perspective that Lumen can bring to client side development. As Lumen continues to develop we'll continue to experiment with ideas and see what works best. I actually have a dbmon demo built but Hans and Luke were hand compiling the source and the DOM ast was just too much which is why the presentation had a very simple div/p tag. Once the compiler is ready for use I hope to write more about the dbmon demo and show of some crashing table rows. Should be cool!

artfulhippo 7y ago

This is some of the best news this summer! Anyone that wants to reduce the influence that Google and Facebook have on the Web should look into the Erlang/Elixir ecosystem.

moosingin3space 7y ago
I'm excited about this project too, but how does it "reduce the influence that Google and Facebook have on the Web"? It seems pretty orthogonal if you ask me.
- bcardarella 7y ago
  Considering how much sway Google has over the WASM spec group I would say that Google still has considerable influence upon this project.

namelosw 7y ago

Interesting project. Although it's viable to transpile Erlang or Elixir to JavaScript, without the BEAM runtime everything seems pointless. However if an BEAM is viable we could do interesting things like React fiber equivalent could just be processes.