4 ms·

> The need for concurrency and fault-tolerance made Elixir an obvious choice for this application, but that alone wasn't enough to seal the deal. What? Why doe

> The need for concurrency and fault-tolerance made Elixir an obvious choice for this application, but that alone wasn't enough to seal the deal. What? Why does this make Elixir an obvious choice? Most (if not all) major languages offer concurrency primitives (e.g. goroutines), and fault-tolerance is included as a requirement by default for any sufficiently complex distributed system, but has little to do with the languages/frameworks used to build that system. Not seeing why these requirements make Elixir a better choice than any other language

pawelduda 5y ago
I think it's because Elixir runs in Erlang VM which was built from the ground-up to satisfy these requirements... And it's been around for a very long time with very good track record.
- jimsimmons 5y ago
  That really doesn’t answer the question. What does the BEAM VM provide that cannot be implemented in Rust or Go or even Python.
  - brightball 5y ago
    Immutable memory, preemptive scheduling, per process garbage collection with isolated heap and a 0.5kb memory footprint per process for starters.
    - jimsimmons 5y ago
      Immutable memory — writing functional Python or Rust can come close. Pre emp scheduling — I assume this is to prevent deadlocks? This and everything else is very nice but do we need a full VM for that? I’m not trying to be a jerk but I want to understand why this isn’t a feature implementable as a runtime.
      - SirGiggles 5y ago
        > Pre emp scheduling Userspace pre-emptive schedulers do not incur the same level of overhead that kernel threads would normally have. To address another comment you made in another chain, Erlang accomplishes pre-emption through reductions which is roughly equivalent to ~4000 function calls, if I remember correctly and if I have read the correct documentation, before it yields the Erlang process. Other forms of async to my knowledge essentially queue a task while the main thread of execution continues before awaiting; and the queued task is run in some form of thread pool (or other forms of executors). Without going too deep into this subject (and assuming I have made no logical errors) it is possible to have misbehaving tasks in async tank performance and latency, while the same misbehaving process in Erlang wouldn't. See Saša Jurić's demonstration [1] for an example. Do keep in mind that async is not forte, what I have stated is merely my observations. I write primarily with kernel threads. This is why it is possible to spin up millions of, in Erlang terms, processes with minimal overhead. See this article by the Phoenix Framework team regarding an experiment on having two million active websocket connections on a single (albeit beefy) host [2]. > This and everything else is very nice but do we need a full VM for that? See [3] regarding an effort to port the Erlang process and supervision strategy to Rust. In my naive view, it is possible to write something similar to BEAM in another language, but the more troubling portion of the work is to jury-rig the original language into a concurrency model it isn't designed for whereas Erlang by contrast was created purely for this task. [1] https://www.youtube.com/watch?v=JvBT4XBdoUE https://www.youtube.com/watch?v=JvBT4XBdoUE [2] https://www.phoenixframework.org/blog/the-road-to-2-million-websocket-connections https://www.phoenixframework.org/blog/the-road-to-2-million-... [3] https://github.com/bastion-rs/bastion https://github.com/bastion-rs/bastion
        jimsimmons 5y ago
        Appreciate the answer. I just saw the Sasa talk the other day incidentally and pre emption value is pretty clear now. It’s so clever and well thought out, I think I start too see value of Erlang now
  - innocentoldguy 5y ago
    Private memory for all processes (Go uses public memory) The actor model. Superior memory usage and CPU processing. Supervision trees. …
    - innocentoldguy 5y ago
      Well, you can downvote me all you want, fuckers, but what I said is accurate.
    - jimsimmons 5y ago
      It’s pretty nifty. But I’m dumbfounded these features haven’t been copied over to other async runtimes. This is why I wonder if there’s a hidden catch that Erlang supporters don’t talk much about
  - dnautics 5y ago
    You can implement those things, but then you'd be implementing them. Suppose you open a file (or a tcp socket, or a database transaction) while being connected over an http connection. How many LOC does it take to make sure that file is closed at the end of the http connection, through all possible error conditions? How confident are you that said code is correct? In elixir, it's 0 LOC.
    - jimsimmons 5y ago
      Nice, finally a straight answer. To be clear, this is because Erlang forces you to think about such failures and also because the entire stack being designed for such fault tolerance, correct?
mikeryan 5y ago
I can understand deciding Elixer is a good solution for for concurrency. What I couldn’t understand from the story was why it was needed. Unless the IDE had some sort of real-time shared coding? There’s nothing in the list of requirements that couldn’t be done over http it all seems asynchronous? Scale here seems like it would be small. Three months is almost zero time for a team of 4. I’m glad it all worked out but I have a harder time from the story understanding how using Elixer was the right choice vs a stack they were more familiar and confident on condieribg the constraints.
- sodapopcan 5y ago
  Immutability is its own reward :) Yes yes, available in other langs, but it kinda takes YAGNI in terms of scale off the table since it's a perfectly good fit for small applications as well. It just happens to scale (in terms of concurrency) really well. But if you never need to use those features that's ok—you'll still have an application written in a really nice, expressively terse language.
macintux 5y ago
> fault-tolerance is included as a requirement by default for any sufficiently complex distributed system, but has little to do with the languages/frameworks used to build that system Erlang/Elixir have an extremely compelling story when it comes to fault-tolerance. You can only argue that all languages are created equal in this regard if you fall back to the "everything is Turing-complete so they're equivalent" argument. Erlang and its VM were built from the ground up to work closely together with fault tolerance in mind, in a context where a few minutes of downtime meant severe financial penalties.
lostcolony 5y ago
That's not really true. Or rather, the need for the distributed system to be fault tolerant is, but downplaying the language/framework isn't entirely. Kubernetes has definitely made it so the distributed tooling of Erlang is less valuable; many of the things it provides you can also get with Kubernetes. The tradeoff being that Kubernetes is far more complex (but, also allows for things Erlang doesn't have). But what Kubernetes can't do is help you write correct programs. That is, if an error happens, what do you do? In most languages, you catch the ones you can predict and handle them. For the others, you let them trickle up until someone catches them, or the app crashes. The problem is figuring out where to handle them, and to make sure you really -are- handling the things you know how to handle. Erlang flips this. Handling things becomes far less important as isolating failure. If something goes wrong ~here~, where else is there state that is likely wrong, and how do we get it all back into a good state? This is a really powerful approach, as it ends up being far more robust, for far less work. It's harder to get 'incorrect' behavior this way; rather than have to enumerate the unhappy paths, and how to address each, you just have to declare dependencies between processes and how, should they find themselves on an unhappy path, to drop their state and get a known good one.

pawelduda 5y ago

I think it's because Elixir runs in Erlang VM which was built from the ground-up to satisfy these requirements... And it's been around for a very long time with very good track record.

jimsimmons 5y ago
That really doesn’t answer the question. What does the BEAM VM provide that cannot be implemented in Rust or Go or even Python.
- brightball 5y ago
  Immutable memory, preemptive scheduling, per process garbage collection with isolated heap and a 0.5kb memory footprint per process for starters.
  - jimsimmons 5y ago
    Immutable memory — writing functional Python or Rust can come close. Pre emp scheduling — I assume this is to prevent deadlocks? This and everything else is very nice but do we need a full VM for that? I’m not trying to be a jerk but I want to understand why this isn’t a feature implementable as a runtime.
    - SirGiggles 5y ago
      > Pre emp scheduling Userspace pre-emptive schedulers do not incur the same level of overhead that kernel threads would normally have. To address another comment you made in another chain, Erlang accomplishes pre-emption through reductions which is roughly equivalent to ~4000 function calls, if I remember correctly and if I have read the correct documentation, before it yields the Erlang process. Other forms of async to my knowledge essentially queue a task while the main thread of execution continues before awaiting; and the queued task is run in some form of thread pool (or other forms of executors). Without going too deep into this subject (and assuming I have made no logical errors) it is possible to have misbehaving tasks in async tank performance and latency, while the same misbehaving process in Erlang wouldn't. See Saša Jurić's demonstration [1] for an example. Do keep in mind that async is not forte, what I have stated is merely my observations. I write primarily with kernel threads. This is why it is possible to spin up millions of, in Erlang terms, processes with minimal overhead. See this article by the Phoenix Framework team regarding an experiment on having two million active websocket connections on a single (albeit beefy) host [2]. > This and everything else is very nice but do we need a full VM for that? See [3] regarding an effort to port the Erlang process and supervision strategy to Rust. In my naive view, it is possible to write something similar to BEAM in another language, but the more troubling portion of the work is to jury-rig the original language into a concurrency model it isn't designed for whereas Erlang by contrast was created purely for this task. [1] https://www.youtube.com/watch?v=JvBT4XBdoUE https://www.youtube.com/watch?v=JvBT4XBdoUE [2] https://www.phoenixframework.org/blog/the-road-to-2-million-websocket-connections https://www.phoenixframework.org/blog/the-road-to-2-million-... [3] https://github.com/bastion-rs/bastion https://github.com/bastion-rs/bastion
      - jimsimmons 5y ago
        Appreciate the answer. I just saw the Sasa talk the other day incidentally and pre emption value is pretty clear now. It’s so clever and well thought out, I think I start too see value of Erlang now
- innocentoldguy 5y ago
  Private memory for all processes (Go uses public memory) The actor model. Superior memory usage and CPU processing. Supervision trees. …
  - innocentoldguy 5y ago
    Well, you can downvote me all you want, fuckers, but what I said is accurate.
  - jimsimmons 5y ago
    It’s pretty nifty. But I’m dumbfounded these features haven’t been copied over to other async runtimes. This is why I wonder if there’s a hidden catch that Erlang supporters don’t talk much about
- dnautics 5y ago
  You can implement those things, but then you'd be implementing them. Suppose you open a file (or a tcp socket, or a database transaction) while being connected over an http connection. How many LOC does it take to make sure that file is closed at the end of the http connection, through all possible error conditions? How confident are you that said code is correct? In elixir, it's 0 LOC.
  - jimsimmons 5y ago
    Nice, finally a straight answer. To be clear, this is because Erlang forces you to think about such failures and also because the entire stack being designed for such fault tolerance, correct?

mikeryan 5y ago

I can understand deciding Elixer is a good solution for for concurrency. What I couldn’t understand from the story was why it was needed. Unless the IDE had some sort of real-time shared coding? There’s nothing in the list of requirements that couldn’t be done over http it all seems asynchronous? Scale here seems like it would be small. Three months is almost zero time for a team of 4. I’m glad it all worked out but I have a harder time from the story understanding how using Elixer was the right choice vs a stack they were more familiar and confident on condieribg the constraints.

sodapopcan 5y ago
Immutability is its own reward :) Yes yes, available in other langs, but it kinda takes YAGNI in terms of scale off the table since it's a perfectly good fit for small applications as well. It just happens to scale (in terms of concurrency) really well. But if you never need to use those features that's ok—you'll still have an application written in a really nice, expressively terse language.

macintux 5y ago

> fault-tolerance is included as a requirement by default for any sufficiently complex distributed system, but has little to do with the languages/frameworks used to build that system Erlang/Elixir have an extremely compelling story when it comes to fault-tolerance. You can only argue that all languages are created equal in this regard if you fall back to the "everything is Turing-complete so they're equivalent" argument. Erlang and its VM were built from the ground up to work closely together with fault tolerance in mind, in a context where a few minutes of downtime meant severe financial penalties.

lostcolony 5y ago

That's not really true. Or rather, the need for the distributed system to be fault tolerant is, but downplaying the language/framework isn't entirely. Kubernetes has definitely made it so the distributed tooling of Erlang is less valuable; many of the things it provides you can also get with Kubernetes. The tradeoff being that Kubernetes is far more complex (but, also allows for things Erlang doesn't have). But what Kubernetes can't do is help you write correct programs. That is, if an error happens, what do you do? In most languages, you catch the ones you can predict and handle them. For the others, you let them trickle up until someone catches them, or the app crashes. The problem is figuring out where to handle them, and to make sure you really -are- handling the things you know how to handle. Erlang flips this. Handling things becomes far less important as isolating failure. If something goes wrong ~here~, where else is there state that is likely wrong, and how do we get it all back into a good state? This is a really powerful approach, as it ends up being far more robust, for far less work. It's harder to get 'incorrect' behavior this way; rather than have to enumerate the unhappy paths, and how to address each, you just have to declare dependencies between processes and how, should they find themselves on an unhappy path, to drop their state and get a known good one.