6 ms·

The main insight is in Joe Armstrong's thesis. "Making reliable distributed systems in the presence of software errors" https://erlang.org/download/armstrong_th

The main insight is in Joe Armstrong's thesis. "Making reliable distributed systems in the presence of software errors" https://erlang.org/download/armstrong_thesis_2003.pdf https://erlang.org/download/armstrong_thesis_2003.pdf (I think the original title was "In the presence of hardware errors", emphasis mine) All the other things flow from that thesis and understanding. You can recreate behaviours described in the repo doc using Erlang primitives very easily, and they are very hard to recreate in pretty much any other language. Because Erlang is very much literally about lightweight processes and message passing. Only: - every part of the system knows about lightweight processes and messages - every part of the system is engineered around them - the underlying VM is not only re-entrant in almost every single function, it's also extremely resilient, and can almost guarantee that when something dies, only that process dies, and the rest of the system receives a guaranteed notification that this process dies There are more, but without at least that you can't really recreate Erlang's standard behaviours. For example, you can't recreate this in Go or Rust because `panic` will kill your entire program unconditionally.

ThemalSpan 4y ago
I agree up to the last point. You can catch panics at the thread level right? I mean, more generally, what is Erlang implemented in?
- Arch-TK 4y ago
  Erlang is implemented on top of BEAM which has its own scheduler and schedules its own idea of lightweight processes.
  - dolmen 4y ago
    It looks like features that the Go runtime provides.
- ModernMech 4y ago
  Of course you could implement a language like Erlang in Rust, but I think the point is that you would have to do exactly that in order to do in Rust what Erlang does at the language level.
- Idiot_in_Vain 4y ago
  >> I mean, more generally, what is Erlang implemented in? Pure magic, that can't be recreated in any regular language :)
- dmitriid 4y ago
  > I mean, more generally, what is Erlang implemented in? "Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang." - Virding's Law ;) So first you have to implement all those things. And then use them.
- d3ckard 4y ago
  In C. You don’t operate on thread level directly, threads are just executors of processes managed by VM, usually the same amount as vCPU cores.
- octacat 4y ago
  You can in C/C++. Though, the point is that erlang has a strategy how to clean and recover from such error (some data can be lost, rarely we can crash the whole system for some errors). I mean, you can write erlang-way in almost any language, just in this case you need to adopt all the libraries to follow the same principles. Some languages implement similar error handling strategy by just creating a separate process per request (hello, php). We know how to clean after a worker dies (just let the worker die). Just in that case supervisor strategy is very simple.
ilyt 4y ago
> For example, you can't recreate this in Go or Rust because `panic` will kill your entire program unconditionally. You can recover() panic() just fine in Go.
- dmitriid 4y ago
  These are like sledgehammers when you need a screwdriver. It's very hard to build Erlang-like versions of supervision trees using those tools.
  - masklinn 4y ago
    > These are like sledgehammers when you need a screwdriver. Also by the very nature of Go your application state is as likely as not to be fucked, so even if you did handroll monitors and links, and thus could build supervision trees, they wouldn’t be of much use.
- ramchip 4y ago
  Yeah, OP took a shortcut here :) The nuance is that most languages let you build reliable programs _if your code is correct_ - if you're using defers, context handlers, finalizer, cleaning up state in shared data structure, etc. Erlang's goal is to be reliable "in the presence of software errors", that is, even if your code is buggy. If a request handler process dies inside an Erlang web app, whatever file or socket it opened, memory it allocated, shared resource it acquired (e.g. a DB connection) will be reclaimed. This is true without having to write any error handling in your code. The way it's done is that the VM handles the basic stuff like memory and files, and it provides signals that libraries (like a DB connection pool for instance) can use to know if a process fails and clean up as needed. In other words the process that fails is not responsible for its own clean up. At some point some code must of course be correct for this to work. Like, if the DB connection pool library doesn't monitor processes that borrow connections, it could leak the connection when such a process dies. But the point is that this part (the "error kernel") can be a small, well-tested part of the overall system; whereas in a classic program, the entire codebase has to handle errors correctly to guarantee overall stability.
- komadori 4y ago
wongarsu 4y ago
> For example, you can't recreate this in Go or Rust If you're willing to make your "lightweight processes" OS threads you could kind of make it work. E.g. Rust gives you both panic hooks (to notify everyone else that you died) and catch_unwind to contain the effect of a panic (which generally stops at a thread boundary anyways). But of course that only scales to a couple hundred or thousand threads, so you probably have to sacrifice a lot of granularity. And any library that links to C/C++ code has the potential to bring the whole process down (unless you make your "lightweight processes" just "OS processes", but that just makes the scaling problems worse)
- brightball 4y ago
  OS threads would defeat the purpose though. You can run millions of BEAM processes. Threads are a lot more expensive and still run the risk of taking over your system with an infinite loop.
  - toast0 4y ago
    OS threads are actually premptive, so an infinite loop is really less of a deal than one in Erlang. Erlang is really not premptive, it's just that function calls are automatically (potential) yield points, and you can't loop without function calls because of the construction of the language; if somehow you did manage to get a real loop into a process, that scheduler would be stuck forever, and you'd end up with a broken BEAM when you triggered something that requires coordination between schedulers --- I forget what sorts of things do that, but code purging can't finish without each scheduler doing housekeeping, etc. Meanwhile, your OS thread will just keep eating CPU, but everything else is fine. You're right about the number of threads though. It takes a lot more memory to run an OS thread than a BEAM process, and I'm not sure OS schedulers will manage that many threads even if the memory is there (but I could be wrong... getting things to work nicely at 50k threads may be sufficient for millions)
    - brightball 4y ago
      Unless code is exiting the BEAM itself, an infinite loop that bypasses yield points shouldn't be possible. Doing this at such a granular level is also one of the reasons that you can run a database that thousands of processes are accessing, within the same runtime, without a performance impact to the overall system.
neonsunset 4y ago
Most modern async runtimes let you do that. A (worker) thread dying isn't an issue in Rust, Go, C# and etc. Sure, each goes about error handling in a slightly different way either opt-in or opt-out or enforced (when unwrapping Error<T, E>) but other than that the advantages of Erlang/Elixir have faded over time because the industry has caught up. p.s.: C# has not one but two re-entrancy syntax options - 'async/await' and 'IEnumerable<T>/yield return'. You can use latter to conveniently implement state machines.
- gamache 4y ago
  Sometimes the advantage is not just the ability to do something -- I mean, by that logic, only bare-metal systems languages like Rust, C, Zig etc could have advantages over other languages. Erlang (and by extension Elixir) has the advantage that the programmer can think at a higher level about their system. You don't have to write or configure a scheduler. You don't have to invent supervision trees. You can be sure that the concurrently-running parts of your system cannot possibly affect each other's memory footprint (though Rust gives a robust answer to this problem as well). It doesn't make a perfect fit for every problem, but there is still a decent-sized space of problems -- I'd say "highly concurrent, but not highly parallel" -- where Erlang gives the programmer a headstart.
  - valenterry 4y ago
    The thing is, we mostly try to keep applications stateless, unless it's necessary to keep state for performance (think realtime onlinegames, hft, ...). And in those cases there is simply no need for e.g. supervision trees because there is nothing to restart. You still need stuff like retrying, but this is supported by all major concurrency libraries / effect systems. In fact, Erlang has fallen behind here in terms of what the language offers (not what the BEAM offers, the BEAM is still top notch imho) State is mostly moved to either the database or message queues or similar, which is pretty good.
    - OkayPhysicist 4y ago
      We try to keep applications stateless because state handling in most programming languages is pathological. Erlang presents a different solution to the underlying problem of state management, rather than the trying to solve the higher-level of "how best to support this one particular solution to the problem of state management". Namely: The message queues are part of the language. They're built right in, for ease of use. Your caching layer is built into the language, for ease of use and faster performance.
EdwardDiego 4y ago
18 or so years ago, I was a phone monkey for a government department who was initially learning to code for fun, but quickly realised programming could be a great way to earn a living. I read about Erlang on /r/programming, which was having a new fad for a new shiny language, as was the custom back then. And I desperately followed all the /r/programming fads because I was worried that I'd end up irrelevant if I wasn't skilled up in the latest Haskell web framework. But one of those fads was Erlang, and it intrigued me so much, that I ended up printing Mr Armstrong's thesis, stuck it in a manila folder, and read it, slowly, cover to cover while waiting for, or riding on, my bus to my contact centre job, and I've still got it in my bookcase. His thinking on resiliency in the face of inevitable failures, and on safe concurrency, has shaped my thinking, and proved invaluable repeatedly and is very relevant today. It's like their phone switches were distributed microservices running in a container orchestrator, before it was cool.
- gjvc 4y ago
  But one of those fads was Erlang, and it intrigued me so much, that I ended up printing Mr Armstrong's thesis, stuck it in a manila folder, and read it I am now inspired to do the same, thank you!

ThemalSpan 4y ago

I agree up to the last point. You can catch panics at the thread level right? I mean, more generally, what is Erlang implemented in?

Arch-TK 4y ago
Erlang is implemented on top of BEAM which has its own scheduler and schedules its own idea of lightweight processes.
- dolmen 4y ago
  It looks like features that the Go runtime provides.
ModernMech 4y ago
Of course you could implement a language like Erlang in Rust, but I think the point is that you would have to do exactly that in order to do in Rust what Erlang does at the language level.
Idiot_in_Vain 4y ago
>> I mean, more generally, what is Erlang implemented in? Pure magic, that can't be recreated in any regular language :)
dmitriid 4y ago
> I mean, more generally, what is Erlang implemented in? "Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang." - Virding's Law ;) So first you have to implement all those things. And then use them.
d3ckard 4y ago
In C. You don’t operate on thread level directly, threads are just executors of processes managed by VM, usually the same amount as vCPU cores.
octacat 4y ago
You can in C/C++. Though, the point is that erlang has a strategy how to clean and recover from such error (some data can be lost, rarely we can crash the whole system for some errors). I mean, you can write erlang-way in almost any language, just in this case you need to adopt all the libraries to follow the same principles. Some languages implement similar error handling strategy by just creating a separate process per request (hello, php). We know how to clean after a worker dies (just let the worker die). Just in that case supervisor strategy is very simple.

ilyt 4y ago

> For example, you can't recreate this in Go or Rust because `panic` will kill your entire program unconditionally. You can recover() panic() just fine in Go.

dmitriid 4y ago
These are like sledgehammers when you need a screwdriver. It's very hard to build Erlang-like versions of supervision trees using those tools.
- masklinn 4y ago
  > These are like sledgehammers when you need a screwdriver. Also by the very nature of Go your application state is as likely as not to be fucked, so even if you did handroll monitors and links, and thus could build supervision trees, they wouldn’t be of much use.
ramchip 4y ago
Yeah, OP took a shortcut here :) The nuance is that most languages let you build reliable programs _if your code is correct_ - if you're using defers, context handlers, finalizer, cleaning up state in shared data structure, etc. Erlang's goal is to be reliable "in the presence of software errors", that is, even if your code is buggy. If a request handler process dies inside an Erlang web app, whatever file or socket it opened, memory it allocated, shared resource it acquired (e.g. a DB connection) will be reclaimed. This is true without having to write any error handling in your code. The way it's done is that the VM handles the basic stuff like memory and files, and it provides signals that libraries (like a DB connection pool for instance) can use to know if a process fails and clean up as needed. In other words the process that fails is not responsible for its own clean up. At some point some code must of course be correct for this to work. Like, if the DB connection pool library doesn't monitor processes that borrow connections, it could leak the connection when such a process dies. But the point is that this part (the "error kernel") can be a small, well-tested part of the overall system; whereas in a classic program, the entire codebase has to handle errors correctly to guarantee overall stability.
komadori 4y ago

wongarsu 4y ago

> For example, you can't recreate this in Go or Rust If you're willing to make your "lightweight processes" OS threads you could kind of make it work. E.g. Rust gives you both panic hooks (to notify everyone else that you died) and catch_unwind to contain the effect of a panic (which generally stops at a thread boundary anyways). But of course that only scales to a couple hundred or thousand threads, so you probably have to sacrifice a lot of granularity. And any library that links to C/C++ code has the potential to bring the whole process down (unless you make your "lightweight processes" just "OS processes", but that just makes the scaling problems worse)

brightball 4y ago
OS threads would defeat the purpose though. You can run millions of BEAM processes. Threads are a lot more expensive and still run the risk of taking over your system with an infinite loop.
- toast0 4y ago
  OS threads are actually premptive, so an infinite loop is really less of a deal than one in Erlang. Erlang is really not premptive, it's just that function calls are automatically (potential) yield points, and you can't loop without function calls because of the construction of the language; if somehow you did manage to get a real loop into a process, that scheduler would be stuck forever, and you'd end up with a broken BEAM when you triggered something that requires coordination between schedulers --- I forget what sorts of things do that, but code purging can't finish without each scheduler doing housekeeping, etc. Meanwhile, your OS thread will just keep eating CPU, but everything else is fine. You're right about the number of threads though. It takes a lot more memory to run an OS thread than a BEAM process, and I'm not sure OS schedulers will manage that many threads even if the memory is there (but I could be wrong... getting things to work nicely at 50k threads may be sufficient for millions)
  - brightball 4y ago
    Unless code is exiting the BEAM itself, an infinite loop that bypasses yield points shouldn't be possible. Doing this at such a granular level is also one of the reasons that you can run a database that thousands of processes are accessing, within the same runtime, without a performance impact to the overall system.

neonsunset 4y ago

Most modern async runtimes let you do that. A (worker) thread dying isn't an issue in Rust, Go, C# and etc. Sure, each goes about error handling in a slightly different way either opt-in or opt-out or enforced (when unwrapping Error<T, E>) but other than that the advantages of Erlang/Elixir have faded over time because the industry has caught up. p.s.: C# has not one but two re-entrancy syntax options - 'async/await' and 'IEnumerable<T>/yield return'. You can use latter to conveniently implement state machines.

gamache 4y ago
Sometimes the advantage is not just the ability to do something -- I mean, by that logic, only bare-metal systems languages like Rust, C, Zig etc could have advantages over other languages. Erlang (and by extension Elixir) has the advantage that the programmer can think at a higher level about their system. You don't have to write or configure a scheduler. You don't have to invent supervision trees. You can be sure that the concurrently-running parts of your system cannot possibly affect each other's memory footprint (though Rust gives a robust answer to this problem as well). It doesn't make a perfect fit for every problem, but there is still a decent-sized space of problems -- I'd say "highly concurrent, but not highly parallel" -- where Erlang gives the programmer a headstart.
- valenterry 4y ago
  The thing is, we mostly try to keep applications stateless, unless it's necessary to keep state for performance (think realtime onlinegames, hft, ...). And in those cases there is simply no need for e.g. supervision trees because there is nothing to restart. You still need stuff like retrying, but this is supported by all major concurrency libraries / effect systems. In fact, Erlang has fallen behind here in terms of what the language offers (not what the BEAM offers, the BEAM is still top notch imho) State is mostly moved to either the database or message queues or similar, which is pretty good.
  - OkayPhysicist 4y ago
    We try to keep applications stateless because state handling in most programming languages is pathological. Erlang presents a different solution to the underlying problem of state management, rather than the trying to solve the higher-level of "how best to support this one particular solution to the problem of state management". Namely: The message queues are part of the language. They're built right in, for ease of use. Your caching layer is built into the language, for ease of use and faster performance.

EdwardDiego 4y ago

18 or so years ago, I was a phone monkey for a government department who was initially learning to code for fun, but quickly realised programming could be a great way to earn a living. I read about Erlang on /r/programming, which was having a new fad for a new shiny language, as was the custom back then. And I desperately followed all the /r/programming fads because I was worried that I'd end up irrelevant if I wasn't skilled up in the latest Haskell web framework. But one of those fads was Erlang, and it intrigued me so much, that I ended up printing Mr Armstrong's thesis, stuck it in a manila folder, and read it, slowly, cover to cover while waiting for, or riding on, my bus to my contact centre job, and I've still got it in my bookcase. His thinking on resiliency in the face of inevitable failures, and on safe concurrency, has shaped my thinking, and proved invaluable repeatedly and is very relevant today. It's like their phone switches were distributed microservices running in a container orchestrator, before it was cool.

gjvc 4y ago
But one of those fads was Erlang, and it intrigued me so much, that I ended up printing Mr Armstrong's thesis, stuck it in a manila folder, and read it I am now inspired to do the same, thank you!