5 ms·

The BEAM might be the most under-rated tech in all of open source. Example -- whatsapp. Why Elixir + Erlang are not more popular for high concurrency projects

The BEAM might be the most under-rated tech in all of open source. Example -- whatsapp. Why Elixir + Erlang are not more popular for high concurrency projects is a mystery to me.

honkycat 1y ago
everyone is trying to hire the best, but caters to the lowest common denominator. "But what if we hire an engineer to stupid to learn a new language in a few weeks???" and on and on and on
paradox460 1y ago
I believe it's purely marketing Other languages (Java, C#, Go) are supported by massive corporate backers, who have vested interest in seeing them succeed. Erlang's own corporate owner tried to strangle it in the crib, and even since then they've been standoffishness towards anything apart from technical resources. We didn't really see much marketing like material arise until things like Elixir came about, and even that is more following the ruby model, which is very dev oriented. And the world to which Elixir emerged in 2014 is very different than the world rails sprung into in 2004. Devs can usually be convinced about how good BEAMlangs are, but without the glossy, you have a harder time convincing the MBA set
- bmitc 1y ago
  Erlang wasn't open sourced until the late 90s. Erlang was never really marketed. Elixir arose fairly quickly. Erlang and Elixir are nice because concurrency isn't tacked onto the language like those other languages.
sargun 1y ago
Investment gap is what I'd say too. While Rust, Go, Python, etc... have had massive backers that have managed to invest a ton more into things like static analysis, type checking, and developer ergonomics, the Erlang ecosystem hasn't necessarily had the same love, and instead the major users have typically chosen to pivot, or build something outside of the BEAM.
bsder 1y ago
> Why Elixir + Erlang are not more popular for high concurrency projects is a mystery to me. 1) Because the amount of "high concurrency" you can handle with a single machine with "standard" languages keeps moving up every year. 2) Because you had a single, nigh undocumented, very impenetrable to port implementation of BEAM for the longest time. 3) Because "Erlang" isn't the magic. OTP is the magic. And that requires that you shift your mindset dramatically. And you can do the OTP abstractions in other languages! 4) I think Scala sucked up a lot of Erlang's oxygen for a long time. Erlang is helping with that.
- d3ckard 1y ago
  1. Fair, but at the same time - OTP is the only sane concurrency framework. To the point I still struggle to comprehend why it wasn't copied. 2. Fair. 3. I am not aware of a single language doing OTP abstractions, especially implementing preemptive scheduler for green threads. 4. Possibly, but I haven't seen a single instance of a company choosing between the two. I think marketing is a way better explanation to be honest. Jose lives in Poland, was very active and a lot of Ruby shops moved to Elixir.
  - lamuswawir 1y ago
    There have been multiple attempts in various languages to implement Erlang-like processes and claims of parity, Erlang processes remain unbeaten at what they do.
  - foldr 1y ago
    Go has a preemptive scheduler for green threads.
    - d3ckard 1y ago
      Oh, indeed, since 1.14. Nice to see them moving this direction.
- agos 1y ago
  OTP is the magic but the ability of the Beam to handle that many processes gracefully is part of the magic
schonfinkel 1y ago
And now with Gleam moving past version 1.0 last year, the BEAM also boasts a decent amount of "flavors" of functional programming. You can go with Erlang/OTP (the alien tech that has been battle tested for almost 30 years), Elixir for it's great webdev ecosystem, Gleam if you like an ML-like programming language on the BEAM. LFE is also another great option if need a BEAM-native LISP.
- agos 1y ago
  How I wish that Gleam had OTP! Elixir is also getting some tooling love in the form of a new language server and some very much appreciated type checking
  - dimator 1y ago
    Forgive my ignorance, I was looking at this the other day, isn't this otp for gleam? https://github.com/gleam-lang/otp https://github.com/gleam-lang/otp
    - hazn 1y ago
      yes but i believe it is still in the ”beta” stage, as gleam is implementing the OTP by itself and not wrapping the erlang implementation. you have to import the otp package in gleam to use the famous actor architecture
d3ckard 1y ago
At the same time, the least secret "secret sauce" out there. BBC moved to Elixir (quite recent news), so I guess traction is there.
franticgecko3 1y ago
>Why Elixir + Erlang are not more popular for high concurrency projects is a mystery to me. I work at an Erlang shop. For Erlang to be useful you need to have massive scale, millions of DAU. Yes Elixir might run your website serving thousands of DAU but Erlang and the BEAM was not invented for you. Few companies have the scale that makes Erlang a reasonable choice. More pressing these days I believe is that the Erlang ecosystem is all or nothing, the BEAM is like its own little operating system and has extremely (IMHO) complicated setup and configuration with many moving parts: ERTS, epmd, Rebar,... You don't need or shouldn't use container tech and k8s with Erlang because it's already doing that stuff in its own Erlang way - but it's the Erlang way it's extremely unique and unfamiliar to most. When you have the right use case for Erlang and you see it in action it is truly like black magic, it's been a highlight of my career to work with it.
- borromakot 1y ago
  Elixir has been quite easy to set up, configure and deploy, at least for the last 5-6 years. I can't speak for Erlang though.
  - michaelcampbell 1y ago
    > Elixir has been quite easy to set up, configure and deploy, at least for the last 5-6 years. One of your sibling commenters noted... > I've basically found it [Elixir] the only language more difficult than python to set up an environment for. Quite the difference. I'm not saying either one of you is wrong; I'm sure you both experienced what you say you did, it's just interesting to see the dichotomy, literally (as of this writing) next to each other.
- throwawaymaths 1y ago
  elixir softens a whole ton of the sharp edges around getting started and there are only a handful of "gotchas" coming from other hlls (like all terms being immutable when you pass to a function -- but thats honestly almost a problem with the "mainstream" languages).
  - 1y ago
ludicity 1y ago
We swapped our Squarespace website out for a Phoenix application yesterday and it was such a delightful experience.
Thaxll 1y ago
Because it's a single stack that leaks too many things in my opinion, it's not just code but it's also handling infra, process restart, cache, queue, db ect ... And it only works with Erlang, so it's an all or nothing setup which imo is very restrictive. Kubernetes does a lot of things that Beam does but better and completely language neutral. Then you have builtin queues and db in Erlang but they are inferior to industry standard and again only work with Erlang.
- zbentley 1y ago
  Honestly, that mixing of concerns is the biggest benefit, at least for me. While Erlang clearly lost the popularity war, using it (well, Elixir in my case) feels like going to an alternate timeline where it wasn’t a given that a good application used separately developed tools for frontend, backend, database, cache, service discovery, load balancing, containerization, orchestration, and so on. We really could have been on a platform that incorporated all of those things into a single runtime. The way I explain Erlang to folks is: “it’s not a competitor to Python running Django in Gunicorn; it’s a competitor for Python+Django+Gunicorn in Docker on Kubernetes on Linux, talking via a service mesh and storing data in disk-persisted Redis.” In many ways, swapping out only the top of that stack with Erlang is going to leave you worse off than you were before. Sadly, the Erlang renaissance was years late to the boom of containerization+orchestration+distributed systems, so we’ll never know how good (or bad) things could have been. Ah well, c’est la vie.

The BEAM might be the most under-rated tech in all of open source. Example -- whatsapp. Why Elixir + Erlang are not more popular for high concurrency projects is a mystery to me.

honkycat 1y ago

everyone is trying to hire the best, but caters to the lowest common denominator. "But what if we hire an engineer to stupid to learn a new language in a few weeks???" and on and on and on

paradox460 1y ago

I believe it's purely marketing Other languages (Java, C#, Go) are supported by massive corporate backers, who have vested interest in seeing them succeed. Erlang's own corporate owner tried to strangle it in the crib, and even since then they've been standoffishness towards anything apart from technical resources. We didn't really see much marketing like material arise until things like Elixir came about, and even that is more following the ruby model, which is very dev oriented. And the world to which Elixir emerged in 2014 is very different than the world rails sprung into in 2004. Devs can usually be convinced about how good BEAMlangs are, but without the glossy, you have a harder time convincing the MBA set

bmitc 1y ago
Erlang wasn't open sourced until the late 90s. Erlang was never really marketed. Elixir arose fairly quickly. Erlang and Elixir are nice because concurrency isn't tacked onto the language like those other languages.

sargun 1y ago

Investment gap is what I'd say too. While Rust, Go, Python, etc... have had massive backers that have managed to invest a ton more into things like static analysis, type checking, and developer ergonomics, the Erlang ecosystem hasn't necessarily had the same love, and instead the major users have typically chosen to pivot, or build something outside of the BEAM.

bsder 1y ago

> Why Elixir + Erlang are not more popular for high concurrency projects is a mystery to me. 1) Because the amount of "high concurrency" you can handle with a single machine with "standard" languages keeps moving up every year. 2) Because you had a single, nigh undocumented, very impenetrable to port implementation of BEAM for the longest time. 3) Because "Erlang" isn't the magic. OTP is the magic. And that requires that you shift your mindset dramatically. And you can do the OTP abstractions in other languages! 4) I think Scala sucked up a lot of Erlang's oxygen for a long time. Erlang is helping with that.

d3ckard 1y ago
1. Fair, but at the same time - OTP is the only sane concurrency framework. To the point I still struggle to comprehend why it wasn't copied. 2. Fair. 3. I am not aware of a single language doing OTP abstractions, especially implementing preemptive scheduler for green threads. 4. Possibly, but I haven't seen a single instance of a company choosing between the two. I think marketing is a way better explanation to be honest. Jose lives in Poland, was very active and a lot of Ruby shops moved to Elixir.
- lamuswawir 1y ago
  There have been multiple attempts in various languages to implement Erlang-like processes and claims of parity, Erlang processes remain unbeaten at what they do.
- foldr 1y ago
  Go has a preemptive scheduler for green threads.
  - d3ckard 1y ago
    Oh, indeed, since 1.14. Nice to see them moving this direction.
agos 1y ago
OTP is the magic but the ability of the Beam to handle that many processes gracefully is part of the magic

schonfinkel 1y ago

And now with Gleam moving past version 1.0 last year, the BEAM also boasts a decent amount of "flavors" of functional programming. You can go with Erlang/OTP (the alien tech that has been battle tested for almost 30 years), Elixir for it's great webdev ecosystem, Gleam if you like an ML-like programming language on the BEAM. LFE is also another great option if need a BEAM-native LISP.

agos 1y ago
How I wish that Gleam had OTP! Elixir is also getting some tooling love in the form of a new language server and some very much appreciated type checking
- dimator 1y ago
  Forgive my ignorance, I was looking at this the other day, isn't this otp for gleam? https://github.com/gleam-lang/otp https://github.com/gleam-lang/otp
  - hazn 1y ago
    yes but i believe it is still in the ”beta” stage, as gleam is implementing the OTP by itself and not wrapping the erlang implementation. you have to import the otp package in gleam to use the famous actor architecture

d3ckard 1y ago

At the same time, the least secret "secret sauce" out there. BBC moved to Elixir (quite recent news), so I guess traction is there.

franticgecko3 1y ago

>Why Elixir + Erlang are not more popular for high concurrency projects is a mystery to me. I work at an Erlang shop. For Erlang to be useful you need to have massive scale, millions of DAU. Yes Elixir might run your website serving thousands of DAU but Erlang and the BEAM was not invented for you. Few companies have the scale that makes Erlang a reasonable choice. More pressing these days I believe is that the Erlang ecosystem is all or nothing, the BEAM is like its own little operating system and has extremely (IMHO) complicated setup and configuration with many moving parts: ERTS, epmd, Rebar,... You don't need or shouldn't use container tech and k8s with Erlang because it's already doing that stuff in its own Erlang way - but it's the Erlang way it's extremely unique and unfamiliar to most. When you have the right use case for Erlang and you see it in action it is truly like black magic, it's been a highlight of my career to work with it.

borromakot 1y ago
Elixir has been quite easy to set up, configure and deploy, at least for the last 5-6 years. I can't speak for Erlang though.
- michaelcampbell 1y ago
  > Elixir has been quite easy to set up, configure and deploy, at least for the last 5-6 years. One of your sibling commenters noted... > I've basically found it [Elixir] the only language more difficult than python to set up an environment for. Quite the difference. I'm not saying either one of you is wrong; I'm sure you both experienced what you say you did, it's just interesting to see the dichotomy, literally (as of this writing) next to each other.
throwawaymaths 1y ago
elixir softens a whole ton of the sharp edges around getting started and there are only a handful of "gotchas" coming from other hlls (like all terms being immutable when you pass to a function -- but thats honestly almost a problem with the "mainstream" languages).
- 1y ago

ludicity 1y ago

We swapped our Squarespace website out for a Phoenix application yesterday and it was such a delightful experience.

Thaxll 1y ago

Because it's a single stack that leaks too many things in my opinion, it's not just code but it's also handling infra, process restart, cache, queue, db ect ... And it only works with Erlang, so it's an all or nothing setup which imo is very restrictive. Kubernetes does a lot of things that Beam does but better and completely language neutral. Then you have builtin queues and db in Erlang but they are inferior to industry standard and again only work with Erlang.

zbentley 1y ago
Honestly, that mixing of concerns is the biggest benefit, at least for me. While Erlang clearly lost the popularity war, using it (well, Elixir in my case) feels like going to an alternate timeline where it wasn’t a given that a good application used separately developed tools for frontend, backend, database, cache, service discovery, load balancing, containerization, orchestration, and so on. We really could have been on a platform that incorporated all of those things into a single runtime. The way I explain Erlang to folks is: “it’s not a competitor to Python running Django in Gunicorn; it’s a competitor for Python+Django+Gunicorn in Docker on Kubernetes on Linux, talking via a service mesh and storing data in disk-persisted Redis.” In many ways, swapping out only the top of that stack with Erlang is going to leave you worse off than you were before. Sadly, the Erlang renaissance was years late to the boom of containerization+orchestration+distributed systems, so we’ll never know how good (or bad) things could have been. Ah well, c’est la vie.