11 ms·

This writeup make me even more convinced of Elixir becoming one of the large players when it comes to hugely scaling applications. If there is one thing I trul

This writeup make me even more convinced of Elixir becoming one of the large players when it comes to hugely scaling applications. If there is one thing I truly love about Elixir, it is the easiness of getting started, while standing on the shoulders of a giant that is the Erlang VM. You can start by building a simple, not very demanding application with it, yet once you hit a large scale, there is plenty of battle-proven tools to save you massive headaches and costly rewrites. Still, I feel, that using Elixir is, today, still a large bet. You need to convince your colleagues as much as your bosses / customers to take the risk. But you can rest assured it will not fail you as you need to push it to the next level. Nothing comes for free, and at the right scale, even the Erlang VM is not a silver bullet and will require your engineering team to invest their talent, time and effort to fine tune it. Yet, once you dig deep enough into it, you'll find plenty of ways to solve your problem at a lower cost as compared to other solutions. I see a bright future for Elixir, and a breath of fresh air for Erlang. It's such a great time to be alive!

toolz 9y ago
I think the next big push in elixir is tooling around metrics. Once there are hard numbers to market to non-techies I think we'll see a large shift towards elixir.
- rubyn00bie 9y ago
  Just look for Erlang tools, there are loads available. The only problem with Elixir right now is sometimes people forget to look at Erlang for already solved problems/tooling/etc. Erlang/BEAM has been alive and well in some very serious mission critical applications for 20 years. Just look at how telecoms used it (its origin).
  - _asummers 9y ago
    We use Elixir at work, and Erlang tools have a bit of a discovery problem. You can peek at download counts on Hex, if the author uses it, but that doesn't take account CI systems and such that inflate DL counts, or GitHub only repos that are not on Hex (see things like e.g. shell history prior to OTP20). That said, when there is no Elixir tool to do what I want, the very next thing I do is look for an Erlang tool, and there is often one that gets me most of the way to where I need to go. Wrapper libs are nice sugar, but they're almost always unnecessary when you can just call Erlang directly (String v charlist conversions notwithstanding!).
    - fbernier 9y ago
      Download count is a weak metric for evaluating the quality of a library. I don't think it's a good idea to skip the few minutes it takes to read parts of the documentation and critical parts of the code.
      - _asummers 9y ago
        I agree completely. I never said that it should be used for evaluation or that you shouldn't read the code or docs. But then the question is... how do you find high quality libraries without something like download count? The issue is signaling good libraries, with poor discovery. I subscribe to many "libraries worth looking at" mailing lists, the awesome lists, etc. but that is still a pretty coarse net. Searching for libraries on GitHub if you don't know the name already or don't really know what you're specifically looking for is an even worse experience. Combine that with relatively poor fuzzy search on Hex, where even if you DO know what you're looking for, it leads to a very subpar experience. For example, "e-commerce" leads to different results than "commerce". [0][1] These then exclude [2], which is an ecommerce library available on Hex without that keyword. [0] https://hex.pm/packages?_utf8=%E2%9C%93&search=ecommerce&sort=downloads https://hex.pm/packages?_utf8=%E2%9C%93&search=ecommerce&sor... [1] https://hex.pm/packages?search=commerce&sort=downloads https://hex.pm/packages?search=commerce&sort=downloads [2] https://hex.pm/packages/ryal_core https://hex.pm/packages/ryal_core
        ericmj 9y ago
        Search on hex.pm is not great and can definitely be improved, but we are currently at the limit of postgres full text search afaik, unless there are some magic things to tweak. This means we need to switch to other technologies to improve the search which is a bigger task. But regardless of how good search is it wouldn't find ryal_core when searching for "commerce" if that word or any similar is nowhere in the metadata. The description for the package ryal_core is "The core of Ryal." which doesn't do the search any favours.
        _asummers 9y ago
  - ramchip 9y ago
    I don't think that's the problem for metrics. Exometer is the go-to Erlang library, but its dependencies are a nightmare (e.g. https://github.com/Feuerlabs/exometer/issues/154 https://github.com/Feuerlabs/exometer/issues/154 or https://github.com/Feuerlabs/exometer#dependency-management https://github.com/Feuerlabs/exometer#dependency-management), it's essentially unmaintained, and integrating it with Ecto or Phoenix is roll-your-own. I think there is very much a need for a more modern library that plays nicely with Hex.
    - _asummers 9y ago
      We've had success with Elixometer, though the dependency pinning was a pain. All but two are fully on Hex now. https://gist.github.com/asummers/3479eb6e3914d9b9a24c9566468453ca https://gist.github.com/asummers/3479eb6e3914d9b9a24c9566468...
      - ramchip 9y ago
        Thanks, I might have to use this soon :)
    - Aqua_Geek 9y ago
      > I think there is very much a need for a more modern library [...] Building a solid standalone monitoring/metrics package is apparently next up on the "todo list" for Chris McCord (creator of Phoenix): https://www.youtube.com/watch?v=pfFpIjFOL-I https://www.youtube.com/watch?v=pfFpIjFOL-I
- manukall 9y ago
  I've recently launched https://pryin.io https://pryin.io, an application performance monitoring tool made for Elixir and Phoenix. It hooks into Phoenix and gives you insights into how long your request / channels take and what Ecto queries are run / how long those take. You can also manually augment pretty much anything else (background jobs, API calls, ...). Plus it keeps track of some important BEAM metrics like memory consumption.
  - gregpardo 9y ago
    Looks cool. Might want to have a demo site running for people to poke around.
bsder 9y ago
> I see a bright future for Elixir, and a breath of fresh air for Erlang. Is Erlang really that much of a barrier? Erlang was a touch odd, but I didn't find the language itself that mind-bending. Wrapping my head around the proper way to structure things and the proper use of the OTP libraries was much more time-consuming. That level of architectural thought doesn't magically go away because you changed the language. About the only thing I found irritating was a lack of language support for mutable hash tables, but I thought they fixed that at some point. And Erlang's bit packing/unpacking borders on magical in terms of expressiveness and speed. I wish other languages would adopt it.
- vertex-four 9y ago
  Unfortunately, a lot of people appear to hate Erlang's syntax to the point that it's a barrier to adoption - or Elixir never would've picked up at all. I don't understand it myself, but there we have it.
  - fbernier 9y ago
    I've seen this argument about the syntax countless times and during my first looks at Erlang code I couldn't understand much of it, but after sitting down and deciding to learn it it's an extremely simple syntax. The hard/long thing to fully understand is truly OTP.
- jmcgough 9y ago
  It became open source in the 90s. It's been used by several high-traffic sites/apps to support chat clients (Riot Games, WhatsApp, and I think Fetlife), but outside of that has seen very little adoption. I think it's a lot harder to sell people on a language that looks syntactically unintelligible to a lot of web developers. Most people I know working with Elixir now are/were Rubyists, so it's easy for them to make the leap from that and start to understand all of the awesome things they get from BEAM and OTP.
  - jhgg 9y ago
    Elixir/Erlang is great, but in addition to knowing the language, there is a huge library of tools within OTP/ERTS that you need to understand to effectively use it and skills learned from other languages don't map too well to Erlang/Elixir as they would to other languages (like going from Python to Go for example). In addition to learning OTP, you also have to learn how a functional language works if you're not familiar. A lot of things to learn to adopt Elixir/Erlang, but it's well worth it! I just don't think there's much reason for people to make that leap as there are much more familiar languages that can be used to solve the problems most developers are trying to solve.
janekm 9y ago
I read it a little differently. The whole article is about how Erlang/Elixir fails at its core reason for existence (fast message passing between distributed processes) and all the complicated work-arounds they had to implement to avoid actually using this core feature of Erlang.
- mononcqc 9y ago
  I have never seen a language or framework used at a large scale that did not require digging, understanding, and tweaking of the underlying machinery for its specific use case. The true failure of the platform happens when you cannot do these tweaks and adaptation, or that their cost is shadowed by having written it in a more appropriate technology at a lower time/cost/effort budget.
  - janekm 9y ago
    That's a good point. The article is certainly a good summary of what's needed to make Erlang/Elixir scale and a reminder that there are no "magic bullets".
- quaunaut 9y ago
  I don't think that's what Erlang was made for at all? Erlang was never about speed, it's about reliability, availability, and ease of concurrency. And Elixir just makes it easier to access those tools.
  - janekm 9y ago
    It may well be a misconception on my part - I assumed that as such message passing is such a core feature of using Erlang for distributed systems one should be able to treat it as a low-cost operation.
    - quaunaut 9y ago
      You can generally, just Discord is getting to a level of scale where the issue has more to do with architecture than language. All of their solutions in this post were Elixir-based solutions, and have very clean, easy APIs(and so far not very much code- all those libraries that solved their problems came out to ~400 LOC). I guess the message here is, "There's never a magic bullet, but you've got everything you need to make yours."
deleted 9y ago
[deleted]
tmat 9y ago
We're an Elixir shop and I wouldn't change it for the world. We're betting on it heavily.
censhin 9y ago
The challenge is going to be adoption, plain and simple. I worked two years on an Erlang project, and most of our frustration came from the lack of community to lean on when we needed help understanding best practices. Also, when you find only one library to solve your problem that has been abandoned for at least two years that is never fun. Few projects actually need the level of scalability that Erlang brings, and the cost of building and running an Erlang project as an enterprise application is non-trivial. People need to understand this if they want Elixir to succeed where Erlang hasn't.
- jlouis 9y ago
  The key benefit of Erlang isn't the scalability in my experience. It is resilience. Once your system is deployed, it tend to run with 0 fatalities for months on end. We have systems where we've had 0 maintenance days on them for months and even years. In turn, you can focus your development effort on new systems, new features, and scaling existing infrastructure since you are not bogged down in maintenance mode. Even better, you can often postpone errors in a system which are non-fatal. I often wait until I have a couple of bugs in a system and then I'll work on that system for a few days fixing them all. In systems where even the smallest amount of brittleness destroys them, this isn't really an option. Another experience of mine is that Erlang systems tend to take a bit longer to write compared to cranking out a fast solution in another language. But the payoff is that your Erlang solution tend to have better robustness as load increases and people start using the system. The community is small however, so you'll need to write more code yourself in-house as a result. For some problem spaces, this will hurt a lot because there are off-the-shelf solutions in other languages. On the other hand, larger projects which are more specialized can benefit from you knowing your own code better.
  - censhin 9y ago
    I agree with what you said. What I was really trying to get across was the need for a real community and wider adoption.
  - lostcolony 9y ago
    For anyone reading this, I worked in a shop where we built an Erlang system, choosing the language specifically for the high uptimes we needed. We nailed it so well that IT proactively reboots the system just because they don't trust that it's still working properly when it runs without issue for months on end. Done right, Erlang can be insanely resilient.
corobo 9y ago
It convinces me that with a selection of good devs you can hugely scale anything.
cooervo 9y ago
Truly the strenght of elixir comes for Erlang. I wonder if Scala is also as good as erlang for creating robust and fail proof distributed systems?
- biokoda 9y ago
  The point of Erlang is that it makes writing robust distributed systems easy and fast. The language itself is tiny. Scala on the other hand is not remotely small and easy to understand.
- justinhj 9y ago
  Scala paired with Akka is definitely good for that, I don't know how it compares performance wise with Elixir, would be interested to know. We're building a game server backend with lots of messaging in Akka.
  - tormeh 9y ago
    Scala/Akka is a lot faster than Erlang, but it also lacks per-actor GC, which means that latency is going to be higher in a Scala/Akka system. Also, the per-actor heaps in Beam means there's no risk of two Erlang actors having a reference to the same memory, whereas that's easy to do in Scala/Akka.
    - OvermindDL1 9y ago
      Actually you'd be surprised at just how fast the BEAM passes messages, especially as it also includes linking, monitoring, cross-process, cross-system, among other features. When creating a multi-thread atomic messaging system in C++ long ago I was able to out-perform the BEAM (though I'd not rank it as significantly out-perform), but once I added failure handling features among others then it never got close to BEAM's speed again in raw message passing. However, the BEAM is not very fast on executing actual code, it is like Python in that way where it is good to slave out CPU-heavy work; the BEAM is built for async IO, and yes, running a JVM (or C++) system as a node on a BEAM mesh is fantastic for that (or a port, or NIF for small work).
- 9y ago

toolz 9y ago

I think the next big push in elixir is tooling around metrics. Once there are hard numbers to market to non-techies I think we'll see a large shift towards elixir.

rubyn00bie 9y ago
Just look for Erlang tools, there are loads available. The only problem with Elixir right now is sometimes people forget to look at Erlang for already solved problems/tooling/etc. Erlang/BEAM has been alive and well in some very serious mission critical applications for 20 years. Just look at how telecoms used it (its origin).
- _asummers 9y ago
  We use Elixir at work, and Erlang tools have a bit of a discovery problem. You can peek at download counts on Hex, if the author uses it, but that doesn't take account CI systems and such that inflate DL counts, or GitHub only repos that are not on Hex (see things like e.g. shell history prior to OTP20). That said, when there is no Elixir tool to do what I want, the very next thing I do is look for an Erlang tool, and there is often one that gets me most of the way to where I need to go. Wrapper libs are nice sugar, but they're almost always unnecessary when you can just call Erlang directly (String v charlist conversions notwithstanding!).
  - fbernier 9y ago
    Download count is a weak metric for evaluating the quality of a library. I don't think it's a good idea to skip the few minutes it takes to read parts of the documentation and critical parts of the code.
    - _asummers 9y ago
      I agree completely. I never said that it should be used for evaluation or that you shouldn't read the code or docs. But then the question is... how do you find high quality libraries without something like download count? The issue is signaling good libraries, with poor discovery. I subscribe to many "libraries worth looking at" mailing lists, the awesome lists, etc. but that is still a pretty coarse net. Searching for libraries on GitHub if you don't know the name already or don't really know what you're specifically looking for is an even worse experience. Combine that with relatively poor fuzzy search on Hex, where even if you DO know what you're looking for, it leads to a very subpar experience. For example, "e-commerce" leads to different results than "commerce". [0][1] These then exclude [2], which is an ecommerce library available on Hex without that keyword. [0] https://hex.pm/packages?_utf8=%E2%9C%93&search=ecommerce&sort=downloads https://hex.pm/packages?_utf8=%E2%9C%93&search=ecommerce&sor... [1] https://hex.pm/packages?search=commerce&sort=downloads https://hex.pm/packages?search=commerce&sort=downloads [2] https://hex.pm/packages/ryal_core https://hex.pm/packages/ryal_core
      - ericmj 9y ago
        Search on hex.pm is not great and can definitely be improved, but we are currently at the limit of postgres full text search afaik, unless there are some magic things to tweak. This means we need to switch to other technologies to improve the search which is a bigger task. But regardless of how good search is it wouldn't find ryal_core when searching for "commerce" if that word or any similar is nowhere in the metadata. The description for the package ryal_core is "The core of Ryal." which doesn't do the search any favours.
        _asummers 9y ago
- ramchip 9y ago
  I don't think that's the problem for metrics. Exometer is the go-to Erlang library, but its dependencies are a nightmare (e.g. https://github.com/Feuerlabs/exometer/issues/154 https://github.com/Feuerlabs/exometer/issues/154 or https://github.com/Feuerlabs/exometer#dependency-management https://github.com/Feuerlabs/exometer#dependency-management), it's essentially unmaintained, and integrating it with Ecto or Phoenix is roll-your-own. I think there is very much a need for a more modern library that plays nicely with Hex.
  - _asummers 9y ago
    We've had success with Elixometer, though the dependency pinning was a pain. All but two are fully on Hex now. https://gist.github.com/asummers/3479eb6e3914d9b9a24c9566468453ca https://gist.github.com/asummers/3479eb6e3914d9b9a24c9566468...
    - ramchip 9y ago
      Thanks, I might have to use this soon :)
  - Aqua_Geek 9y ago
    > I think there is very much a need for a more modern library [...] Building a solid standalone monitoring/metrics package is apparently next up on the "todo list" for Chris McCord (creator of Phoenix): https://www.youtube.com/watch?v=pfFpIjFOL-I https://www.youtube.com/watch?v=pfFpIjFOL-I
manukall 9y ago
I've recently launched https://pryin.io https://pryin.io, an application performance monitoring tool made for Elixir and Phoenix. It hooks into Phoenix and gives you insights into how long your request / channels take and what Ecto queries are run / how long those take. You can also manually augment pretty much anything else (background jobs, API calls, ...). Plus it keeps track of some important BEAM metrics like memory consumption.
- gregpardo 9y ago
  Looks cool. Might want to have a demo site running for people to poke around.

bsder 9y ago

> I see a bright future for Elixir, and a breath of fresh air for Erlang. Is Erlang really that much of a barrier? Erlang was a touch odd, but I didn't find the language itself that mind-bending. Wrapping my head around the proper way to structure things and the proper use of the OTP libraries was much more time-consuming. That level of architectural thought doesn't magically go away because you changed the language. About the only thing I found irritating was a lack of language support for mutable hash tables, but I thought they fixed that at some point. And Erlang's bit packing/unpacking borders on magical in terms of expressiveness and speed. I wish other languages would adopt it.

vertex-four 9y ago
Unfortunately, a lot of people appear to hate Erlang's syntax to the point that it's a barrier to adoption - or Elixir never would've picked up at all. I don't understand it myself, but there we have it.
- fbernier 9y ago
  I've seen this argument about the syntax countless times and during my first looks at Erlang code I couldn't understand much of it, but after sitting down and deciding to learn it it's an extremely simple syntax. The hard/long thing to fully understand is truly OTP.
jmcgough 9y ago
It became open source in the 90s. It's been used by several high-traffic sites/apps to support chat clients (Riot Games, WhatsApp, and I think Fetlife), but outside of that has seen very little adoption. I think it's a lot harder to sell people on a language that looks syntactically unintelligible to a lot of web developers. Most people I know working with Elixir now are/were Rubyists, so it's easy for them to make the leap from that and start to understand all of the awesome things they get from BEAM and OTP.
- jhgg 9y ago
  Elixir/Erlang is great, but in addition to knowing the language, there is a huge library of tools within OTP/ERTS that you need to understand to effectively use it and skills learned from other languages don't map too well to Erlang/Elixir as they would to other languages (like going from Python to Go for example). In addition to learning OTP, you also have to learn how a functional language works if you're not familiar. A lot of things to learn to adopt Elixir/Erlang, but it's well worth it! I just don't think there's much reason for people to make that leap as there are much more familiar languages that can be used to solve the problems most developers are trying to solve.

janekm 9y ago

I read it a little differently. The whole article is about how Erlang/Elixir fails at its core reason for existence (fast message passing between distributed processes) and all the complicated work-arounds they had to implement to avoid actually using this core feature of Erlang.

mononcqc 9y ago
I have never seen a language or framework used at a large scale that did not require digging, understanding, and tweaking of the underlying machinery for its specific use case. The true failure of the platform happens when you cannot do these tweaks and adaptation, or that their cost is shadowed by having written it in a more appropriate technology at a lower time/cost/effort budget.
- janekm 9y ago
  That's a good point. The article is certainly a good summary of what's needed to make Erlang/Elixir scale and a reminder that there are no "magic bullets".
quaunaut 9y ago
I don't think that's what Erlang was made for at all? Erlang was never about speed, it's about reliability, availability, and ease of concurrency. And Elixir just makes it easier to access those tools.
- janekm 9y ago
  It may well be a misconception on my part - I assumed that as such message passing is such a core feature of using Erlang for distributed systems one should be able to treat it as a low-cost operation.
  - quaunaut 9y ago
    You can generally, just Discord is getting to a level of scale where the issue has more to do with architecture than language. All of their solutions in this post were Elixir-based solutions, and have very clean, easy APIs(and so far not very much code- all those libraries that solved their problems came out to ~400 LOC). I guess the message here is, "There's never a magic bullet, but you've got everything you need to make yours."

deleted 9y ago

[deleted]

tmat 9y ago

We're an Elixir shop and I wouldn't change it for the world. We're betting on it heavily.

censhin 9y ago

The challenge is going to be adoption, plain and simple. I worked two years on an Erlang project, and most of our frustration came from the lack of community to lean on when we needed help understanding best practices. Also, when you find only one library to solve your problem that has been abandoned for at least two years that is never fun. Few projects actually need the level of scalability that Erlang brings, and the cost of building and running an Erlang project as an enterprise application is non-trivial. People need to understand this if they want Elixir to succeed where Erlang hasn't.

jlouis 9y ago
The key benefit of Erlang isn't the scalability in my experience. It is resilience. Once your system is deployed, it tend to run with 0 fatalities for months on end. We have systems where we've had 0 maintenance days on them for months and even years. In turn, you can focus your development effort on new systems, new features, and scaling existing infrastructure since you are not bogged down in maintenance mode. Even better, you can often postpone errors in a system which are non-fatal. I often wait until I have a couple of bugs in a system and then I'll work on that system for a few days fixing them all. In systems where even the smallest amount of brittleness destroys them, this isn't really an option. Another experience of mine is that Erlang systems tend to take a bit longer to write compared to cranking out a fast solution in another language. But the payoff is that your Erlang solution tend to have better robustness as load increases and people start using the system. The community is small however, so you'll need to write more code yourself in-house as a result. For some problem spaces, this will hurt a lot because there are off-the-shelf solutions in other languages. On the other hand, larger projects which are more specialized can benefit from you knowing your own code better.
- censhin 9y ago
  I agree with what you said. What I was really trying to get across was the need for a real community and wider adoption.
- lostcolony 9y ago
  For anyone reading this, I worked in a shop where we built an Erlang system, choosing the language specifically for the high uptimes we needed. We nailed it so well that IT proactively reboots the system just because they don't trust that it's still working properly when it runs without issue for months on end. Done right, Erlang can be insanely resilient.

corobo 9y ago

It convinces me that with a selection of good devs you can hugely scale anything.

cooervo 9y ago

Truly the strenght of elixir comes for Erlang. I wonder if Scala is also as good as erlang for creating robust and fail proof distributed systems?

biokoda 9y ago
The point of Erlang is that it makes writing robust distributed systems easy and fast. The language itself is tiny. Scala on the other hand is not remotely small and easy to understand.
justinhj 9y ago
Scala paired with Akka is definitely good for that, I don't know how it compares performance wise with Elixir, would be interested to know. We're building a game server backend with lots of messaging in Akka.
- tormeh 9y ago
  Scala/Akka is a lot faster than Erlang, but it also lacks per-actor GC, which means that latency is going to be higher in a Scala/Akka system. Also, the per-actor heaps in Beam means there's no risk of two Erlang actors having a reference to the same memory, whereas that's easy to do in Scala/Akka.
  - OvermindDL1 9y ago
    Actually you'd be surprised at just how fast the BEAM passes messages, especially as it also includes linking, monitoring, cross-process, cross-system, among other features. When creating a multi-thread atomic messaging system in C++ long ago I was able to out-perform the BEAM (though I'd not rank it as significantly out-perform), but once I added failure handling features among others then it never got close to BEAM's speed again in raw message passing. However, the BEAM is not very fast on executing actual code, it is like Python in that way where it is good to slave out CPU-heavy work; the BEAM is built for async IO, and yes, running a JVM (or C++) system as a node on a BEAM mesh is fantastic for that (or a port, or NIF for small work).
9y ago