7 ms·

Unpacking Elixir: Resilience

213 points by lawik 3y ago

rdtsc 3y ago
A very nice description of Erlang/Elixir resilience. > Fundamentally, if there is nothing useful your code can do on a failure, no mitigation, no meaningful fallback, then you might as well have it blow up and rely on the underlying tree to make whatever is most useful out of it. I would add this is possible only because process heaps are isolated. Other systems with traditional threads/co-routines/green threads can mimic supervision trees, but unless they have isolated heaps and can safely crash at any time never affecting any other processes, it would be hard to achieve the same safety properties. > Scheduling and preempting was intended to produce consistently low latencies but it also protects you from heavy workloads bogging things down, it prevents infinitely looping bugs from slowing the system to a crawl and in general makes things resilient to things not being ideal all the time. Well put. Reliability was one of the initial requirements of Erlang, but soft real-time properties was another one. This is hard to do, but BEAM VM does a great job there. To get this right the VM has to be able to preempt processes, even if they run a tight CPU bound loop. A misbehaving process may not just crash, it could just endlessly recurse, burning CPU cycles, but still shouldn't bring the system down. > The strategies are normally one_for_one, one_for_all and one_for_rest Minor correction, the last one is rest_for_one https://www.erlang.org/doc/man/supervisor.html#supervision-principles https://www.erlang.org/doc/man/supervisor.html#supervision-p...
ecmascript 3y ago
The thing about Elixir resilience is that it takes experience to fully appreciate what it does and realize that in a web app, the actual use case when this becomes important is not that often to be totally honest. I can give an example where this could be useful, I once worked on a node-app that couldn't use clustering mode because reasons and we only had 1 server running this particular piece of code. It turned out that we had a bug that made the node app crash whenever a user posted some invalid data. The problem here was that the app itself was a SPA and the users could just keep on posting even if it failed and they did in frustration. So then app crashed and took a few seconds to reload, then crashed again. This mean that the entire api went down while the user was posting and thus could not respond to any other requests. This would never happen in Elixir and the load would just continue being ok even if 100 users at the same time would keep posting bad data. The bad thing about Elixir resilience is that it is only applied to application logic. The rest of the time shit can go wrong is just the same as any other app since most elixir projects use the same kind of tooling (postgres, some web server in front etc). Not that many seems to use the built in mnesia database, no downtime deployment etc. The BEAM comes with many cool feature in theory but very few actually utilizes them so this 99.99999% uptime rarely comes into effect. The amount of time I've had downtime on apps because of things in the application logic like the first story I mentioned has been very, very few and most of the time it's something else entirely and that thing does Elixir not really help with most of the time. Sure you could utilize all the cool features of the BEAM but it seems like in the absolute majority of cases the amount of work is simply too great for it to be worth the time investment required.
weitzj 3y ago
Partially related question: There was an article about the implementation of supervisors and you can’t use the “let is crash philosophy” in the whole erlang/beam stack but there is a small (loc) implementation in C which has to be proven safely in order for the “let it crash philosophy” to work. I don’t know if this was the definition and implementation of a supervisor itself or a process itself. Does anybody know what I mean and which article I am referring to? If so, I would be glad if you could post a link to the article
zelphirkalt 3y ago
What I also like about Elixir is, that it is a sort of nice language. What I mean by that is, that it has convenient language features like tail-call optimization, immutable by default data structures, modules (seemingly proper ones, but I am still learning, so I still need to check things like "Can I limit what a module exports?"), and great pattern matching. Then of course there is the heritage from building on top of BeamVM and Erlang: Lightweight processes. I also prefer having `do` and `end` markers for things, so that one can write expressions inline, like `fn args -> foobar end`. Having such markers enables structural code editing and things like putting the point at the `do` and then by one keyboard shortcut select everything until `end`.
thruflo 3y ago
One of the great things about the Elixir ecosystem is that, in so many cases, the libraries you’re using make the appropriate use of supervision trees so you don’t have to. Obviously it depends on what you’re building and you can craft your own supervision structures if you want to. But most of the time you can get the benefits of BEAM resiliency without having to drop down into the details.
- jlouis 3y ago
  This sort-of stems from the underlying OTP principles from Erlang. Each application has their own isolated supervision tree, and your system contains a number of applications. Hence, a library which contains a service is likely to have their own internal supervision tree. Your code can then sort-of latch onto this tree and piggyback on its existence. This is very common in web applications because when a new connection is opened, you spawn a process to handle the connection. If connections are (de-)multiplexed into channels like in HTTP/2 and HTTP/3, you'd just move up one layer and spawn a process per channel. Likewise, if you have a limited resource such as Postgres connections, they are usually maintained in a pool, and access is queued until a connection from the pool is ready for use. This can circumvent the need for something like pg_bouncer in some use-cases.
- pests 3y ago
  The one thing I dislike about Elixir is the confusion caused by how some libraries execute their functionality in-process and others spawn a supervision tree and message pass - putting your work in a possible queue that can get overloaded. Its a common pitfall for more inexperienced devs and I think its just a symptom of the language conventions. Everyone is taught genservers with a client-side API existing inside the same module as handle_ functions and it blurs the distiction some. I think if the convention was to create a separate module to implement the client-side convenience functions then the client-server split would be more obvious.
  - lawik 3y ago
    Yeah, there are a few common mistakes that can get really troublesome when library authors make them. Assuming how you want to structure execution (by using GenServers in the library usually). Another one is only one instance of config. Meaning if you have a plan to use the library for multiple distinct activities or with multiple accounts, you have trouble. Both simple config and GenServers have their place in libraries. But if a library doesn't expose the fundamentals it gets gnarly quickly.
nighmi 3y ago
Elixir seems to be picking up insane steam right now. Every day or two there is a fascinating Elixir post here and its promise seems too good to resist. Has anyone else latched their cart onto this horse?
- bicx 3y ago
  I built a fintech backend system using Eljxir a few years ago, and I loved it. However, I had to use Go and Python as a sort of escape hatch when companies offered massively time-saving SDKs in those languages but not Elixir. The ecosystem doesn’t seem to have really grown much since then. Looking around at the state of things, lots of important OSS packages have not been maintained, and that’s concerning. I absolutely love Elixir and the careful thought put into so much of the language and primary frameworks. However, I have become more skeptical that the actual ecosystem will hit critical mass. I really wish it would. I see a lot of articles on HN about Elixir, but my feeling is that it’s just more interesting than the constant stream of JS and Python we’ve had for decades now.
  - ch4s3 3y ago
    In my experience a lot of OSS elixir libraries are basically done. They get updated if the language changes, but otherwise they just do what they do well and they resist feature creep.
  - conradfr 3y ago
    Elixir is one language where you can find a library on Github that has not been updated in 6 years but will work flawlessly. I actually do small PRs to update deps etc on those kinds of repo and people are usually are OK to merge.
- lawik 3y ago
  As someone active in the community I wouldn't say there has been any particularly massive ramp up recently. Someone refernced ElixirConf US and some of the news-flow is certainly from that. Elixir has been building a steady steam pressure for a long time and I think steady is still the case. As someone writing about Elixir I will say that it has good traction here as a general rule. I think Erlang appeals to the CS-interested, Elixir appeals to a lot of Ruby and startup folks. The ecosystem also innovates a decent bit. I think it punches above its weight on the HN front page. It is a healthy ecosystem and seems to grow. (edit: a spelling)
sriram_sun 3y ago
Nice article! FTA: "One for one indicates that if a child crashes it, it and only it should be considered when restarting." What is "it" in this context?
- jlouis 3y ago
  The one_for_one policy means that if a child process terminates, it will get restarted. It is appropriate when the child processes managed by the supervisor are (relatively) independent. The contrasting policy is one_for_all where a terminating child will also induce a restart of all the other children of the supervisor. This is appropriate when there's a number of important dependencies between the children processes, and you'd better just rebuild the whole subtree. It is key to run the thought process of system failure. You need to understand what will happen if a certain part of your system errors out in an unforeseen way and make sure your restart policies are likely to right the ship.
- amilner42 3y ago
  “It” is the child. So if the supervisor (“parent”) has other children, they won’t be restarted
- actionfromafar 3y ago
  The should be a first comma after “crashes” IMHO.

213 points by lawik 3y ago

A very nice description of Erlang/Elixir resilience. > Fundamentally, if there is nothing useful your code can do on a failure, no mitigation, no meaningful fallback, then you might as well have it blow up and rely on the underlying tree to make whatever is most useful out of it. I would add this is possible only because process heaps are isolated. Other systems with traditional threads/co-routines/green threads can mimic supervision trees, but unless they have isolated heaps and can safely crash at any time never affecting any other processes, it would be hard to achieve the same safety properties. > Scheduling and preempting was intended to produce consistently low latencies but it also protects you from heavy workloads bogging things down, it prevents infinitely looping bugs from slowing the system to a crawl and in general makes things resilient to things not being ideal all the time. Well put. Reliability was one of the initial requirements of Erlang, but soft real-time properties was another one. This is hard to do, but BEAM VM does a great job there. To get this right the VM has to be able to preempt processes, even if they run a tight CPU bound loop. A misbehaving process may not just crash, it could just endlessly recurse, burning CPU cycles, but still shouldn't bring the system down. > The strategies are normally one_for_one, one_for_all and one_for_rest Minor correction, the last one is rest_for_one https://www.erlang.org/doc/man/supervisor.html#supervision-principles https://www.erlang.org/doc/man/supervisor.html#supervision-p...

ecmascript 3y ago

The thing about Elixir resilience is that it takes experience to fully appreciate what it does and realize that in a web app, the actual use case when this becomes important is not that often to be totally honest. I can give an example where this could be useful, I once worked on a node-app that couldn't use clustering mode because reasons and we only had 1 server running this particular piece of code. It turned out that we had a bug that made the node app crash whenever a user posted some invalid data. The problem here was that the app itself was a SPA and the users could just keep on posting even if it failed and they did in frustration. So then app crashed and took a few seconds to reload, then crashed again. This mean that the entire api went down while the user was posting and thus could not respond to any other requests. This would never happen in Elixir and the load would just continue being ok even if 100 users at the same time would keep posting bad data. The bad thing about Elixir resilience is that it is only applied to application logic. The rest of the time shit can go wrong is just the same as any other app since most elixir projects use the same kind of tooling (postgres, some web server in front etc). Not that many seems to use the built in mnesia database, no downtime deployment etc. The BEAM comes with many cool feature in theory but very few actually utilizes them so this 99.99999% uptime rarely comes into effect. The amount of time I've had downtime on apps because of things in the application logic like the first story I mentioned has been very, very few and most of the time it's something else entirely and that thing does Elixir not really help with most of the time. Sure you could utilize all the cool features of the BEAM but it seems like in the absolute majority of cases the amount of work is simply too great for it to be worth the time investment required.

weitzj 3y ago

Partially related question: There was an article about the implementation of supervisors and you can’t use the “let is crash philosophy” in the whole erlang/beam stack but there is a small (loc) implementation in C which has to be proven safely in order for the “let it crash philosophy” to work. I don’t know if this was the definition and implementation of a supervisor itself or a process itself. Does anybody know what I mean and which article I am referring to? If so, I would be glad if you could post a link to the article

zelphirkalt 3y ago

What I also like about Elixir is, that it is a sort of nice language. What I mean by that is, that it has convenient language features like tail-call optimization, immutable by default data structures, modules (seemingly proper ones, but I am still learning, so I still need to check things like "Can I limit what a module exports?"), and great pattern matching. Then of course there is the heritage from building on top of BeamVM and Erlang: Lightweight processes. I also prefer having `do` and `end` markers for things, so that one can write expressions inline, like `fn args -> foobar end`. Having such markers enables structural code editing and things like putting the point at the `do` and then by one keyboard shortcut select everything until `end`.

thruflo 3y ago

One of the great things about the Elixir ecosystem is that, in so many cases, the libraries you’re using make the appropriate use of supervision trees so you don’t have to. Obviously it depends on what you’re building and you can craft your own supervision structures if you want to. But most of the time you can get the benefits of BEAM resiliency without having to drop down into the details.

jlouis 3y ago
This sort-of stems from the underlying OTP principles from Erlang. Each application has their own isolated supervision tree, and your system contains a number of applications. Hence, a library which contains a service is likely to have their own internal supervision tree. Your code can then sort-of latch onto this tree and piggyback on its existence. This is very common in web applications because when a new connection is opened, you spawn a process to handle the connection. If connections are (de-)multiplexed into channels like in HTTP/2 and HTTP/3, you'd just move up one layer and spawn a process per channel. Likewise, if you have a limited resource such as Postgres connections, they are usually maintained in a pool, and access is queued until a connection from the pool is ready for use. This can circumvent the need for something like pg_bouncer in some use-cases.
pests 3y ago
The one thing I dislike about Elixir is the confusion caused by how some libraries execute their functionality in-process and others spawn a supervision tree and message pass - putting your work in a possible queue that can get overloaded. Its a common pitfall for more inexperienced devs and I think its just a symptom of the language conventions. Everyone is taught genservers with a client-side API existing inside the same module as handle_ functions and it blurs the distiction some. I think if the convention was to create a separate module to implement the client-side convenience functions then the client-server split would be more obvious.
- lawik 3y ago
  Yeah, there are a few common mistakes that can get really troublesome when library authors make them. Assuming how you want to structure execution (by using GenServers in the library usually). Another one is only one instance of config. Meaning if you have a plan to use the library for multiple distinct activities or with multiple accounts, you have trouble. Both simple config and GenServers have their place in libraries. But if a library doesn't expose the fundamentals it gets gnarly quickly.

nighmi 3y ago

Elixir seems to be picking up insane steam right now. Every day or two there is a fascinating Elixir post here and its promise seems too good to resist. Has anyone else latched their cart onto this horse?

bicx 3y ago
I built a fintech backend system using Eljxir a few years ago, and I loved it. However, I had to use Go and Python as a sort of escape hatch when companies offered massively time-saving SDKs in those languages but not Elixir. The ecosystem doesn’t seem to have really grown much since then. Looking around at the state of things, lots of important OSS packages have not been maintained, and that’s concerning. I absolutely love Elixir and the careful thought put into so much of the language and primary frameworks. However, I have become more skeptical that the actual ecosystem will hit critical mass. I really wish it would. I see a lot of articles on HN about Elixir, but my feeling is that it’s just more interesting than the constant stream of JS and Python we’ve had for decades now.
- ch4s3 3y ago
  In my experience a lot of OSS elixir libraries are basically done. They get updated if the language changes, but otherwise they just do what they do well and they resist feature creep.
- conradfr 3y ago
  Elixir is one language where you can find a library on Github that has not been updated in 6 years but will work flawlessly. I actually do small PRs to update deps etc on those kinds of repo and people are usually are OK to merge.
lawik 3y ago
As someone active in the community I wouldn't say there has been any particularly massive ramp up recently. Someone refernced ElixirConf US and some of the news-flow is certainly from that. Elixir has been building a steady steam pressure for a long time and I think steady is still the case. As someone writing about Elixir I will say that it has good traction here as a general rule. I think Erlang appeals to the CS-interested, Elixir appeals to a lot of Ruby and startup folks. The ecosystem also innovates a decent bit. I think it punches above its weight on the HN front page. It is a healthy ecosystem and seems to grow. (edit: a spelling)

sriram_sun 3y ago

Nice article! FTA: "One for one indicates that if a child crashes it, it and only it should be considered when restarting." What is "it" in this context?

jlouis 3y ago
The one_for_one policy means that if a child process terminates, it will get restarted. It is appropriate when the child processes managed by the supervisor are (relatively) independent. The contrasting policy is one_for_all where a terminating child will also induce a restart of all the other children of the supervisor. This is appropriate when there's a number of important dependencies between the children processes, and you'd better just rebuild the whole subtree. It is key to run the thought process of system failure. You need to understand what will happen if a certain part of your system errors out in an unforeseen way and make sure your restart policies are likely to right the ship.
amilner42 3y ago
“It” is the child. So if the supervisor (“parent”) has other children, they won’t be restarted
actionfromafar 3y ago
The should be a first comma after “crashes” IMHO.