4 ms·

How does restarting the process fix the crash? If the process crashed because a file was missing, it will still be missing when the process is restarted. Is an

How does restarting the process fix the crash? If the process crashed because a file was missing, it will still be missing when the process is restarted. Is an infinite crash-loop considered success in Erlang?

lawn 1y ago
Typically you then let the error bubble up in the supervisor tree if restarting multiple times doesn't fix it. Of course there are still errors that can't be recovered from, in which case the whole program may finally crash.
- dns_snek 1y ago
  > in which case the whole program may finally crash. This may happen if you let it, but it's basically never the desired outcome. If you were handling a user request, it should stop by returning a HTTP 500 to the client, or if you were processing a background job of some sort, it should stop with a watchdog process marking the job as a failure, not with the entire system crashing.
  - Muromec 1y ago
    returning HTTP 500 as early as possible is an example of "let it crash" approach outside of Erlang.
    - dns_snek 1y ago
      That's not what "let it crash" is about. Letting something crash in Erlang means that a process (actor) is allowed to crash, but then it gets restarted to try again, which would resolve the situation in case of transient errors. The equivalent of "let it crash" outside of Erlang is a mountain of try-catch statements and hand-rolled retry wrappers with time delays, with none of the observability and tooling that you get in Erlang.
ramchip 1y ago
I recommend https://ferd.ca/the-zen-of-erlang.html https://ferd.ca/the-zen-of-erlang.html starting from "if my configuration file is corrupted, restarting won't fix anything". The tl;dr is it helps with transient bugs.
- kimi 1y ago
  ...and does no harm for unfixable bugs. It's the logical equivalent of "switch off and on again" that as we know fixes most issues by itself, but happening only on a part of your software deployment, so most of it will keep running.
- bccdee 1y ago
  > if you feel that your well-understood regular failure case is viable, then all your error handling can fall-through to that case. This is my favourite line, because it generalizes the underlying principle beyond the specific BEAM/OTP model in a way that carries over well to the more common sort of database-backed services that people tend to write.
corysama 1y ago
I’m only an armchair expert on Erlang. But, having looked into it repeatedly for a couple decades, my take-away is the “Let it crash” slogan is good. But, also presented a bit out of context. Or, at least assuming context that most people don’t have. Erlang is used in situations involving a zillion incoming requests. If an individual request fails… Maybe it was important. Maybe it wasn’t. If it was important, it’s expected they’ll try again. What’s most important is that the rest of the requests are not interrupted. What makes Erlang different is that it is natural and trivial to be able to shut down an individual request on the event of an error without worrying about putting any other part of the system into a bad state. You can pull this off in other languages via careful attention to the details of your request-handling code. But, the creators of the Erlang language and foundational frameworks have set their users up for success via careful attention to the design of the system as a whole. That’s great in the contexts in which Erlang is used. But, in the context of a Java desktop app like Open Office, it’s more like saying “Let it throw”. “It” being some user action. And, the slogan being to have a language and framework with such robust exception handling built-in that error handling becomes trivial and nearly invisible.
- nine_k 1y ago
  Let it crash, so that if something goes wrong, it does not do so silently. Let it crash, because a relevant manager will detect it, report it, clean it up, and restart it, without you having to write a line of code for that. Let it crash as soon as possible, so that any problem (like a crash loop) is readily visible. It's very easy to replace arbitrary bits of Erlang code in a running system, without affecting the rest of it. "Fix it in prod" is better than "miss it in prod", especially when you cannot stop the prod ever.
  - 0x445442 1y ago
    Are individual agents deployable on their own or does the entire "app" of agents need to be deployed as a single group? If individually deployable, what does this look like from a version control and a CI/CD perspective?
    - 1y ago
masklinn 1y ago
> Is an infinite crash-loop considered success in Erlang? Of course not, but usually that's not what happens, instead a process crashes because some condition was not considered, the corresponding request is aborted, and a supervisor restarts the process (or doesn't because the acceptor spawns a process per request / client). Or a long-running worker got into an incorrect state and crashed, and a supervisor will restart it in a known good state (that's a pretty common thing to do in hardware, BEAM makes that idiomatic in software).
- gopher_space 1y ago
  Both of your examples look like infinite crash-loops if your work needs to be correct more than it needs to be available. E.g. there aren't any known good states prior to an unexpected crash, you're just throwing a hail mary because the alternatives are impractical.
  - masklinn 1y ago
    > there aren't any known good states prior to an unexpected crash If there aren't any good states then the program straight up doesn't work in the first place, which gets diagnosed pretty quickly before it hits the field. > your work needs to be correct more than it needs to be available. "correctness over availability" tends to not be a thing, if you assume you can reach perfect and full correctness then either you never release or reality quickly proves you wrong in the field. So maximally resilient and safe systems generally plan for errors happening and how to recover from them instead of assuming they don't. There are very few fully proven non-trivial programs, and there were even less 40 years ago. And Erlang / BEAM was designed in a telecom context, so availability is the prime directive. Which is also why distribution is built-in: if you have a single machine and it crashes you have nothing.
  - Muromec 1y ago
    If it has no good states you probably know it before deploying to production.
  - dmsnell 1y ago
    When a process crashes, its supervisor restarts it according to some policy. These specify whether to restart the sibling process in their startup order or to only restart the crashed process. But a supervisor also sets limits, like “10 restarts in a timespan of 1 second.” Once the limits are reached, the supervisor crashes. Supervisors have supervisors. In this scenario the fault cascades upward through the system, triggering more broad restarts and state-reinitializations until the top-level supervisor crashes and takes the entire system down with it. An example might bee losing a connection to the database. It’s not an expected fault to fail while querying it, so you let it crash. That kills the web request, but then the web server ends up crashing too because too many requests failed, then a task runner fails for similar reasons. The logger is still reporting all this because it’s a separate process tree, and the top-level app supervisor ends up restarting the entire thing. It shuts everything off, tries to restart the database connection, and if that works everything will continue, but if not, the system crashes completely. Expected faults are not part of “let it crash.” E.g. if a user supplies a bad file path or network resource. The distinction is subjective and based around the expectations of the given app. Failure to read some asset included in the distribution is both unlikely and unrecoverable, so “let it crash” allows the code to be simpler in the happy path without giving up fault handling or burying errors deeper into the app or data.
victorbjorklund 1y ago
Elixir dev: It does not solve all issues. But sometimes you have some kind of rare bug that just happens once X,Z and Y happens in a specific order. If it is restarted it might not happen that way again. Or it might be a temporary problem. You are reaching for an API and it temporarily has issues. It might not have it anymore in 50 ms. But of course if it crashes because you are reading a file that does not exist it doesnt solve the issue (but it avoids crashing the whole system).
- victorbjorklund 1y ago
  Note that let is crash doesnt mean we shouldnt fix bugs. It is more about if there is a bug we havent fixed it is better to make the crash just crash a tiny part of the program than the whole program
  - worthless-trash 1y ago
    Or more importantly, you can't design robust recovery and retry systems.
conradfr 1y ago
If the rest of the program is still running while you fix it, yes? Also, restarting endlessly is just one strategy between multiple others.
jlouis 1y ago
It's not going to be missing the next time around. Usually the file is missing due to some concurrency-problem where the file only gets to exist a little later. A process restart certainly fixes this. If the problem persists, a larger part of the supervision tree is restarted. This eventually leads to a crash of the full application, if nothing can proceed without this application existing in the Erlang release. The key point is that there's a very large class of errors which is due to the concurrent interaction of different parts of the system. These problems often go away on the next try, because the risk of them occurring is low.

lawn 1y ago

Typically you then let the error bubble up in the supervisor tree if restarting multiple times doesn't fix it. Of course there are still errors that can't be recovered from, in which case the whole program may finally crash.

dns_snek 1y ago
> in which case the whole program may finally crash. This may happen if you let it, but it's basically never the desired outcome. If you were handling a user request, it should stop by returning a HTTP 500 to the client, or if you were processing a background job of some sort, it should stop with a watchdog process marking the job as a failure, not with the entire system crashing.
- Muromec 1y ago
  returning HTTP 500 as early as possible is an example of "let it crash" approach outside of Erlang.
  - dns_snek 1y ago
    That's not what "let it crash" is about. Letting something crash in Erlang means that a process (actor) is allowed to crash, but then it gets restarted to try again, which would resolve the situation in case of transient errors. The equivalent of "let it crash" outside of Erlang is a mountain of try-catch statements and hand-rolled retry wrappers with time delays, with none of the observability and tooling that you get in Erlang.

ramchip 1y ago

I recommend https://ferd.ca/the-zen-of-erlang.html https://ferd.ca/the-zen-of-erlang.html starting from "if my configuration file is corrupted, restarting won't fix anything". The tl;dr is it helps with transient bugs.

kimi 1y ago
...and does no harm for unfixable bugs. It's the logical equivalent of "switch off and on again" that as we know fixes most issues by itself, but happening only on a part of your software deployment, so most of it will keep running.
bccdee 1y ago
> if you feel that your well-understood regular failure case is viable, then all your error handling can fall-through to that case. This is my favourite line, because it generalizes the underlying principle beyond the specific BEAM/OTP model in a way that carries over well to the more common sort of database-backed services that people tend to write.

corysama 1y ago

I’m only an armchair expert on Erlang. But, having looked into it repeatedly for a couple decades, my take-away is the “Let it crash” slogan is good. But, also presented a bit out of context. Or, at least assuming context that most people don’t have. Erlang is used in situations involving a zillion incoming requests. If an individual request fails… Maybe it was important. Maybe it wasn’t. If it was important, it’s expected they’ll try again. What’s most important is that the rest of the requests are not interrupted. What makes Erlang different is that it is natural and trivial to be able to shut down an individual request on the event of an error without worrying about putting any other part of the system into a bad state. You can pull this off in other languages via careful attention to the details of your request-handling code. But, the creators of the Erlang language and foundational frameworks have set their users up for success via careful attention to the design of the system as a whole. That’s great in the contexts in which Erlang is used. But, in the context of a Java desktop app like Open Office, it’s more like saying “Let it throw”. “It” being some user action. And, the slogan being to have a language and framework with such robust exception handling built-in that error handling becomes trivial and nearly invisible.

nine_k 1y ago
Let it crash, so that if something goes wrong, it does not do so silently. Let it crash, because a relevant manager will detect it, report it, clean it up, and restart it, without you having to write a line of code for that. Let it crash as soon as possible, so that any problem (like a crash loop) is readily visible. It's very easy to replace arbitrary bits of Erlang code in a running system, without affecting the rest of it. "Fix it in prod" is better than "miss it in prod", especially when you cannot stop the prod ever.
- 0x445442 1y ago
  Are individual agents deployable on their own or does the entire "app" of agents need to be deployed as a single group? If individually deployable, what does this look like from a version control and a CI/CD perspective?
  - 1y ago

masklinn 1y ago

> Is an infinite crash-loop considered success in Erlang? Of course not, but usually that's not what happens, instead a process crashes because some condition was not considered, the corresponding request is aborted, and a supervisor restarts the process (or doesn't because the acceptor spawns a process per request / client). Or a long-running worker got into an incorrect state and crashed, and a supervisor will restart it in a known good state (that's a pretty common thing to do in hardware, BEAM makes that idiomatic in software).

gopher_space 1y ago
Both of your examples look like infinite crash-loops if your work needs to be correct more than it needs to be available. E.g. there aren't any known good states prior to an unexpected crash, you're just throwing a hail mary because the alternatives are impractical.
- masklinn 1y ago
  > there aren't any known good states prior to an unexpected crash If there aren't any good states then the program straight up doesn't work in the first place, which gets diagnosed pretty quickly before it hits the field. > your work needs to be correct more than it needs to be available. "correctness over availability" tends to not be a thing, if you assume you can reach perfect and full correctness then either you never release or reality quickly proves you wrong in the field. So maximally resilient and safe systems generally plan for errors happening and how to recover from them instead of assuming they don't. There are very few fully proven non-trivial programs, and there were even less 40 years ago. And Erlang / BEAM was designed in a telecom context, so availability is the prime directive. Which is also why distribution is built-in: if you have a single machine and it crashes you have nothing.
- Muromec 1y ago
  If it has no good states you probably know it before deploying to production.
- dmsnell 1y ago
  When a process crashes, its supervisor restarts it according to some policy. These specify whether to restart the sibling process in their startup order or to only restart the crashed process. But a supervisor also sets limits, like “10 restarts in a timespan of 1 second.” Once the limits are reached, the supervisor crashes. Supervisors have supervisors. In this scenario the fault cascades upward through the system, triggering more broad restarts and state-reinitializations until the top-level supervisor crashes and takes the entire system down with it. An example might bee losing a connection to the database. It’s not an expected fault to fail while querying it, so you let it crash. That kills the web request, but then the web server ends up crashing too because too many requests failed, then a task runner fails for similar reasons. The logger is still reporting all this because it’s a separate process tree, and the top-level app supervisor ends up restarting the entire thing. It shuts everything off, tries to restart the database connection, and if that works everything will continue, but if not, the system crashes completely. Expected faults are not part of “let it crash.” E.g. if a user supplies a bad file path or network resource. The distinction is subjective and based around the expectations of the given app. Failure to read some asset included in the distribution is both unlikely and unrecoverable, so “let it crash” allows the code to be simpler in the happy path without giving up fault handling or burying errors deeper into the app or data.

victorbjorklund 1y ago

Elixir dev: It does not solve all issues. But sometimes you have some kind of rare bug that just happens once X,Z and Y happens in a specific order. If it is restarted it might not happen that way again. Or it might be a temporary problem. You are reaching for an API and it temporarily has issues. It might not have it anymore in 50 ms. But of course if it crashes because you are reading a file that does not exist it doesnt solve the issue (but it avoids crashing the whole system).

victorbjorklund 1y ago
Note that let is crash doesnt mean we shouldnt fix bugs. It is more about if there is a bug we havent fixed it is better to make the crash just crash a tiny part of the program than the whole program
- worthless-trash 1y ago
  Or more importantly, you can't design robust recovery and retry systems.

conradfr 1y ago

If the rest of the program is still running while you fix it, yes? Also, restarting endlessly is just one strategy between multiple others.

jlouis 1y ago

It's not going to be missing the next time around. Usually the file is missing due to some concurrency-problem where the file only gets to exist a little later. A process restart certainly fixes this. If the problem persists, a larger part of the supervision tree is restarted. This eventually leads to a crash of the full application, if nothing can proceed without this application existing in the Erlang release. The key point is that there's a very large class of errors which is due to the concurrent interaction of different parts of the system. These problems often go away on the next try, because the risk of them occurring is low.