10 ms·
Don't “let it crash”, let it heal
- bgdkbtv 1y agoThis is great, thanks for sharing! I've been thinking about improving error handling in my liveview app and this might be a nice way to start.
- bitwize 1y agoI don't code in Erlang or Elixir, aside from messing about. But I've found that letting an entire application crash is something that I can do under certain circumstances, especially when "you have a very big problem and will not go to space today". For example, if there's an error reading some piece of data that's in the application bundle and is needed to legitimately start up in the first place (assets for my game for instance). Then upon error it just "screams and dies" (spits out a stack trace and terminates).
- borromakot 1y agoErrors during initialization of a BEAM language application will crash the entire program, and you can decide to exit/crash a program if you get into some unrecoverable state. The important thing is the design of individual crashable/recoverable units.
- goosejuice 1y agohttps://erlang.org/pipermail/erlang-questions/2003-March/007870.html https://erlang.org/pipermail/erlang-questions/2003-March/007... The origin, as far as I know it. I think it still holds, is insightful, as a general case. Let it heal seems pretty close to what Joe was getting at.
- vrnvu 1y ago>>This organization corresponds nicely to a idealized human organization of bosses and workers - bosses say what is to be done, workers do stuff. Bosses do quality control and check that things get done, if not they fire people re-organize and tell other people to do the stuff. If they fail (the bosses) they get sacked etc. <<note I said, idealized organization, usually if projects fail the bosses get promoted and given more workers for their next project>> We miss you Joe :)
- tombert 1y agoHe was one of my favorite humans; the few emails I exchanged with him were funny and insightful.
- refactor_master 1y agoQuestion as a complete outsider: If I run idempotent Python applications in Kubernetes containers and they crash, Kubernetes will eventually restart them. Of course, knowing what to do on IO errors is nicer than destroying and restarting everything with a really bigger hammer (as the article also mentions, you can serve a better error message for whoever has to “deal” with the problem), but eventually they should end up in the same workable state. Is this conceptually similar, but perhaps at code-level instead?
- goosejuice 1y agoSomewhat, yes but it's much less powerful. In the BEAM these are trees of supervisors and monitors/links that choose how to restart and receive the stacktrace/error reason of the failure respectively. This gives a lot of freedom on how to handle the failure. In k8s, it's often just a dumb monitor/controller that knows little about how to remediate the issue on boot. Nevermind the boot time penalty. https://hexdocs.pm/elixir/1.18.4/Supervisor.html https://hexdocs.pm/elixir/1.18.4/Supervisor.html BEAM apps run great on k8s.
- adastra22 1y agoConceptually similar, different implementation. The perhaps most visible difference is that supervisors aren’t polling application state but are rather notified about errors (crashes), and restarting is extremely low latency. Erlang/BEAM was invented for telephony, and it is possible for this to happen on the middle of a protocol and the user not even notice.
- valenterry 1y agoIn general, if you can move any kind of logic to a lower level, that's better. For example, testing that kubernetes restarts work correctly is tricky and requires a complicated setup. Testing that an erlang process/actor behaves as expected is basically a unit test.
- emoII 1y agoI bet the kubernetes project has test for that, why should I as an application developer care about testing something other than my own code?
- adastra22 1y ago“Reset on error” might be a better phrasing.
- HexDecOctBin 1y agoHow does restarting the process fix the crash? If the process crashed because a file was missing, it will still be missing when the process is restarted. Is an infinite crash-loop considered success in Erlang?
- lawn 1y agoTypically you then let the error bubble up in the supervisor tree if restarting multiple times doesn't fix it. Of course there are still errors that can't be recovered from, in which case the whole program may finally crash.
- dns_snek 1y ago> in which case the whole program may finally crash. This may happen if you let it, but it's basically never the desired outcome. If you were handling a user request, it should stop by returning a HTTP 500 to the client, or if you were processing a background job of some sort, it should stop with a watchdog process marking the job as a failure, not with the entire system crashing.
- Muromec 1y agoreturning HTTP 500 as early as possible is an example of "let it crash" approach outside of Erlang.
- dns_snek 1y agoThat's not what "let it crash" is about. Letting something crash in Erlang means that a process (actor) is allowed to crash, but then it gets restarted to try again, which would resolve the situation in case of transient errors. The equivalent of "let it crash" outside of Erlang is a mountain of try-catch statements and hand-rolled retry wrappers with time delays, with none of the observability and tooling that you get in Erlang.
- ramchip 1y agoI recommend https://ferd.ca/the-zen-of-erlang.html https://ferd.ca/the-zen-of-erlang.html starting from "if my configuration file is corrupted, restarting won't fix anything". The tl;dr is it helps with transient bugs.
- BobbyTables2 1y agoHackers also love auto-restarting services. Exploitation of vulnerabilities isn’t always 100% reliable. Heap grooming might be limited or otherwise inadequate. A quick automatic restart keeps them in business without any other human interaction involved.
- teiferer 1y agoTook me a minute to realize what you meant with "hackers". Quite the irony, given the name of the site we are having this conversation on.
- valenterry 1y agoThere are a few stages, and each improves on the previous ones: 1. Detect crashes at runtime and by default stop/crash to prevent continuing with invalid program state 2. Detect crashes at runtime and handle them according to the business context (e.g. crash or retry or fallback-to or ...) to prevent bad UX through crashes. 3. Detect potential crashes at compile-time to prevent the dev from forgetting to handle them according to the business context 4. Don't just detect the possibility of crashes but also the specific type and context to prevent the dev from making a logical mistake and causing a potential runtime error during error handling according to the business context An example for stage 4 would be that the compiler checks that a fall-back option will actually always resolve the errors and not potentially introduce a new error / error type. Such as falling back to another URL does not actually always resolve the problem, there still needs to be handling for when the request to the alternative URL fails. The philosophy described in the article is basically just stage 1 and a (partial) default restart instead of a default crash, which is maybe a slight improvement but not really sufficient, at least not by my personal standards.
- creatonez 1y agoBased on your list there is an opportunity to define stage -1 of error handling sanity, the Eval-Rinse-Reload loop, as implemented by FuckItJS, the original Javascript Error Steamroller: https://github.com/mattdiamond/fuckitjs https://github.com/mattdiamond/fuckitjs > Through a process known as Eval-Rinse-Reload-And-Repeat, FuckItJS repeatedly compiles your code, detecting errors and slicing those lines out of the script. To survive such a violent process, FuckItJS reloads itself after each iteration, allowing the onerror handler to catch every single error in your terribly written code. > [...] > This will keep evaluating your code until all errors have been sliced off like mold on a piece of perfectly good bread. Whether or not the remaining code is even worth executing, we don't know. We also don't particularly care.
- valenterry 1y agoOh, thank you for the nostalgic reminder of that one. I read that a decade ago and found it hilarious.
- monkeyelite 1y agoThis seems specific to BEAM as crashing a fast-cgi process is fine and response will be handled correctly with Apache or nginx.
- snickerbockers 1y ago>When people say “let it crash”, they are referring to the fact that practically any exited process in your application will be subsequently restarted. Because of this, you can often be much less defensive around unexpected errors. You will see far fewer try/rescue, or matching on error states in Elixir code. I just threw up in my mouth when I read this. I've never used this language so maybe my experience doesn't apply here but I'm imagining all the different security implications that ive seen arise from failing to check error codes.
- josevalim 1y agoThat’s actually a good example. Imagine someone forgot to check the error code from an API response. In some languages, they may attempt to parse it as if it was successful request, and succeed, leading to a result with nulls, empty arrays, or missing data that then spreads through the system. In Elixir, parsing would most likely fail thanks to pattern matching [1] and if it by any chance that fails in a core part of the system, that failure will be isolated and that particular component can be restarted. Elixir is not about willingly ignoring error codes or failure scenarios. It is about naturally limiting the blast radius of errors without a need to program defensively (as in writing code for scenarios you don’t know “just in case”). 1: https://dashbit.co/blog/writing-assertive-code-with-elixir https://dashbit.co/blog/writing-assertive-code-with-elixir
- toast0 1y agoOk, so it's not really that you're not checking error codes. It's that you can write stuff like ok = whatever(). If whatever is successful and idomatic, it returns ok, or maybe a tuple of {ok, SomeReturn}. In that case, execution would continue. If it returns an error tuple like {error, Reason}... "Let it crash" says you can just let it crash... You didn't have anything better to do, the built in crash because {error, Reason} will do fine. Or you could do a case whatever of ok -> ok; {error, nxdomain} -> ok end. If it was fine to get nxdomain error, but any other error isn't acceptable... It will just crash, and that's good or at least ok. Better than having to enumerate all the possible errors, or having a catchall that then explicitly throws an eeror. It's especially hard to enumerate all possible errors because the running system can change and may return a new error that wasn't enumerated when the requesting code was written. There's lots of places where crashing isn't actually what you want, and you have to capture all errors, explicitly log it, and then move on... But when you can, checking for success or success and a handful of expected and recoverable errors is very nice.
- praptak 1y agoA condition that "should not happen" might still be a problem specific to a particular request. If you "just crash" it turns this request from one that only triggers a http 500 response to one that crashes the process. This increases the risk of Query of Death scenarios where the frontend that needs to serve this particular request starts retrying it with different backends and triggers restarts faster than the processes come back up. So being too eager to "just crash" may turn a scenario where you fail to serve 1% of requests into a scenario where you serve none because all your processes keep restarting.
- zwnow 1y agoThis is funny given Elixir/Erlangs whole idea is "let it crash". In Go I just have a Recovery Middleware for any type of problem. Don't know how other langs do it tho
- knome 1y agoerlang doesn't crash the program, it crashes the thread. erlang has a layered management system built in as part of OTP (open telecom platform, erlang was built for running highly concurrent telephony hardware). when a thread crashes, it dies and signals its parent. the parent then decides what to do. usually, that's just restarting the worker. maybe if ten workers have crashed in a minute, the manager itself will die and restart. issues bubble up, and managers restart subsystems automatically. for some things, like parsing user data, you might never cause the manager to die, and just always restart the worker. the article, if you should choose to read it, is explaining that people have the misconception you appear to be having due to the 'let it fail' catchphrase. it goes into detail about this system, when failing is appropriate, and when trying to work around errors is appropriate. as erlang uses greenthreads, restarting a thread for a user API is effectively instant and free.
- zwnow 1y agoIt's not a misconception given that Elixir Forum and its Discords members will say that to you. Also I never assumed the whole program crashed so why would you explain this to me? Why would one Blog guy know it better than a lot of other Elixir devs?
- atoav 1y agoThe truth is that different errors have to lead to different results if you want a good organisational outcome. These could be: - Fundamental/Fatal error: something without the process cannot function, e.g. we are missing an essential config option. Exiting with an error is totally adequate. You can't just heal from that as it would involve guessing information you don't have. Admins need to fix it - Critical error: something that should not ever occur, e.g. having an active user without password and email. You don't exit, you skip it if thst is possible and ensure the first occurance is logged and admins are contacted - Expected/Regular error: something that is expected to happen during the normal operations of the service, e.g. the other server you make requests to is being restarted and thus unreachable. Here the strategy may vary, but it could be something like retrying with random exponential backoff. Or you could briefly accept the values provided by that server are unknown and periodically retry to fill the unknown values. Or you could escalate that into a critical error after a certain amount of retries. - Warnings: These are usually about something being not exactly ideal, but do not impede with the flow of the program at all. Usually has to do with bad data quality If you can proceed without degrading the integrity of the system you should, the next thing is to decide jow important it is for humans to hear about it.
- anthk 1y agoUnix/BSD -> Crash, fix, restart. GNU/MIT/Lisp -> Detect, offer a fix, continue.
- hesus_ruiz 1y agoIt is very strange that a post trying to explain the concept of "let it crash" in Elixir (which runs on the BEAM VM) does not mention the doctoral thesis of Joe Armstrong: "Making reliable distributed systems in the presence of software errors". It must be compulsory lecture for anybody interested in reliable systems, even if they do not use the BEAM VM. https://www.diva-portal.org/smash/record.jsf?pid=diva2%3A1042000&dswid=6730 https://www.diva-portal.org/smash/record.jsf?pid=diva2%3A104...
- deleted 1y ago[deleted]
- plainOldText 1y agoSome core ideas from the paper for the inpatient (failures, isolation, healing): - Failures are inevitabe, so systems must be designed to EXPECT and recover from them, NOT AVOID them completely. - Let it crash philosophy allows components to FAIL and RECOVER quickly using supervision trees. - Processes should be ISOLATED and communicate via MESSAGE PASSING, which prevents cascading failures. - Supervision trees monitor other processes and RESTART them when they fail, creating a self-healing architecture.
- PicassoCTs 1y agohttps://fsharpforfunandprofit.com/rop/ https://fsharpforfunandprofit.com/rop/ Railway orientated programming to the rescue?
- adregan 1y agoThere are a couple of patterns for accomplishing this in Elixir. One is to build multiple function heads that pattern match on the arguments. If it’s an error tuple, pass it along. Build up your pipeline and handle any errors at the end. Another is to use the `with else`[0] expression for building up a railroad. This has the benefit of not having to teach your functions how to pass along errors. Error handling in the else block can be a little gnarly. I find it a little more manual than languages that have a `runEffect` or compose operator. In large part that’s due to the :ok, :error tuples being more of a convention than a primitive like Either/Result. 0: https://elixirschool.com/en/lessons/basics/control_structures#with-3 https://elixirschool.com/en/lessons/basics/control_structure...
- IshKebab 1y agoAh this makes sense. I always thought "let it crash" made it sound like Elixir devs just don't bother with error checking, like writing Java without any `catch`es, or writing Rust that only uses `.unwrap()`. If they just mean "processes should be restartable" then that sounds way more reasonable. Similar idea to this but less fancy: https://flawless.dev/ https://flawless.dev/ It's a pretty terrible slogan if it makes your language sound worse than it actually is.
- johnisgood 1y agoAs someone has linked it: https://erlang.org/pipermail/erlang-questions/2003-March/007870.html https://erlang.org/pipermail/erlang-questions/2003-March/007... It is about self-healing, too.
- zmgsabst 1y agoI think it’s more subtle: Imagine that you’re trying to access an API, which for some reason fails. “Let it crash” isn’t an argument against handling the timeout, but rather that you should only retry a few, bounded times rather than (eg) exponentially back off indefinitely. When you design from that perspective, you just fail your request processing (returning the request to the queue) and make that your manager’s problem. Your managing process can then restart you, reassign the work to healthy workers, etc. If your manager can’t get things working and the queue overflows, it throws it into dead letters and crashes. That might restart the server, it might page oncall, etc. The core idea is that within your business logic is the wrong place to handle system health — and that many problems can be solved by routing around problems (ie, give task to a healthy worker) or restarting a process. A process should crash when it isn’t scoped to handle the problem it’s facing (eg, server OOM, critical dependency offline, bad permissions). Crashing escalates the problem until somebody can resolve it.
- vendiddy 1y agoI think the slogan was meant to be provocative but unfortunately it has been misinterpreted more often than not. For example, imagine you're working with a 3rd party API and, according to the documentation, it is supposed to return responses in a certain format. What if suddenly that API stops working? Or what if the format changes? You could write code to handle that "what if" scenario, but then trying to handle every hypothetical your code becomes bloated, more complicated, and hard to understand. So in these cases, you accept that the system will crash. But to ensure reliability, you don't want to bring down the whole system. So there are primitives that let you control the blast radius of the crash if something unexpected happens. Let it crash does not mean you skip validating user input. Those are issues that you expect to happen. You handle those just as you would in any programming language.
- tmcb 1y agoIt is very common to interpret taglines by their face value, and I believe the author did just that, although the point brought up is valid. In order to “let it crash”, we must design the system in a way that crashes would not be catastrophic, stability wise. Letting it crash is not a commandment, though: it is a reminder that, in most cases, a smart healing strategy might be overkill.
- borromakot 1y agoAuthor: I'm literally explaining not to interpret the tag line at face value.
- tmcb 1y agoMaybe I didn’t make myself clear. “Let it crash” is not something that should be thought of at the component level, it should be thought of at the system level. The fact that the application crashes “gracefully” or not is not what is really important. You should design the system in a crash-friendly way, and not to write the application and think: “oh, I believe it is OK to let it crash here”.
- borromakot 1y agoThen I don't think you understand how the phrase is used in Elixir/Erlang. The phrase is about letting processes crash.
- tmcb 1y agoNo need for the snarky comment. If I am wrong, that is fine. Of course Joe Armstrong could explain what I meant, but in a much better way: https://erlang.org/pipermail/erlang-questions/2003-March/007870.html https://erlang.org/pipermail/erlang-questions/2003-March/007... (edit: see the "Why was error handling designed like this?" part for reference) My personal interpretation is that systems must be able to handle crashing processes gracefully. There is no benefit in letting processes crash just for the sake of it.
- stcg 1y ago"Let it crash" is a sentence that gets attention. It makes a person want to know more about it, as it sounds controversial and different. "Let it heal" doesn't have that.
- jonhohle 1y agoIt also has a deeper philosophical meaning of unexpected software bugs should be noisy and obvious instead of causing silently corruption or misleading user experience. If monitoring doesn’t catch the failure, customers will and it can be fixed right away (whether it’s the software, a hardware error, dependency issue, etc.). A web service returning a 500 error code is a lot more obvious than a 200 with an invalid payload. A crashed app with a stack trace is easier to debug and will cause more user feedback than an app than hangs in a retry loop. When I had to deal with these things in the Java world, it meant not blindly handling or swallowing exceptions that business code had no business caring about. Does your account management code really think it knows how to properly handle an InterruptedException? Unless your answer is rollback and reset the interrupted flag it’s probably wrong. Can’t write a test for a particular failure scenario? That better blow up loudly with enough context that makes it possible to understand the error condition (and then write a test for it).
- reddit_clone 1y agoI have always felt 'Let it crash' is an unfortunate turn of phrase that gets a lot of negative attention. A 'crash' in most other language/ecosystem means likely a catastrophic failure of the application, ending in a core dump. Erlang's error handling is way more nuanced than that blunt phrase indicates.
- juped 1y agoThere's really not more that's useful to say than the relevant section (4.4) of Joe Armstrong's thesis says: >How does our philosophy of handling errors fit in with coding practices? What kind of code must the programmer write when they find an error? The philosophy is let some other process fix the error, but what does this mean for their code? The answer is let it crash. By this I mean that in the event of an error, then the program should just crash. But what is an error? For programming purpose we can say that: >• exceptions occur when the run-time system does not know what to do. >• errors occur when the programmer doesn’t know what to do. >If an exception is generated by the run-time system, but the programmer had foreseen this and knows what to do to correct the condition that caused the exception, then this is not an error. For example, opening a file which does not exist might cause an exception, but the programmer might decide that this is not an error. They therefore write code which traps this exception and takes the necessary corrective action. >Errors occur when the programmer does not know what to do. Programmers are supposed to follow specifications, but often the specification does not say what to do and therefore the programmer does not know what to do. >[...] >The defensive code detracts from the pure case and confuses the reader—the diagnostic is often no better than the diagnostic which the compiler supplies automatically. Note that this "program" is a process. For a process doing work, encountering something it can't handle is an error per the above definitions, and the process should just die, since there's nothing better for it to do; for a supervisor process supervising such processes-doing-work, "my child process exited" is an exception at worst, and usually not even an exception since the standard library supervisor code already handles that.
- Jtsummers 1y agoI think a lot of folks who have never looked at Erlang or Elixir and BEAM before misunderstand this concept because they don't understand how fine-grained processes are, or can be, in Erlang. A very important note: Processes in BEAM languages are cheap, both to create and for context switching, compared to OS threads. While design-wise they offer similar capabilities, this cost difference results in a substantially different approach to design in Erlang than in systems where the cost of introducing and switching between threads is more expensive. In a more conventional language where concurrency is relatively expensive, and assuming you're not an idiot who writes 1-10k SLOC functions, you end up with functions that have a "single responsibility" (maybe not actually a single responsibility, but closer to it than having 100 duties in one function) near the bottom of your call tree, but they all exist in one thread of execution. In a system, hypothetical, created in this model if your lowest level function is something like: retrieve_data(db_connection, query_parameters) -> data And the database connection fails, would you attempt to restart the database connection in this function? Maybe, but that'd be bad design. You'd most likely raise an exception or change the signature so you could express an error return, in Rust and similar it would become something like: retrieve_data(db_connection, query_parameters) -> Result<data, error> Somewhere higher in the call stack you have a handler which will catch the exception or process the error and determine what to do. That is, the function `retrieve_data` crashes, it fails to achieve its objective and does not attempt any corrective action (beyond maybe a few retries in case the error is transient). In Erlang, you have a supervision tree which corresponds to this call tree concept but for processes. The process handling data retrieval, having been given some db_conn handler and the parameters, will fail for some reason. Instead of handling the error in this process, the process crashes. The failure condition is passed to the supervisor which may or may not have a handler for this situation. You might put the simple retry policy in the supervisor (that basic assumption of transient errors, maybe a second or third attempt will succeed). It might have other retry policies, like trying the request again but with a different db_connection (that other one must be bad for some reason, perhaps the db instance it references is down). If it continues to fail, then this supervisor will either handle the error some other way (signaling to another process that the db is down, fix it or tell the supervisor what to do) or perhaps crash itself. This repeats all the way up the supervision tree, ultimately it could mean bringing down the whole system if the error propagates to a high enough level. This is conceptually no different than how errors and exceptions are handled in sequential, non-concurrent systems. You have handlers that provide mechanisms for retrying or dealing with the errors, and if you don't the error is propagated up (hopefully you don't continue running in a known-bad state) until it is handled or the program crashes entirely. In languages that offer more expensive concurrency (traditional OS threads), the cost of concurrency (in memory and time) means you end up with a policy that sits somewhere between Erlang's and a straight-line sequential program. Your threads will be larger than Erlang processes so they'll include more error handling within themselves, but ultimately they can still fail and you'll have a supervisor of some sort that determines what happens next (hopefully). As more languages move to cheap concurrency (Go's goroutines, Java's virtual threads), system designs have a chance to shift closer to Erlang than that straight-line sequential approach if people are willing to take advantage of it.