31 ms·

How “let it fail” leads to simpler code

199 points by _benj 4y ago

_benj 4y ago
I think the distinction between expected and unexpected errors can easily fall through the cracks and writing code in a way that an unexpected error doesn’t break everything is quite powerful. Golang makes it easy to ignore errors that can be ignored and defer/recover provide a way to implement a way to “let it fail” There’s even an implementation of supervisor trees for Go [0] :) [0] https://github.com/thejerf/suture https://github.com/thejerf/suture
- rad_gruchalski 4y ago
  This would be my go to for anything _supervisor_ in golang: https://github.com/asynkron/protoactor-go#supervision https://github.com/asynkron/protoactor-go#supervision.
sergiotapia 4y ago
For our liveview project, a lot of the bugs we find are edge cases in the pattern match. We find the bug in appsignal, build another arity match and go on with our day. It's pretty cool. I've been working in Elixir exclusively since 2016. I do think a lot of the Let It Fail is just marketing from Elixir (and BEAM) but there is a lot of truth in it. In reality you will most definitely not write everything under an explicit supervisor. You will just see errors in function clause matches and add another arity.
- rad_gruchalski 4y ago
  The origin of Let it Crash dates to „Making reliable distributed systems in the presence of sodware errors”, Joe Armstrong, 2003: https://erlang.org/download/armstrong_thesis_2003.pdf https://erlang.org/download/armstrong_thesis_2003.pdf, section 4.4.
  - sergiotapia 4y ago
    I know. I'm saying for my projects, I never had to reach that level of robustness and distribution. I imagine the vast majority of Elixir/Phoenix projects are the same, especially with the language gaining traction and new projects being spun up with the language.
  - smackeyacky 4y ago
    Ha. From now on, we should refer to badly written services as "sodware". Sounds like something The Register would use a lot.
- stu2b50 4y ago
  > In reality you will most definitely not write everything under an explicit supervisor. That's the point of the supervisor _tree_. Certainly not every process will have its own supervisor, but all processes should be linked to other processes, which are linked to other processors, and at some point you have a process that is quite fundamental to the application and has a supervisor. As long as there's a supervisor somewhere on that tree, the whole subtree will be restarted, hopefully in a non-erroneous state, and the application will continue on its merry way.
- toast0 4y ago
  > We find the bug in appsignal, build another arity match and go on with our day. Failing (as crashing is now termed ;) immediately when the data didn't match the pattern is exactly the let it fail approach. If the data doesn't meet the expectations, there's nothing to do but crash. Maybe you've got a nice supervision tree, maybe not, but crashing immediately where things didn't match expectations usually gives you the right place to start looking; maybe it was some reasonable data, so you just handle it. Maybe it is unreasonable, so you need to look at where it came from, but usually (not always, of course) you just got the data and are pattern matching it, so you know where it came from too.
trav4225 4y ago
I've seen a lot of new developers shocked by this approach, which surprises me a little. They seem to think that it's up to the application to handle all errors, even those of the programmer(s). This, of course, is unreasonable since it would essentially require knowing all the bugs in advance. :-)
- allenu 4y ago
  I'm a big fan of the "crash early" strategy. I write in Swift primarily, and if I suspect a state is impossible to reach, I'll add a fatalError() so that in development, if it turns out I'm wrong, I spot it right away. (Something I learned from another dev I worked with, who was very productive.) Unfortunately, a lot of other devs hate to see that your code may actually crash and start asking questions about what scenario could cause it and asking if maybe there's a more gentle way to get out of the error. So, I'll often back down and start having softer error-handling, but on the whole it does complicate things further as the errors cascade and now you have to reason about handling combination of errors that have low likelihood of happening. So, to me, just having an early crash is way better.
  - uoaei 4y ago
    > if I suspect a state is impossible to reach, I'll add a fatalError() Does Swift have assert statements? If so, is there a reason you chose this method instead?
    - allenu 4y ago
      Yes, Swift has assert statements. I tend to use them a lot as well, but in shipping code, they don't terminate the app. There are still some places where I'd prefer to terminate the app early rather than continue on. To be clear, I tend to use assert statements more than fatalErrors.
  - aaaaaaaaaaab 4y ago
    Yeah. My pet peeve is `guard let … else { return }` instead of force-unwrapping. Like, why do people think that silently swallowing an error is better than crashing loudly and clearly?
akdor1154 4y ago
I'd be very interested to see non-BEAM approaches to enabling this - i kind of end up in the same pattern thanks to "expected? Return an Error<E>. Unexpected? Throw." However, the supervising part is then difficult. How do people approach this in Python? NodeJS? Rust? .NET?
- jetbalsa 4y ago
  Its common for me in PHP land to call die("Error! $e"); when something really bad has happened in a request and I really don't want anything to happen
  - akdor1154 4y ago
    Yeah nice, PHP's request scoping is a really good fit for this.
  - theluketowers 4y ago
    That's not without its drawbacks however, especially when you consider tools like swoole and laravel octane that keep your application in a long running process, so if you use die you actually kill one of the server workers instead of just terminating the execution of the request
- vvanders 4y ago
  You can take the same patterns and apply them, it just takes a little more rigor since beam/otp had this built in. My favorite is hardware (or software) watchdog timers. Very simple to implement and surprisingly effective.
- game-of-throws 4y ago
  If you can stomach kubernetes, you get this for free for all languages. Just panic!() or die(), and you'll get a fresh pod in a few seconds. Outside of k8s I try to use the lang's preferred tool. Python -> supervisord, Node -> pm2, etc.
- andrewingram 4y ago
  If you’re doing web stuff, frameworks will basically have this built in. Typically the framework’s own request handler will have something that catches all uncaught exceptions within the scope of that request and return a 500 response. So within the context of a single request you can usually adopt a “let it crash philosophy”. In fact, it’s something most people seem to do intuitively.
smackeyacky 4y ago
I don't agree with this approach. Say you have a network service that relies on other network services. It is not difficult to write those such that they know to back off / retry when something disappears. It's extremely useful in a lot of situations: if you do work on a laptop that gets regularly unplugged, having running test services that know to reconnect makes your life easier. In production, having things automatically reconnect means a lot less restarting of services once whatever root cause problem is corrected. Just shrugging and giving up ends up being a lot more work in the end. I like to tell junior developers to catch everything they can, and handle it or die as nicely as possible. Of course you can't plan for everything, but you can write around network and disk issues and issue warnings in a way that makes the root cause more obvious. That involves catching errors.
- verdagon 4y ago
  I'd imagine retrying/reconnecting is compatible with the general "let it fail" approach. If you just sent a message/request to an actor/server and it still hasn't responded after 5 seconds, you can send another. It wouldn't matter whether that actor/server died from a regular error or a "let it fail" error, the retrying would still work the same.
  - _gabe_ 4y ago
    > If you just sent a message/request to an actor/server and it still hasn't responded after 5 seconds, you can send another. This depends on the message. I hope amazon doesn't just send another message if my transaction didn't complete in 5 seconds. I think like all pieces of wisdom, sometimes it's OK to let it fail, and sometimes its OK to handle the errors. If anyone ever tells me to always do X or never do X, it's typically not sound advice. The one thing we can always count on is generic advice failing sometimes :) (And even in this article, you shouldn't always try to handle known errors and never try to handle unknown errors, there will be exceptions)
    - ShroudedNight 4y ago
      > I hope amazon doesn't just send another message if my transaction didn't complete in 5 seconds. I do, but only while also associating a stable nonce to the transaction.
Mr_P 4y ago
For all the hate that Java tends to get, the language natively supports this distinction between: * Expected errors - Checked Exceptions * Unexpected errors - Unchecked Exceptions Idiomatic Java also makes heavy use of asserts, e.g. using the Guava Preconditions library.
- josephcsible 4y ago
  The problem is that while Java the language itself does support that distinction, a lot of built-in stuff really messes it up. For example, exceptions from closing a file are unexpected, but are an IOException which is checked anyway. Also, even the support that is in the language isn't first-class; e.g., lack of exception polymorphism.
  - zmmmmm 4y ago
    I think that's a symptom of the fact that the distinction is really artificial at the language level anyway. Whether something is expected or not is a function of the requirements. Even OutOfMemory can be expected and handled in certain types of applications (esp. since it gets thrown for things like file handles rather than true memory). And then there are all kinds of cases where routine exceptions like file not found are in fact, unexpected errors (as discussed in TFA). Perhaps some sort of language level solution could have been found (eg: have explicit interfaces to mark exceptions as expected or unexpected and then exceptions are assigned that using generics or something), but that ship has sailed long ago.
    - rowls66 4y ago
      This is right, therefore, in most cases, a library should throw a checked exception, and the caller should decide whether it is an expected error and either handle it or rethrow it, or it is unexpected and rethrow a RuntimeException.
- shepherdjerred 4y ago
  Unchecked vs Checked is one of the things I like least about Java. Programmers tend to make everything Unchecked because it leads to easier code for API users at the cost of correctness/error handling.
verdagon 4y ago
I think that the "let it fail" approach is often inevitable, even when we try to use Result<T, E>. Often, we see an "unknown" variant in the error enum, as a catch-all for a library's unexpected errors. Then, anyone who calls them must also have an "unknown" enum. And anyone who calls them, and so on. In the end, this "unknown" variant is similar to a panic, in that there's very few reasonable reactions to it: Log it, cancel the request, return error 500, perhaps retry. For this reason, I often recommend people to just use assertions and panics.
- hyperhopper 4y ago
  While everything you said is correct, there are still significant advantages to the 'result' method. Sometimes you want to return 200 even if most of the backends fail. Sometimes one part may want to retry based on any error. Even aside from this, disallowing exceptions leads to a very predictable control flow, and makes program state able to be expressed in the type system, which is useful for many reasons on it's own. While yes, it's often just like an exception or panic, I'll take that over exceptions in my code any day
  - verdagon 4y ago
    I realize I was ambiguous; I didn't mean to say "just use assertions and panics", I meant "just use assertions and panics for unexpected errors", my apologies. I wouldn't recommend someone only use Result<T, E> and never panic. If we do, then anything that might indirectly be invalid, such as a map lookup or an array index, will have ? operators on it, often every line of some functions. In the end, our control flow is just as unpredictable as if we just used panics and our signal is lost in the noise. For this reason, I think a blend of Result<T, E> and panics/assertions is really the way to go.
kbyatnal 4y ago
the best TLDR I've seen of this philosophy is - if you find yourself writing all over your codebase try { ... } catch (e) { console.log(e) } then you should probably just "let it fail" since you can't actually handle the error
- deleted 4y ago
  [deleted]
- charcircuit 4y ago
  That construction adds a layer of "let it fail." For example you might want to have your web server be able to mark a connection as failed instead of only being able to mark the entire server as having failed.
- anupamchugh 4y ago
  That's impressive!
rektide 4y ago
I spent a good part of this week overhauling a microservice where most fucntions were a giant try/catch & would maybe throw a new error. Just getting rid of the try catches & letting the code fail has been a huge help in seeing what is going wrong as the code executes. I also am delighted to see the idea of expected errors here. Another thing I've been doing for a long time is tagging erros with an expected = true property when it's something we expect to see, like, oh, we went to get this oauth token but the credentials were wrong. Expectedness shows up in thr logs now & we can see mych more clearly where there are real problems.
ramraj07 4y ago
A corollary or generalized interpretation of this approach (and someone please specify if there’s a formal term for this) is: “fail locally, and immediately.” What I mean is that once something unexpected happens your code should ideally fail in that step itself. The simplest most common example I’ve seen with python programmers is when they pass around dicts as arguments in complex code bases. Methods expect various keys to be present, and often methods also have fail safe defaults if some keys are absent. The defaults are written for the specification, sure, but often they also tolerate unexpected exceptions that happened upstream. Now when an unexpected exception happens, your program fails somewhere else and the stacktrace is useless. The only way to figure out what went wrong is to debug it line by line. With python there’s still no elegant solution. I’m now trying to ensure all my methods are typed and use dataclasses and pydantic classes to type and group these parameters but there’s still opportunities for these “fail later” errors. Solutions and suggestions would be appreciated!
- bzxcvbn 4y ago
  >Solutions and suggestions would be appreciated! Use a language with strong typing?
  - aunderscored 4y ago
    Python is strongly typed. You want statically typed. (Instead of duck typed / dynamically typed)
    - jahewson 4y ago
      FWIW “strong” typing is a colloquial term and lacks any precise meaning.
    - bzxcvbn 4y ago
      Can you guess what this code does? class foo: pass obj = foo() obj.bar = "I thought Python was strongly typed?" print(obj.bar) And even better: class foo: a = 42 obj = foo() print(obj.a) del foo.a print(obj.a) Whatever your opinion on what the imprecise sentence "strongly typed language" should mean, these are definitely not features of one.
robocat 4y ago
The article doesn’t seem to look at how resources are cleaned up when a BEAM process crashes. https://elixirforum.com/t/understanding-the-advantages-of-let-it-crash-term/9748/16 https://elixirforum.com/t/understanding-the-advantages-of-le... says “All resources are owned by a process in Erlang, and the VM guarantees clean-up of resources once the process dies”. My Google-fu failed me when I searched for more details about Erlang process cleanup of resources, or how to register cleanup actions (e.g. delete some temporary file on crash).
- btkostner 4y ago
  I’d take a look at the terminate callback for Elixir GenServers. https://hexdocs.pm/elixir/1.12/GenServer.html#c:terminate/2 https://hexdocs.pm/elixir/1.12/GenServer.html#c:terminate/2
- jeanlucas 4y ago
  I am not sure what you mean by that. Which resources do you mean? Everything happening in your program is running inside the Erlang VM, even file writing... And each process inside the VM does their own garbage collection. The only way I could see the VM itself crashing is if it can't allocate memory, then it all crashes.
  - robocat 4y ago
    Let’s assume we are using Linux. Firstly, an Erlang VM process is not a Linux child process: “Erlang processes are lightweight, operate in (memory) isolation from other processes, and are scheduled by Erlang's Virtual Machine (VM). The creation time of process is very low, the memory footprint of a just spawned process is very small, and a single Erlang VM can have millions of processes running.”. An Erlang program can use kernel resources. Objects related to file descriptors, filesystem data, the process namespace, and signal handlers. Kernel timers, semaphores, memory allocations (e.g. for FFI), sockets, DMA, etcetera. I presume the Erlang library code tracks resources (such as file access function tracking file descriptors per Erlang process), so they can be cleaned up if we “let-it-fail”. I also presume there is a canonical way to register to clean up resources - the equivalent to atexit()[1]. [1] https://man7.org/linux/man-pages/man3/atexit.3.html https://man7.org/linux/man-pages/man3/atexit.3.html
WalterBright 4y ago
I learned from working on aviation systems is that when a system enters an unknown state, it must be disabled and locked out. In software, this is known as an assertion failure. When the assert trips, the program is, by definition, in an unknown state. A program cannot reasonably be allowed to continue in an unknown state - it may launch nuclear missiles. The only thing to be done is exit directly, do not pass Go, do not collect $200.
- glouwbug 4y ago
  What if the plane is mid flight?
  - bicx 4y ago
    It won’t be in mid-flight for much longer.
  - WalterBright 4y ago
    Engage the backup. Everything flight critical is dual.
- mmcnl 4y ago
  I like this mindset.
- roeles 4y ago
  Thanks for posting this. I have worked on non critical flight software and thought that this philosophy might work well. I wonder how easy the certification is for such software? For work I might have to write Do178 code in the future.
  - WalterBright 4y ago
    I use it in the software I write. I should do a presentation sometime about how the aviation industry should be influencing software development.
    - 4y ago
winter_blue 4y ago
In a new language, I'd like to see exceptions being allowed in pure code, but prohibited in non-pure code. (Non-pure here meaning code with side effects.) In pure code, an exception could essentially be passed up, and transformed into an error return value at the point where it's called by non-pure code.
- mike_hock 4y ago
  So then you make a pure function `throw_exception` and now you can throw from impure code.
  - winter_blue 4y ago
    In non-pure code, we would want to force the programmer to handle the exception. In this case, wherever `throw_exception` is called, the programmer would have to handle the exception, and either refactor the return value to indicate an error, change the return value to an Optional (and silently fail by returning an empty Optional), or cause/trigger a side effect (like terminating the thread/program termination early, with an exit code).
jeanlucas 4y ago
That's why Erlang is cool, to me.
angarg12 4y ago
One of the pieces of software I'm most proud of is a service to manage the dynamic part of our infrastructure. It uses control theory and let it fail to great effect. The service reads the state of the system, and applies change to converge to a configured policy. If it encounter an error, it doesn't try to handle or fix it, it just fails and logs a fine grained metric, plus a general error metric. The system fails all the time at this scale, but heals itself pretty quickly. In over 1 year of operation it hasn't caused a single incident, and it has survived all outages.
- kortilla 4y ago
  This is just kubernetes right? Declarative desired state model. Containers created and destroyed to get there. Crashes happen, metrics are incremented, load balancers route around the crashing pod until they recover (or are replaced), etc.
- arunaugustine 4y ago
  Yes, please give us more info about using control theory and how one might think about building such a system please..
  - wsc981 4y ago
    My guess would be add assertions everywhere instead of throwing exceptions.
    - Karellen 4y ago
      Why not throw exceptions, and just never use try/catch? That way, all exceptions are uncaught and should terminate the program, in a way that takes advantage of the programming language's native error reporting facilities.
      - hsn915 4y ago
        assertion failures terminate the program immediately. exceptions usually trigger cleanup code. If your cleanup code is mostly closing files and clearing memory, then it's useless because the OS will do that for your crashed program anyway.
at_a_remove 4y ago
I struggle to find the correct descriptor for a counter-example, wherein You Really Want Success for the process as a whole, but it is acceptable for a sliver of it to fail, in the the context of ETL. I have an ETL I am told (I switched jobs) that is still working, from 2008. It was built to be a tank, and I also did another forbidden thing: Pokemon Exception Handling. It's a guideline, not a law of physics, and it is fine to resort to a general error catch when you really don't know every possible error (and let's be honest, if you have enough libraries in the mix, some surprises will happen) and you want the other 99.999% of the data to go through. Yes, this one little thing didn't load, and let's log that, let's examine that and figure out how to prevent that going forward, but overall, the rest of the program must continue. How did it get so tanklike? Every time a little bit failed and it got logged, I figured out what went wrong, fixed it, and then tried to generalize a class of similar errors. After a while, I got into Things I Was Told Would Never Happen in the data we ingested and programmed for when never happened. Reader, never came a little sooner than expected. Anyway, I largely agree with the idea but there are places where you want the exact opposite, and I think it is important to look for those places lest this heuristic become so stiff it can lose utility.
- e9 4y ago
  Funny thing, I had same experience. I also built robust ETL. It was for ingesting and manipulating financial data from 30+ different banks and I also did Pokemon Exception Handling to make it robust. In general my philosophy is: I don't want to wake up at 4am unless it's urgent. What can I do to gracefully handle failures to achieve that goal?
  - at_a_remove 4y ago
    Exactly, thank you. At the time I was transitioning from a jack of all trades person to strictly a programmer. I despise getting called in the middle of the night. My feeling is that, if you have a foundational business process like this, it should be designed to be maintained if there is a serious problem, and it ought to keep working. I know, haha, "the Internet is a series of tubes" but I really wanted this thing to be like a chunk of very uninteresting ductwork that just moves air from one place to another: it should just do its job with as much fanfare.
czei002 4y ago
Other frameworks like express (nodejs) or actix (Rust) also don't crash if you "throw" in an request handler so this doesn't sounds very exciting to me. The interesting question for me is how retries are handled after an error occurred? For example, if the error happens in an http request handler, does the request still fails with 500 or is it magically retried by Erlang while keeping the request hanging? For internal service calls how are retries working? i.e. how can I configure that a request is retried after a failure? I guess Erlang does this and this is the power behind it? The example of a missing file seems not very good since its a problem that is probably not solved by waiting. A better example is probably a busy DB that is temporary not reachable?
- jon-wood 4y ago
  The Erlang VM is built around message passing, so in the case of a dropped database connection your application code would pass a message to the database API asking for the results of a SQL query. The database API is currently trying to reconnect, so it’s not processing that queue of messages, but once it gets there it’ll pick up the message, run the query, and then send one back to the process that asked. This is all largely transparent to your application code, beyond being able to set some preferences in your return message handler around things like how long you’re willing to wait. The whole “let it crash” thing comes from Erlang’s process supervision - in practice the DB API isn’t actually retrying. It’s continually failing to connect and if it can’t the process just crashes. The supervisor then notices and starts a new process in its place, this continues until either a process successfully starts or the configured retry count is hit. If the retry count is exceeded then the supervisor crashes, either taking the entire application with it, or more commonly being restarted itself by the next supervisor up the tree.
- jswny 4y ago
  The point is that Erlang (the BEAM VM) follows this pattern everywhere, not just for web requests. It’s baked into the language and runtime, and that’s infinitely more powerful and customizable than not crashing in a request handler.
kgeist 4y ago
>ignore unexpected exceptions Isn't it the way it already is in practice, not something specific to Erlang? If an exception is unexpected, usually there won't be an exception handler for it, otherwise a developer pretty much expected it. Developers are generally lazy so in my practice the default is usually to let it fail, and there's usually going to be an exception handler that does something other than logging and quitting only if there's a serious reason to do so. Maybe a more useful distinction could rather be "business logic errors" vs. everything else ("infrastructure errors", "programming errors" and "input validation errors"). Business logic should clearly define what should be done when an error happens, to avoid inconsistent state. But infrastructure-level errors or programming errors, you can't do much about them, other than log and/or retry.
- gabeio 4y ago
  > If an exception is unexpected, usually there won't be an exception handler for it That honestly depends on how the language and program are written, python is a great and horrible example of where you can handle any exception even ones that were just created by the program: try: crash_hard_here() except: # by default (and unfortunately) will catch *everything* pass # and this is one of the worst offenders inside an except, to just outright ignore the exception and continue as if nothing happened and to not even log it. I can not tell you the amount of production code where I've seen catch all exceptions, and they are the lazy way to know something will not "crash" even though much worse things can happen now.
  - blub 4y ago
    This catch and ignore seems to be straight out of the (sadly not real) Visual Basic design patterns book :-)
- blub 4y ago
  In C++ for example there’s an exception hierarchy and several classes of errors similar to what you enumerated, so the handler might catch a specific exception as a more generic one. There’s also … which catches all exceptions. This is fine, as one has the choice of exposing error details to outside code or not.
kaczordon 4y ago
I believe this style of coding is known as “Design by contract”. Basically assigning a range and domain to each function.
mewse 4y ago
I was a believer in the "my code should never crash, no matter what" school of thought until I shipped a Dreamcast game with an out-of-date opening cutscene. It was an in-engine opening cutscene which was very nearly final; the file we shipped was about two or three weeks out of date compared against the version that should have gone on the disc (It had one missing shape key on a character's face at the end of a shot, and a couple other missing elements). My code was wrangling the whole animation; doing all the stuff which our at-the-time-primitive animation system couldn't do itself (animating texture coordinates and etc). And my code was just silently handling all the errors it ran into so that we never even noticed that anything was wrong. The difference was subtle enough that in the twelve years since the game was released, nobody but the original animator has ever noticed and mentioned it to me (and that, years after release). But that one experience and knowing how much worse it could have been was enough to convince me that "crashes early and crashes loudly with as much detail as possible" is by far the better strategy. At least for entertainment products. And doubly so for entertainment products which can't be patched after release. (for clarity, this screw-up was 100% my fault. The animators had made the final changes to the cutscene data files in plenty of time for inclusion in the final build, I just somehow didn't import the changed data files into the game when I made the matching changes to the code side, and then my code didn't throw any errors to tell me or anyone else on the project that anything was wrong.)
- mike_hock 4y ago
  Or even better, it should be the latter during development and the former in the released version. You don't want your released game to crash in level 11 if the player happens to look behind the wrong lightpole because a texture is missing, but you do want to notice that in development.
  - XorNot 4y ago
    I remember back when I messed with D3D (around 8?) I was surprised to learn this is how the Debug build mode of it works. In Debug you can completely screw up your pipeline and scene handling and...everything will work just fine. Or appear to. Switch to a Release build and all those failures are suddenly very obvious - and this is almost exactly what you wouldn't want.
mpweiher 4y ago
Dataflow offers an interesting variation of not polluting the happy path with error handling: just do nothing. How so? Well, the happy path passes data to the next filter. In the case of an error, simply do not do that, and the next filter will be none-the-wiser. Optionally, log the error, possibly with enough information that the "logger" could retry the operation if appropriate. But as others have pointed out, there often isn't much that can be done.
mkl95 4y ago
One of the issues I take with this mentality is that companies are always looking for excuses to sell bad software. As an engineer, I really don't enjoy maintaining crappy SaaS, particularly if they are built with weakly typed, dynamic languages. If something is obviously prone to fail you should empower people to do something about it, because usually "letting it fail" will lead to your users switching to your competitor's offering.
smallerfish 4y ago
I've heard this expressed as "write brittle code", and I'm a strong advocate for it. Looking up a user by id, and get no results? Rather than either passing null up the stack, or wrapping null with an optional.empty, throw an exception! It's the client code's problem if it somehow got hold of an id that doesn't exist. (Yes, ymmv depending on the system, e.g. if you're dealing with eventual consistency then maybe do something different.) As the article says, this of course doesn't mean that you shouldn't handle user errors, or even known system errors.
torginus 4y ago
The article mentions Erlang, a functional language - which gives an interesting contrast - it is a functional language which are all about mapping out all behavior so that no undefined behavior can exist, and basically force you to consider every possibility (of course that doesn't account for stuff like network errors). Wouldn't the same scheme be better suited for a procedural language, with deliberately dirty code full of gotchas? I write a ton of hacky scripts, like last time I needed to rewrite 1000s of xml-s I wrote a crude regex replace for it. It worked 99% of the time, and I fixed up the rest manually. This sort of thinking - that a subprocess might fail due to whatever reason, including sloppy code - but the whole process will keep trucking on would be perfect for the this paradigm. Additionally this would open the path for stuff like trivial hot code replacement - since in this system, a subprocess that crashes every time - like an invalid program would be handled by the system.
wolffiex 4y ago
This is the way. Exception handling is often one of the worst aspect of a production codebase, especially since it is typically added late. Though error handling strategies benefit from careful design, they are usually added piecemeal. Making errors louder and more problematic is the best way to get them the attention they deserve. This is not a new concept and it seems to be one of the core components of the Erlang Weltanschauung. It can be generalized further to systems as the principle of "Crash-Only Software," as advanced in this classic paper: https://dslab.epfl.ch/pubs/crashonly.pdf https://dslab.epfl.ch/pubs/crashonly.pdf

199 points by _benj 4y ago

I think the distinction between expected and unexpected errors can easily fall through the cracks and writing code in a way that an unexpected error doesn’t break everything is quite powerful. Golang makes it easy to ignore errors that can be ignored and defer/recover provide a way to implement a way to “let it fail” There’s even an implementation of supervisor trees for Go [0] :) [0] https://github.com/thejerf/suture https://github.com/thejerf/suture

rad_gruchalski 4y ago
This would be my go to for anything _supervisor_ in golang: https://github.com/asynkron/protoactor-go#supervision https://github.com/asynkron/protoactor-go#supervision.

sergiotapia 4y ago

For our liveview project, a lot of the bugs we find are edge cases in the pattern match. We find the bug in appsignal, build another arity match and go on with our day. It's pretty cool. I've been working in Elixir exclusively since 2016. I do think a lot of the Let It Fail is just marketing from Elixir (and BEAM) but there is a lot of truth in it. In reality you will most definitely not write everything under an explicit supervisor. You will just see errors in function clause matches and add another arity.

rad_gruchalski 4y ago
The origin of Let it Crash dates to „Making reliable distributed systems in the presence of sodware errors”, Joe Armstrong, 2003: https://erlang.org/download/armstrong_thesis_2003.pdf https://erlang.org/download/armstrong_thesis_2003.pdf, section 4.4.
- sergiotapia 4y ago
  I know. I'm saying for my projects, I never had to reach that level of robustness and distribution. I imagine the vast majority of Elixir/Phoenix projects are the same, especially with the language gaining traction and new projects being spun up with the language.
- smackeyacky 4y ago
  Ha. From now on, we should refer to badly written services as "sodware". Sounds like something The Register would use a lot.
stu2b50 4y ago
> In reality you will most definitely not write everything under an explicit supervisor. That's the point of the supervisor _tree_. Certainly not every process will have its own supervisor, but all processes should be linked to other processes, which are linked to other processors, and at some point you have a process that is quite fundamental to the application and has a supervisor. As long as there's a supervisor somewhere on that tree, the whole subtree will be restarted, hopefully in a non-erroneous state, and the application will continue on its merry way.
toast0 4y ago
> We find the bug in appsignal, build another arity match and go on with our day. Failing (as crashing is now termed ;) immediately when the data didn't match the pattern is exactly the let it fail approach. If the data doesn't meet the expectations, there's nothing to do but crash. Maybe you've got a nice supervision tree, maybe not, but crashing immediately where things didn't match expectations usually gives you the right place to start looking; maybe it was some reasonable data, so you just handle it. Maybe it is unreasonable, so you need to look at where it came from, but usually (not always, of course) you just got the data and are pattern matching it, so you know where it came from too.

trav4225 4y ago

I've seen a lot of new developers shocked by this approach, which surprises me a little. They seem to think that it's up to the application to handle all errors, even those of the programmer(s). This, of course, is unreasonable since it would essentially require knowing all the bugs in advance. :-)

allenu 4y ago
I'm a big fan of the "crash early" strategy. I write in Swift primarily, and if I suspect a state is impossible to reach, I'll add a fatalError() so that in development, if it turns out I'm wrong, I spot it right away. (Something I learned from another dev I worked with, who was very productive.) Unfortunately, a lot of other devs hate to see that your code may actually crash and start asking questions about what scenario could cause it and asking if maybe there's a more gentle way to get out of the error. So, I'll often back down and start having softer error-handling, but on the whole it does complicate things further as the errors cascade and now you have to reason about handling combination of errors that have low likelihood of happening. So, to me, just having an early crash is way better.
- uoaei 4y ago
  > if I suspect a state is impossible to reach, I'll add a fatalError() Does Swift have assert statements? If so, is there a reason you chose this method instead?
  - allenu 4y ago
    Yes, Swift has assert statements. I tend to use them a lot as well, but in shipping code, they don't terminate the app. There are still some places where I'd prefer to terminate the app early rather than continue on. To be clear, I tend to use assert statements more than fatalErrors.
- aaaaaaaaaaab 4y ago
  Yeah. My pet peeve is `guard let … else { return }` instead of force-unwrapping. Like, why do people think that silently swallowing an error is better than crashing loudly and clearly?

akdor1154 4y ago

I'd be very interested to see non-BEAM approaches to enabling this - i kind of end up in the same pattern thanks to "expected? Return an Error<E>. Unexpected? Throw." However, the supervising part is then difficult. How do people approach this in Python? NodeJS? Rust? .NET?

jetbalsa 4y ago
Its common for me in PHP land to call die("Error! $e"); when something really bad has happened in a request and I really don't want anything to happen
- akdor1154 4y ago
  Yeah nice, PHP's request scoping is a really good fit for this.
- theluketowers 4y ago
  That's not without its drawbacks however, especially when you consider tools like swoole and laravel octane that keep your application in a long running process, so if you use die you actually kill one of the server workers instead of just terminating the execution of the request
vvanders 4y ago
You can take the same patterns and apply them, it just takes a little more rigor since beam/otp had this built in. My favorite is hardware (or software) watchdog timers. Very simple to implement and surprisingly effective.
game-of-throws 4y ago
If you can stomach kubernetes, you get this for free for all languages. Just panic!() or die(), and you'll get a fresh pod in a few seconds. Outside of k8s I try to use the lang's preferred tool. Python -> supervisord, Node -> pm2, etc.
andrewingram 4y ago
If you’re doing web stuff, frameworks will basically have this built in. Typically the framework’s own request handler will have something that catches all uncaught exceptions within the scope of that request and return a 500 response. So within the context of a single request you can usually adopt a “let it crash philosophy”. In fact, it’s something most people seem to do intuitively.

smackeyacky 4y ago

I don't agree with this approach. Say you have a network service that relies on other network services. It is not difficult to write those such that they know to back off / retry when something disappears. It's extremely useful in a lot of situations: if you do work on a laptop that gets regularly unplugged, having running test services that know to reconnect makes your life easier. In production, having things automatically reconnect means a lot less restarting of services once whatever root cause problem is corrected. Just shrugging and giving up ends up being a lot more work in the end. I like to tell junior developers to catch everything they can, and handle it or die as nicely as possible. Of course you can't plan for everything, but you can write around network and disk issues and issue warnings in a way that makes the root cause more obvious. That involves catching errors.

verdagon 4y ago
I'd imagine retrying/reconnecting is compatible with the general "let it fail" approach. If you just sent a message/request to an actor/server and it still hasn't responded after 5 seconds, you can send another. It wouldn't matter whether that actor/server died from a regular error or a "let it fail" error, the retrying would still work the same.
- _gabe_ 4y ago
  > If you just sent a message/request to an actor/server and it still hasn't responded after 5 seconds, you can send another. This depends on the message. I hope amazon doesn't just send another message if my transaction didn't complete in 5 seconds. I think like all pieces of wisdom, sometimes it's OK to let it fail, and sometimes its OK to handle the errors. If anyone ever tells me to always do X or never do X, it's typically not sound advice. The one thing we can always count on is generic advice failing sometimes :) (And even in this article, you shouldn't always try to handle known errors and never try to handle unknown errors, there will be exceptions)
  - ShroudedNight 4y ago
    > I hope amazon doesn't just send another message if my transaction didn't complete in 5 seconds. I do, but only while also associating a stable nonce to the transaction.

Mr_P 4y ago

For all the hate that Java tends to get, the language natively supports this distinction between: * Expected errors - Checked Exceptions * Unexpected errors - Unchecked Exceptions Idiomatic Java also makes heavy use of asserts, e.g. using the Guava Preconditions library.

josephcsible 4y ago
The problem is that while Java the language itself does support that distinction, a lot of built-in stuff really messes it up. For example, exceptions from closing a file are unexpected, but are an IOException which is checked anyway. Also, even the support that is in the language isn't first-class; e.g., lack of exception polymorphism.
- zmmmmm 4y ago
  I think that's a symptom of the fact that the distinction is really artificial at the language level anyway. Whether something is expected or not is a function of the requirements. Even OutOfMemory can be expected and handled in certain types of applications (esp. since it gets thrown for things like file handles rather than true memory). And then there are all kinds of cases where routine exceptions like file not found are in fact, unexpected errors (as discussed in TFA). Perhaps some sort of language level solution could have been found (eg: have explicit interfaces to mark exceptions as expected or unexpected and then exceptions are assigned that using generics or something), but that ship has sailed long ago.
  - rowls66 4y ago
    This is right, therefore, in most cases, a library should throw a checked exception, and the caller should decide whether it is an expected error and either handle it or rethrow it, or it is unexpected and rethrow a RuntimeException.
shepherdjerred 4y ago
Unchecked vs Checked is one of the things I like least about Java. Programmers tend to make everything Unchecked because it leads to easier code for API users at the cost of correctness/error handling.

verdagon 4y ago

I think that the "let it fail" approach is often inevitable, even when we try to use Result<T, E>. Often, we see an "unknown" variant in the error enum, as a catch-all for a library's unexpected errors. Then, anyone who calls them must also have an "unknown" enum. And anyone who calls them, and so on. In the end, this "unknown" variant is similar to a panic, in that there's very few reasonable reactions to it: Log it, cancel the request, return error 500, perhaps retry. For this reason, I often recommend people to just use assertions and panics.

hyperhopper 4y ago
While everything you said is correct, there are still significant advantages to the 'result' method. Sometimes you want to return 200 even if most of the backends fail. Sometimes one part may want to retry based on any error. Even aside from this, disallowing exceptions leads to a very predictable control flow, and makes program state able to be expressed in the type system, which is useful for many reasons on it's own. While yes, it's often just like an exception or panic, I'll take that over exceptions in my code any day
- verdagon 4y ago
  I realize I was ambiguous; I didn't mean to say "just use assertions and panics", I meant "just use assertions and panics for unexpected errors", my apologies. I wouldn't recommend someone only use Result<T, E> and never panic. If we do, then anything that might indirectly be invalid, such as a map lookup or an array index, will have ? operators on it, often every line of some functions. In the end, our control flow is just as unpredictable as if we just used panics and our signal is lost in the noise. For this reason, I think a blend of Result<T, E> and panics/assertions is really the way to go.

kbyatnal 4y ago

the best TLDR I've seen of this philosophy is - if you find yourself writing all over your codebase try { ... } catch (e) { console.log(e) } then you should probably just "let it fail" since you can't actually handle the error

deleted 4y ago
[deleted]
charcircuit 4y ago
That construction adds a layer of "let it fail." For example you might want to have your web server be able to mark a connection as failed instead of only being able to mark the entire server as having failed.
anupamchugh 4y ago
That's impressive!

rektide 4y ago

I spent a good part of this week overhauling a microservice where most fucntions were a giant try/catch & would maybe throw a new error. Just getting rid of the try catches & letting the code fail has been a huge help in seeing what is going wrong as the code executes. I also am delighted to see the idea of expected errors here. Another thing I've been doing for a long time is tagging erros with an expected = true property when it's something we expect to see, like, oh, we went to get this oauth token but the credentials were wrong. Expectedness shows up in thr logs now & we can see mych more clearly where there are real problems.

ramraj07 4y ago

A corollary or generalized interpretation of this approach (and someone please specify if there’s a formal term for this) is: “fail locally, and immediately.” What I mean is that once something unexpected happens your code should ideally fail in that step itself. The simplest most common example I’ve seen with python programmers is when they pass around dicts as arguments in complex code bases. Methods expect various keys to be present, and often methods also have fail safe defaults if some keys are absent. The defaults are written for the specification, sure, but often they also tolerate unexpected exceptions that happened upstream. Now when an unexpected exception happens, your program fails somewhere else and the stacktrace is useless. The only way to figure out what went wrong is to debug it line by line. With python there’s still no elegant solution. I’m now trying to ensure all my methods are typed and use dataclasses and pydantic classes to type and group these parameters but there’s still opportunities for these “fail later” errors. Solutions and suggestions would be appreciated!

bzxcvbn 4y ago
>Solutions and suggestions would be appreciated! Use a language with strong typing?
- aunderscored 4y ago
  Python is strongly typed. You want statically typed. (Instead of duck typed / dynamically typed)
  - jahewson 4y ago
    FWIW “strong” typing is a colloquial term and lacks any precise meaning.
  - bzxcvbn 4y ago
    Can you guess what this code does? class foo: pass obj = foo() obj.bar = "I thought Python was strongly typed?" print(obj.bar) And even better: class foo: a = 42 obj = foo() print(obj.a) del foo.a print(obj.a) Whatever your opinion on what the imprecise sentence "strongly typed language" should mean, these are definitely not features of one.

robocat 4y ago

The article doesn’t seem to look at how resources are cleaned up when a BEAM process crashes. https://elixirforum.com/t/understanding-the-advantages-of-let-it-crash-term/9748/16 https://elixirforum.com/t/understanding-the-advantages-of-le... says “All resources are owned by a process in Erlang, and the VM guarantees clean-up of resources once the process dies”. My Google-fu failed me when I searched for more details about Erlang process cleanup of resources, or how to register cleanup actions (e.g. delete some temporary file on crash).

btkostner 4y ago
I’d take a look at the terminate callback for Elixir GenServers. https://hexdocs.pm/elixir/1.12/GenServer.html#c:terminate/2 https://hexdocs.pm/elixir/1.12/GenServer.html#c:terminate/2
jeanlucas 4y ago
I am not sure what you mean by that. Which resources do you mean? Everything happening in your program is running inside the Erlang VM, even file writing... And each process inside the VM does their own garbage collection. The only way I could see the VM itself crashing is if it can't allocate memory, then it all crashes.
- robocat 4y ago
  Let’s assume we are using Linux. Firstly, an Erlang VM process is not a Linux child process: “Erlang processes are lightweight, operate in (memory) isolation from other processes, and are scheduled by Erlang's Virtual Machine (VM). The creation time of process is very low, the memory footprint of a just spawned process is very small, and a single Erlang VM can have millions of processes running.”. An Erlang program can use kernel resources. Objects related to file descriptors, filesystem data, the process namespace, and signal handlers. Kernel timers, semaphores, memory allocations (e.g. for FFI), sockets, DMA, etcetera. I presume the Erlang library code tracks resources (such as file access function tracking file descriptors per Erlang process), so they can be cleaned up if we “let-it-fail”. I also presume there is a canonical way to register to clean up resources - the equivalent to atexit()[1]. [1] https://man7.org/linux/man-pages/man3/atexit.3.html https://man7.org/linux/man-pages/man3/atexit.3.html

WalterBright 4y ago

I learned from working on aviation systems is that when a system enters an unknown state, it must be disabled and locked out. In software, this is known as an assertion failure. When the assert trips, the program is, by definition, in an unknown state. A program cannot reasonably be allowed to continue in an unknown state - it may launch nuclear missiles. The only thing to be done is exit directly, do not pass Go, do not collect $200.

glouwbug 4y ago
What if the plane is mid flight?
- bicx 4y ago
  It won’t be in mid-flight for much longer.
- WalterBright 4y ago
  Engage the backup. Everything flight critical is dual.
mmcnl 4y ago
I like this mindset.
roeles 4y ago
Thanks for posting this. I have worked on non critical flight software and thought that this philosophy might work well. I wonder how easy the certification is for such software? For work I might have to write Do178 code in the future.
- WalterBright 4y ago
  I use it in the software I write. I should do a presentation sometime about how the aviation industry should be influencing software development.
  - 4y ago

winter_blue 4y ago

In a new language, I'd like to see exceptions being allowed in pure code, but prohibited in non-pure code. (Non-pure here meaning code with side effects.) In pure code, an exception could essentially be passed up, and transformed into an error return value at the point where it's called by non-pure code.

mike_hock 4y ago
So then you make a pure function `throw_exception` and now you can throw from impure code.
- winter_blue 4y ago
  In non-pure code, we would want to force the programmer to handle the exception. In this case, wherever `throw_exception` is called, the programmer would have to handle the exception, and either refactor the return value to indicate an error, change the return value to an Optional (and silently fail by returning an empty Optional), or cause/trigger a side effect (like terminating the thread/program termination early, with an exit code).

jeanlucas 4y ago

That's why Erlang is cool, to me.

angarg12 4y ago

One of the pieces of software I'm most proud of is a service to manage the dynamic part of our infrastructure. It uses control theory and let it fail to great effect. The service reads the state of the system, and applies change to converge to a configured policy. If it encounter an error, it doesn't try to handle or fix it, it just fails and logs a fine grained metric, plus a general error metric. The system fails all the time at this scale, but heals itself pretty quickly. In over 1 year of operation it hasn't caused a single incident, and it has survived all outages.

kortilla 4y ago
This is just kubernetes right? Declarative desired state model. Containers created and destroyed to get there. Crashes happen, metrics are incremented, load balancers route around the crashing pod until they recover (or are replaced), etc.
arunaugustine 4y ago
Yes, please give us more info about using control theory and how one might think about building such a system please..
- wsc981 4y ago
  My guess would be add assertions everywhere instead of throwing exceptions.
  - Karellen 4y ago
    Why not throw exceptions, and just never use try/catch? That way, all exceptions are uncaught and should terminate the program, in a way that takes advantage of the programming language's native error reporting facilities.
    - hsn915 4y ago
      assertion failures terminate the program immediately. exceptions usually trigger cleanup code. If your cleanup code is mostly closing files and clearing memory, then it's useless because the OS will do that for your crashed program anyway.

at_a_remove 4y ago

I struggle to find the correct descriptor for a counter-example, wherein You Really Want Success for the process as a whole, but it is acceptable for a sliver of it to fail, in the the context of ETL. I have an ETL I am told (I switched jobs) that is still working, from 2008. It was built to be a tank, and I also did another forbidden thing: Pokemon Exception Handling. It's a guideline, not a law of physics, and it is fine to resort to a general error catch when you really don't know every possible error (and let's be honest, if you have enough libraries in the mix, some surprises will happen) and you want the other 99.999% of the data to go through. Yes, this one little thing didn't load, and let's log that, let's examine that and figure out how to prevent that going forward, but overall, the rest of the program must continue. How did it get so tanklike? Every time a little bit failed and it got logged, I figured out what went wrong, fixed it, and then tried to generalize a class of similar errors. After a while, I got into Things I Was Told Would Never Happen in the data we ingested and programmed for when never happened. Reader, never came a little sooner than expected. Anyway, I largely agree with the idea but there are places where you want the exact opposite, and I think it is important to look for those places lest this heuristic become so stiff it can lose utility.

e9 4y ago
Funny thing, I had same experience. I also built robust ETL. It was for ingesting and manipulating financial data from 30+ different banks and I also did Pokemon Exception Handling to make it robust. In general my philosophy is: I don't want to wake up at 4am unless it's urgent. What can I do to gracefully handle failures to achieve that goal?
- at_a_remove 4y ago
  Exactly, thank you. At the time I was transitioning from a jack of all trades person to strictly a programmer. I despise getting called in the middle of the night. My feeling is that, if you have a foundational business process like this, it should be designed to be maintained if there is a serious problem, and it ought to keep working. I know, haha, "the Internet is a series of tubes" but I really wanted this thing to be like a chunk of very uninteresting ductwork that just moves air from one place to another: it should just do its job with as much fanfare.

czei002 4y ago

Other frameworks like express (nodejs) or actix (Rust) also don't crash if you "throw" in an request handler so this doesn't sounds very exciting to me. The interesting question for me is how retries are handled after an error occurred? For example, if the error happens in an http request handler, does the request still fails with 500 or is it magically retried by Erlang while keeping the request hanging? For internal service calls how are retries working? i.e. how can I configure that a request is retried after a failure? I guess Erlang does this and this is the power behind it? The example of a missing file seems not very good since its a problem that is probably not solved by waiting. A better example is probably a busy DB that is temporary not reachable?

jon-wood 4y ago
The Erlang VM is built around message passing, so in the case of a dropped database connection your application code would pass a message to the database API asking for the results of a SQL query. The database API is currently trying to reconnect, so it’s not processing that queue of messages, but once it gets there it’ll pick up the message, run the query, and then send one back to the process that asked. This is all largely transparent to your application code, beyond being able to set some preferences in your return message handler around things like how long you’re willing to wait. The whole “let it crash” thing comes from Erlang’s process supervision - in practice the DB API isn’t actually retrying. It’s continually failing to connect and if it can’t the process just crashes. The supervisor then notices and starts a new process in its place, this continues until either a process successfully starts or the configured retry count is hit. If the retry count is exceeded then the supervisor crashes, either taking the entire application with it, or more commonly being restarted itself by the next supervisor up the tree.
jswny 4y ago
The point is that Erlang (the BEAM VM) follows this pattern everywhere, not just for web requests. It’s baked into the language and runtime, and that’s infinitely more powerful and customizable than not crashing in a request handler.

kgeist 4y ago

>ignore unexpected exceptions Isn't it the way it already is in practice, not something specific to Erlang? If an exception is unexpected, usually there won't be an exception handler for it, otherwise a developer pretty much expected it. Developers are generally lazy so in my practice the default is usually to let it fail, and there's usually going to be an exception handler that does something other than logging and quitting only if there's a serious reason to do so. Maybe a more useful distinction could rather be "business logic errors" vs. everything else ("infrastructure errors", "programming errors" and "input validation errors"). Business logic should clearly define what should be done when an error happens, to avoid inconsistent state. But infrastructure-level errors or programming errors, you can't do much about them, other than log and/or retry.

gabeio 4y ago
> If an exception is unexpected, usually there won't be an exception handler for it That honestly depends on how the language and program are written, python is a great and horrible example of where you can handle any exception even ones that were just created by the program: try: crash_hard_here() except: # by default (and unfortunately) will catch *everything* pass # and this is one of the worst offenders inside an except, to just outright ignore the exception and continue as if nothing happened and to not even log it. I can not tell you the amount of production code where I've seen catch all exceptions, and they are the lazy way to know something will not "crash" even though much worse things can happen now.
- blub 4y ago
  This catch and ignore seems to be straight out of the (sadly not real) Visual Basic design patterns book :-)
blub 4y ago
In C++ for example there’s an exception hierarchy and several classes of errors similar to what you enumerated, so the handler might catch a specific exception as a more generic one. There’s also … which catches all exceptions. This is fine, as one has the choice of exposing error details to outside code or not.

kaczordon 4y ago

I believe this style of coding is known as “Design by contract”. Basically assigning a range and domain to each function.

mewse 4y ago

I was a believer in the "my code should never crash, no matter what" school of thought until I shipped a Dreamcast game with an out-of-date opening cutscene. It was an in-engine opening cutscene which was very nearly final; the file we shipped was about two or three weeks out of date compared against the version that should have gone on the disc (It had one missing shape key on a character's face at the end of a shot, and a couple other missing elements). My code was wrangling the whole animation; doing all the stuff which our at-the-time-primitive animation system couldn't do itself (animating texture coordinates and etc). And my code was just silently handling all the errors it ran into so that we never even noticed that anything was wrong. The difference was subtle enough that in the twelve years since the game was released, nobody but the original animator has ever noticed and mentioned it to me (and that, years after release). But that one experience and knowing how much worse it could have been was enough to convince me that "crashes early and crashes loudly with as much detail as possible" is by far the better strategy. At least for entertainment products. And doubly so for entertainment products which can't be patched after release. (for clarity, this screw-up was 100% my fault. The animators had made the final changes to the cutscene data files in plenty of time for inclusion in the final build, I just somehow didn't import the changed data files into the game when I made the matching changes to the code side, and then my code didn't throw any errors to tell me or anyone else on the project that anything was wrong.)

mike_hock 4y ago
Or even better, it should be the latter during development and the former in the released version. You don't want your released game to crash in level 11 if the player happens to look behind the wrong lightpole because a texture is missing, but you do want to notice that in development.
- XorNot 4y ago
  I remember back when I messed with D3D (around 8?) I was surprised to learn this is how the Debug build mode of it works. In Debug you can completely screw up your pipeline and scene handling and...everything will work just fine. Or appear to. Switch to a Release build and all those failures are suddenly very obvious - and this is almost exactly what you wouldn't want.

mpweiher 4y ago

Dataflow offers an interesting variation of not polluting the happy path with error handling: just do nothing. How so? Well, the happy path passes data to the next filter. In the case of an error, simply do not do that, and the next filter will be none-the-wiser. Optionally, log the error, possibly with enough information that the "logger" could retry the operation if appropriate. But as others have pointed out, there often isn't much that can be done.

mkl95 4y ago

One of the issues I take with this mentality is that companies are always looking for excuses to sell bad software. As an engineer, I really don't enjoy maintaining crappy SaaS, particularly if they are built with weakly typed, dynamic languages. If something is obviously prone to fail you should empower people to do something about it, because usually "letting it fail" will lead to your users switching to your competitor's offering.

smallerfish 4y ago

I've heard this expressed as "write brittle code", and I'm a strong advocate for it. Looking up a user by id, and get no results? Rather than either passing null up the stack, or wrapping null with an optional.empty, throw an exception! It's the client code's problem if it somehow got hold of an id that doesn't exist. (Yes, ymmv depending on the system, e.g. if you're dealing with eventual consistency then maybe do something different.) As the article says, this of course doesn't mean that you shouldn't handle user errors, or even known system errors.

torginus 4y ago

The article mentions Erlang, a functional language - which gives an interesting contrast - it is a functional language which are all about mapping out all behavior so that no undefined behavior can exist, and basically force you to consider every possibility (of course that doesn't account for stuff like network errors). Wouldn't the same scheme be better suited for a procedural language, with deliberately dirty code full of gotchas? I write a ton of hacky scripts, like last time I needed to rewrite 1000s of xml-s I wrote a crude regex replace for it. It worked 99% of the time, and I fixed up the rest manually. This sort of thinking - that a subprocess might fail due to whatever reason, including sloppy code - but the whole process will keep trucking on would be perfect for the this paradigm. Additionally this would open the path for stuff like trivial hot code replacement - since in this system, a subprocess that crashes every time - like an invalid program would be handled by the system.

wolffiex 4y ago

This is the way. Exception handling is often one of the worst aspect of a production codebase, especially since it is typically added late. Though error handling strategies benefit from careful design, they are usually added piecemeal. Making errors louder and more problematic is the best way to get them the attention they deserve. This is not a new concept and it seems to be one of the core components of the Erlang Weltanschauung. It can be generalized further to systems as the principle of "Crash-Only Software," as advanced in this classic paper: https://dslab.epfl.ch/pubs/crashonly.pdf https://dslab.epfl.ch/pubs/crashonly.pdf