6 ms·

> There are situations in which if you detect errors you want the program to continue running, and have only that particular functionality to fail. Yes, like a

> There are situations in which if you detect errors you want the program to continue running, and have only that particular functionality to fail. Yes, like a web server. If a request handler fails by panicking, in a Rust program, you catch the panic, respond with a 500 error and log the panic somewhere. But you continue serving other requests. I talked about this in the blog post. The problem with your strategy is that it requires you to be aware of your own mistakes. That doesn't sound like a robust strategy, unless you're investing huge resources into sophisticated tooling and have drastically restricted the expressivity of your programming environment. That exists and is fine, and I even addressed that in the blog post too.

marshray 4y ago
Great article BTW, loved it! Will certainly become a classic. The web server example scares me. Something happened during execution that the programmer didn't expect. There's a bug in the program. What if the panic is due to memory corruption (less likely in Rust) or internal data structure corruption? Without knowledge to the contrary, swallowing a panic and YOLO'ing execution is driving full speed down the road of very poorly defined behavior. If the programmer had sufficient knowledge to conclude it was safe, they could have just used a Result<> to report the error. So ... don't make panic part of your API, and don't recover from panics?
- burntsushi 4y ago
  Doesn't scare me in a memory safe language. Request handlers tend to be very well isolated. It's standard practice for web servers written in Go too. > So ... don't make panic part of your API, and don't recover from panics? That's the wrong lesson. As my blog says, don't use panicking for error handling. Making panicking part of the API is standard practice. Otherwise `slice[i]` wouldn't exist. (Its API is to panic when `i` is out of bounds.) Lest you think you should just never panic and convert everything into errors, that's addressed in the blog too. :)
  - marshray 4y ago
    As the sibling post said: > I mean, if all bets are off, all bets are truly off. We have to expect that panic (e.g. from slice index operator) exercised an execution path that the library developer had never even considered, much less validated. So then, post-panic, are we left with any guarantees about the validity of postconditions?
    - burntsushi 4y ago
      > So then, post-panic, are we left with any guarantees about the validity of postconditions? Yes, because in order to catch a panic, the routine you're running has to be unwind-safe: https://doc.rust-lang.org/std/panic/trait.UnwindSafe.html https://doc.rust-lang.org/std/panic/trait.UnwindSafe.html If you want to go on crashing your process and restarting it, then that's fine. It's a fine strategy. But you can't ignore that request handlers tend to be very well isolated, and catching a panic from one of them while continuing to service requests actually works well in practice. Oodles of folks are doing this in production. I've been doing it production for years. I have literally, not once, not ever, seen any problems arise because of this. Consider what happens if there's a 'slice[i]' that panics in a regex library. The stack unwinds and any objects associated with that specific search that panicking get dropped. And since 'Regex' implements 'UnwindSafe', it says, "If a panic occurs it's all good to continue using me if you catch the panic." You keep wanting to argue the theory. Rust has UnwindSafe for that. But please, consider looking at the practice too. You don't seem to have acknowledged that at all.
- ninkendo 4y ago
  > Without knowledge to the contrary, swallowing a panic and YOLO'ing execution is driving full speed down the road of very poorly defined behavior. I mean, if you’re worried about code “corrupting” things such that it may not be safe to even serve the next request, you probably need to worry about successful requests as well. I mean, if all bets are off, all bets are truly off. At some point, if you decide to colocate code from multiple “apps” in one server, you gotta have some level of trust in said apps. You’d obviously never do this for untrusted code. And you’d never do this for code that (hand-waving as you have) “corrupts” things in such a way that it’s not safe to handle the next request, panic or no.
  - marshray 4y ago
    > You’d obviously never do this for untrusted code. Doesn't panic imply that the program entered a state from which it was never explicitly designed to recover? Since that state may persist across invocations of a function, doesn't that make the code effectively 'untrusted' for subsequent execution?
    - ninkendo 4y ago
      > Doesn't panic imply that the program entered a state from which it was never explicitly designed to recover? No? I think you’re making a lot of assumptions about the “state” of a program. Panicking doesn’t mean “I maintain state in between requests and that state is now poison and I can’t be called again”. In fact it doesn’t mean anything really. It means someone wrote a code path that panics, no more, no less. Sure it’s possible that code can be written where a panic indicates that there is some poison state and now nothing can be trusted etc. But… the same is possible for code that doesn’t panic, so I don’t really know where we’re going with this discussion.
      - lmm 4y ago
        Well, shouldn't there be a standard way to convey that, that is understood (as a matter of culture) to mean that? And if it's not panic then what is it?
- toast0 4y ago
  What you're looking for is isolation. In a classic forking webserver, you get isolation from the OS; if your request panics, let that process die, and the main process will respawn it; further requests are isolated from the failure (although, if the request paniced after writing broken shared state outside the isolated process, who knows if further requests will work). If the part that paniced isn't isolated, you really should propagate the panic until the panicing part is isolated. But maybe that's the OTP/Erlang talking ;)
  - preseinger 4y ago
    OS processes are far too heavyweight to serve individual requests. OS _threads_ are far too heavyweight, for that matter.
    - lmm 4y ago
      That's actually kind of a myth nowadays. For example the honest answer to the classic C10K question (how do you write a webserver that can serve 10000 requests concurrently) is that you spawn 10k threads and let the OS scheduler handle it and it'll be fine.
      - preseinger 4y ago
        Even a modern Linux kernel on modern hardware will struggle with 10k processes if those processes are doing nontrivial (i.e. syscall-effecting) work. But that benchmark is what, 10 or 15 years old now? Contemporary benchmarks target 100k connections per server (with some constraints).
        int_19h 4y ago
        The question, rather, is whether it'll struggle with 10k threads.
        preseinger 4y ago
        Aye, the cost of context switching between OS threads is generally too high for thread-per-request workloads.
- lmm 4y ago
  > So ... don't make panic part of your API, and don't recover from panics? Don't recover from panic except at high level - retrying or failing that "scope" is ok, but don't try to do business logic around panicing or not (if you absolutely must, catch the panic at the lowest possible level and turn it into a result).
Tainnor 4y ago
> The problem with your strategy is that it requires you to be aware of your own mistakes. That doesn't sound like a robust strategy, unless you're investing huge resources into sophisticated tooling and have drastically restricted the expressivity of your programming environment. That exists and is fine, and I even addressed that in the blog post too. Those "huge resources" and "sophisticated tooling" could just mean something like Erlang, where fault isolation is a core design principle. But fault isolation works in simpler systems, too. Here's a recent example: I wrote a function that automatically generates a table of contents for an HTML document. While extensively testing it, I ran across a number of edge cases that would lead to faults (mostly due to messy underlying libraries, but also due to programmer error). I fixed all that I found, but I still didn't feel comfortable not adding a generic exception handler to the function. In case that an exception is ever thrown in that code, the function will just return the input HTML unchanged (and log an error, of course). That's graceful degradation, and a good example of recovering from a potentially unknown fault. Similar situations can often be found in many applications. Not all functions are essential, and we're not always 100% confident about all parts of the code working correctly. (Note that I'm not talking about Rust here, I lack sufficient knowledge about it. It's possible that this strategy wouldn't work in Rust for reasons of memory-safety or anything else.)

marshray 4y ago

Great article BTW, loved it! Will certainly become a classic. The web server example scares me. Something happened during execution that the programmer didn't expect. There's a bug in the program. What if the panic is due to memory corruption (less likely in Rust) or internal data structure corruption? Without knowledge to the contrary, swallowing a panic and YOLO'ing execution is driving full speed down the road of very poorly defined behavior. If the programmer had sufficient knowledge to conclude it was safe, they could have just used a Result<> to report the error. So ... don't make panic part of your API, and don't recover from panics?

burntsushi 4y ago
Doesn't scare me in a memory safe language. Request handlers tend to be very well isolated. It's standard practice for web servers written in Go too. > So ... don't make panic part of your API, and don't recover from panics? That's the wrong lesson. As my blog says, don't use panicking for error handling. Making panicking part of the API is standard practice. Otherwise `slice[i]` wouldn't exist. (Its API is to panic when `i` is out of bounds.) Lest you think you should just never panic and convert everything into errors, that's addressed in the blog too. :)
- marshray 4y ago
  As the sibling post said: > I mean, if all bets are off, all bets are truly off. We have to expect that panic (e.g. from slice index operator) exercised an execution path that the library developer had never even considered, much less validated. So then, post-panic, are we left with any guarantees about the validity of postconditions?
  - burntsushi 4y ago
    > So then, post-panic, are we left with any guarantees about the validity of postconditions? Yes, because in order to catch a panic, the routine you're running has to be unwind-safe: https://doc.rust-lang.org/std/panic/trait.UnwindSafe.html https://doc.rust-lang.org/std/panic/trait.UnwindSafe.html If you want to go on crashing your process and restarting it, then that's fine. It's a fine strategy. But you can't ignore that request handlers tend to be very well isolated, and catching a panic from one of them while continuing to service requests actually works well in practice. Oodles of folks are doing this in production. I've been doing it production for years. I have literally, not once, not ever, seen any problems arise because of this. Consider what happens if there's a 'slice[i]' that panics in a regex library. The stack unwinds and any objects associated with that specific search that panicking get dropped. And since 'Regex' implements 'UnwindSafe', it says, "If a panic occurs it's all good to continue using me if you catch the panic." You keep wanting to argue the theory. Rust has UnwindSafe for that. But please, consider looking at the practice too. You don't seem to have acknowledged that at all.
ninkendo 4y ago
> Without knowledge to the contrary, swallowing a panic and YOLO'ing execution is driving full speed down the road of very poorly defined behavior. I mean, if you’re worried about code “corrupting” things such that it may not be safe to even serve the next request, you probably need to worry about successful requests as well. I mean, if all bets are off, all bets are truly off. At some point, if you decide to colocate code from multiple “apps” in one server, you gotta have some level of trust in said apps. You’d obviously never do this for untrusted code. And you’d never do this for code that (hand-waving as you have) “corrupts” things in such a way that it’s not safe to handle the next request, panic or no.
- marshray 4y ago
  > You’d obviously never do this for untrusted code. Doesn't panic imply that the program entered a state from which it was never explicitly designed to recover? Since that state may persist across invocations of a function, doesn't that make the code effectively 'untrusted' for subsequent execution?
  - ninkendo 4y ago
    > Doesn't panic imply that the program entered a state from which it was never explicitly designed to recover? No? I think you’re making a lot of assumptions about the “state” of a program. Panicking doesn’t mean “I maintain state in between requests and that state is now poison and I can’t be called again”. In fact it doesn’t mean anything really. It means someone wrote a code path that panics, no more, no less. Sure it’s possible that code can be written where a panic indicates that there is some poison state and now nothing can be trusted etc. But… the same is possible for code that doesn’t panic, so I don’t really know where we’re going with this discussion.
    - lmm 4y ago
      Well, shouldn't there be a standard way to convey that, that is understood (as a matter of culture) to mean that? And if it's not panic then what is it?
toast0 4y ago
What you're looking for is isolation. In a classic forking webserver, you get isolation from the OS; if your request panics, let that process die, and the main process will respawn it; further requests are isolated from the failure (although, if the request paniced after writing broken shared state outside the isolated process, who knows if further requests will work). If the part that paniced isn't isolated, you really should propagate the panic until the panicing part is isolated. But maybe that's the OTP/Erlang talking ;)
- preseinger 4y ago
  OS processes are far too heavyweight to serve individual requests. OS _threads_ are far too heavyweight, for that matter.
  - lmm 4y ago
    That's actually kind of a myth nowadays. For example the honest answer to the classic C10K question (how do you write a webserver that can serve 10000 requests concurrently) is that you spawn 10k threads and let the OS scheduler handle it and it'll be fine.
    - preseinger 4y ago
      Even a modern Linux kernel on modern hardware will struggle with 10k processes if those processes are doing nontrivial (i.e. syscall-effecting) work. But that benchmark is what, 10 or 15 years old now? Contemporary benchmarks target 100k connections per server (with some constraints).
      - int_19h 4y ago
        The question, rather, is whether it'll struggle with 10k threads.
        preseinger 4y ago
        Aye, the cost of context switching between OS threads is generally too high for thread-per-request workloads.
lmm 4y ago
> So ... don't make panic part of your API, and don't recover from panics? Don't recover from panic except at high level - retrying or failing that "scope" is ok, but don't try to do business logic around panicing or not (if you absolutely must, catch the panic at the lowest possible level and turn it into a result).

Tainnor 4y ago

> The problem with your strategy is that it requires you to be aware of your own mistakes. That doesn't sound like a robust strategy, unless you're investing huge resources into sophisticated tooling and have drastically restricted the expressivity of your programming environment. That exists and is fine, and I even addressed that in the blog post too. Those "huge resources" and "sophisticated tooling" could just mean something like Erlang, where fault isolation is a core design principle. But fault isolation works in simpler systems, too. Here's a recent example: I wrote a function that automatically generates a table of contents for an HTML document. While extensively testing it, I ran across a number of edge cases that would lead to faults (mostly due to messy underlying libraries, but also due to programmer error). I fixed all that I found, but I still didn't feel comfortable not adding a generic exception handler to the function. In case that an exception is ever thrown in that code, the function will just return the input HTML unchanged (and log an error, of course). That's graceful degradation, and a good example of recovering from a potentially unknown fault. Similar situations can often be found in many applications. Not all functions are essential, and we're not always 100% confident about all parts of the code working correctly. (Note that I'm not talking about Rust here, I lack sufficient knowledge about it. It's possible that this strategy wouldn't work in Rust for reasons of memory-safety or anything else.)