3 ms·

Re: panics: If you have a single long lived process that must do multiple short-lived things (web requests, say) and a panic in one of them MUST NOT take down t

Re: panics: If you have a single long lived process that must do multiple short-lived things (web requests, say) and a panic in one of them MUST NOT take down the whole process, is that extremely difficult to pull off in Rust? I thought you could set up panic boundaries much like you would use catch-all exception handlers around e.g. each web request or similar, in other languages?

sph 2y ago
Since Rust is not a managed/high-level language, panics are unrecoverable crashes so they need to be dealt at a higher-level, i.e the OS, with appropriate supervisor systems like systemd, or having a master Rust process that spawns subprocesses, and react when one of them abnormally terminates with regular POSIX APIs. On a platform like Elixir, for example, you can deal with process crashes because everything runs on top of a VM, which is at all effects and purposes your OS, and provides process supervision APIs.
- tuetuopay 2y ago
  This is factually incorrect. The behavior you describe with Elixir (sic) is precisely what most Rust async runtimes do. (sic because it's Erlang that's to thank) IMHO that is the sensible thing to do for pretty much any green thread or highly concurrent application. e.g. Golang does the same: panicking will only bring down the goroutine and not the whole process.
- pornel 2y ago
  Rust can be optionally compiled in a panic=abort mode, but by default panics are recoverable. From implementation perspective Rust panics are almost identical to C++ exceptions. For servers that must not suddenly die, it's wise to use panic=unwind and catch_unwind at task/request boundaries (https://doc.rust-lang.org/stable/std/panic/fn.catch_unwind.html https://doc.rust-lang.org/stable/std/panic/fn.catch_unwind.h...) In very early pre-1.0 prototypes Rust was meant to have isolated tasks that are killed on panic. As Rust became more low-level, it turned into terminating a whole OS thread on panic, and since Rust 1.9.0, it's basically just a try/catch with usage guidelines.
- alkonaut 2y ago
  But no few would write a process-per-request web server today for example. And if a single process web server handles 100 requests you would then accept that one bad request one tore down the handling of the 99 others. Even if you have a watchdog that restarts the service after the one request choked, you wouldn't save the 99 requests that were in-flight on the same process. Can't you catch_unwind for each request handler, if one chokes then you just ignore that request. If you worry about that messing anything up, then you can tear down and restart your process after that, so the 99 other requests get a chance to complete?
tuetuopay 2y ago
You can install a global panic handler to avoid bringing the whole process down. Instead of aborting, take the stack trace, print it, perhaps raise to sentry, and kill the specific "work unit" that caused it. This "work unit" can be a thread of a task, depending on how the application is architected. This is precisely what Tokio does: by default, a panic in async code will only bring down the task that panicked instead of the whole application. In the context of a server, where you'll spawn a task for each request, you have no way to bring down the whole application (), only your current scope. (): there could be other issues, like mutex poisoning, which is why nobody uses the stdlib's mutexes. But the general point still stands.
- EE84M3i 2y ago
  > there could be other issues, like mutex poisoning, which is why nobody uses the stdlib's mutexes. What does everyone use instead?
  - tuetuopay 2y ago
    In the context of Tokio, the tokio's native mutexes / locking primitives. For sync code, parking_lot is the de facto replacement for the stdlib's ones. I don't remember where I read it, but it has been admitted that having synchronization primitives with poisoning in the stdlib was a mistake, and "simpler" ones without it. for context: a mutex is poisoned should a panic occur while the mutex is held. it is then assumed the guarded data to be broken or in an unknown state, thus poisoned.
  - selfmodruntime 2y ago
    parking_lot

sph 2y ago

Since Rust is not a managed/high-level language, panics are unrecoverable crashes so they need to be dealt at a higher-level, i.e the OS, with appropriate supervisor systems like systemd, or having a master Rust process that spawns subprocesses, and react when one of them abnormally terminates with regular POSIX APIs. On a platform like Elixir, for example, you can deal with process crashes because everything runs on top of a VM, which is at all effects and purposes your OS, and provides process supervision APIs.

tuetuopay 2y ago
This is factually incorrect. The behavior you describe with Elixir (sic) is precisely what most Rust async runtimes do. (sic because it's Erlang that's to thank) IMHO that is the sensible thing to do for pretty much any green thread or highly concurrent application. e.g. Golang does the same: panicking will only bring down the goroutine and not the whole process.
pornel 2y ago
Rust can be optionally compiled in a panic=abort mode, but by default panics are recoverable. From implementation perspective Rust panics are almost identical to C++ exceptions. For servers that must not suddenly die, it's wise to use panic=unwind and catch_unwind at task/request boundaries (https://doc.rust-lang.org/stable/std/panic/fn.catch_unwind.html https://doc.rust-lang.org/stable/std/panic/fn.catch_unwind.h...) In very early pre-1.0 prototypes Rust was meant to have isolated tasks that are killed on panic. As Rust became more low-level, it turned into terminating a whole OS thread on panic, and since Rust 1.9.0, it's basically just a try/catch with usage guidelines.
alkonaut 2y ago
But no few would write a process-per-request web server today for example. And if a single process web server handles 100 requests you would then accept that one bad request one tore down the handling of the 99 others. Even if you have a watchdog that restarts the service after the one request choked, you wouldn't save the 99 requests that were in-flight on the same process. Can't you catch_unwind for each request handler, if one chokes then you just ignore that request. If you worry about that messing anything up, then you can tear down and restart your process after that, so the 99 other requests get a chance to complete?

tuetuopay 2y ago

You can install a global panic handler to avoid bringing the whole process down. Instead of aborting, take the stack trace, print it, perhaps raise to sentry, and kill the specific "work unit" that caused it. This "work unit" can be a thread of a task, depending on how the application is architected. This is precisely what Tokio does: by default, a panic in async code will only bring down the task that panicked instead of the whole application. In the context of a server, where you'll spawn a task for each request, you have no way to bring down the whole application (), only your current scope. (): there could be other issues, like mutex poisoning, which is why nobody uses the stdlib's mutexes. But the general point still stands.

EE84M3i 2y ago
> there could be other issues, like mutex poisoning, which is why nobody uses the stdlib's mutexes. What does everyone use instead?
- tuetuopay 2y ago
  In the context of Tokio, the tokio's native mutexes / locking primitives. For sync code, parking_lot is the de facto replacement for the stdlib's ones. I don't remember where I read it, but it has been admitted that having synchronization primitives with poisoning in the stdlib was a mistake, and "simpler" ones without it. for context: a mutex is poisoned should a panic occur while the mutex is held. it is then assumed the guarded data to be broken or in an unknown state, thus poisoned.
- selfmodruntime 2y ago
  parking_lot