6 ms·

> Poisoning: Panic Safety in Mutexes This is one of the biggest design flaws in Rust's std, in my opinion. Poisoning mutexes can have its use, but it's very r

> Poisoning: Panic Safety in Mutexes This is one of the biggest design flaws in Rust's std, in my opinion. Poisoning mutexes can have its use, but it's very rare in practice. Usually it's a huge misfeature that only introduces problems. More often than not panicking in a critical section is fine[1], but on the other hand poisoning a Mutex is a very convenient avenue for a denial-of-service attack, since a poisoned Mutex will just completely brick a given critical section. I'm not saying such a project doesn't exist, but I don't think I've ever seen a project which does anything sensible with Mutex's `Poisoned` error besides ignoring it. It's always either an `unwrap` (and we know how well that can go [2]), or do the sensible thing and do this ridiculous song-and-dance: let guard = match mutex { Ok(guard) => guard, Err(poisoned) => poisoned.into_inner() }; Suffice to say, it's a pain. So in a lot of projects when I need a mutex I just add `parking_lot`, because its performance is stellar, and it doesn't have the poisoning insanity to deal with. [1] -- obviously it depends on a case-by-case basis, but if you're using such a low level primitive you should know what you're doing [2] -- https://blog.cloudflare.com/18-november-2025-outage/#memory-preallocation https://blog.cloudflare.com/18-november-2025-outage/#memory-...

thayne 11mo ago
There are cases where it is useful. I had a case where if the mutex was poisened it was possible to reset the lock to a safe state (by writing a new value to the locked content). Or you may want to drop some resource or restart some operation instead of panicing if it is poisoned. But I agree that the default behavior should be that the user doesn't have to worry about it.
LegionMammal978 11mo ago
> It's always either an `unwrap` (and we know how well that can go [2]) If a mutex has been poisoned, then something must have already panicked, likely in some other thread, so you're already in trouble at that point. It's fine to panic in a critical section if something's horribly wrong, the problem comes with blindly continuing after a panic in other threads that operate on the same data. In general, you're unlikely to know what that panic was, so you have no clue if the shared data might be incompletely modified or otherwise logically corrupted. In general, unless I were being careful to maintain fault boundaries between threads or tasks (the archetypical example being an HTTP server handling independent requests), I'd want a panic in one thread to cascade into stopping the program as soon as possible. I wouldn't want to swallow it up and keep using the same data like nothing's wrong.
- kouteiheika 11mo ago
  > If a mutex has been poisoned, then something must have already panicked, likely in some other thread, so you're already in trouble at that point. I find that in the majority of cases you're essentially dealing with one of two cases: 1) Your critical sections are tiny and you know you can't panic, in which case dealing with poisoning is just useless busywork. 2) You use a Mutex to get around Rust's "shared xor mutable" requirement. That is, you just want to temporarily grab a mutable reference and modify an object, but you don't have any particular atomicity requirements. In this case panicking is no different than if you would panic on a single thread while modifying an object through a plain old `&mut`. Here too dealing with poisoning is just useless busywork. > I'd want a panic in one thread to cascade into stopping the program as soon as possible. Sure, but you don't need mutex poisoning for this.
  - LegionMammal978 11mo ago
    > 1) Your critical sections are tiny and you know you can't panic, in which case dealing with poisoning is just useless busywork. Many people underestimate how many things can panic in corner cases. I've found quite a few unsafe functions in various crates that were unsound due to integer-overflow panics that the author hadn't noticed. Knowing for a fact that your operation cannot panic is the exception rather than the rule, and while it's unfortunate that the std Mutex doesn't accomodate non-poisoning mutexes, I see poisoning as a reasonable default. (If Mutex::lock() unwrapped the error automatically, then very few people would even think about the "useless busywork" of the poison bit. For a similar example, the future types generated for async functions contain panic statements in case they are polled after completion, and no one complains about those.) > 2) You use a Mutex to get around Rust's "shared xor mutable" requirement. That is, you just want to temporarily grab a mutable reference and modify an object, but you don't have any particular atomicity requirements. Then I'd stick to a RefCell. Unless it's a static variable in a single-threaded program, in which case I usually just write some short wrapper functions if I find the manipulation too tedious.
- kprotty 11mo ago
  > so you have no clue if the shared data might be incompletely modified or otherwise logically corrupted. One can make a panic wrapper type if they cared: It's what the stdlib Mutex currently does: MutexGuard checks if its panicking during drop using `std::thread::panicking()`, and if so, sets a bool on the Mutex. The next acquirer checks for that bool & knows state may be corrupted. No need to bake this into the Mutex itself.
sunshowers 11mo ago
To the contrary, the projects I've been part of have had no end of issues related to being cancelled in the middle of a critical section [1]. I consider poisoning to be table stakes for a mutex. [1] https://sunshowers.io/posts/cancelling-async-rust/#the-pain-of-tokio-mutexes https://sunshowers.io/posts/cancelling-async-rust/#the-pain-...
- kouteiheika 11mo ago
  Well, I mean, if you've made the unfortunate decision to hold a Mutex across await points...? This is completely banned in all of my projects. I have a 100k+ LOC project running in production, that is heavily async and with pervasive usage of threads and mutexes, and I never had a problem, precisely because I never hold a mutex across an await point. Hell, I don't even use async mutexes - I just use normal synchronous parking lot mutexes (since I find the async ones somewhat pointless). I just never hold them across await points.
  - sunshowers 11mo ago
    As I said in the article, we avoid Tokio mutexes entirely for the exact reason that being cancelled in the middle of a critical section is bad. In Rust, there are two sources of cancellations in the middle of a critical section: async cancellations and panics. Ergo, panicking in the middle of a critical section is also bad, and mutexes ought to detect that and mark their internal state as corrupted as a result.
    - kouteiheika 11mo ago
      > Ergo, panicking in the middle of a critical section is also bad, and mutexes ought to detect that and mark their internal state as corrupted as a result. I fundamentally disagree with this. Panicking in the middle of an operation that is supposed to be atomic is bad. If it's not supposed to be atomic then it's totally fine, just as panicking when you hold a plain old `&mut` is fine. Not every use of a `Mutex` is protecting an atomic operation that depends on not being cancelled for its correctness, and even for those situations where you do it's a better idea to prove that a panic cannot happen (if possible) or gracefully handle the panic. I really don't see a point of mutex poisoning in most cases. You can either safely panic while you're holding a mutex (because your code doesn't care about atomicity), or you simply write your code in such a way that it's still correct even if you panic (e.g. if you temporarily `.take()` something in your critical section then you write a wrapper which restores it on `Drop` in case of a panic). The only thing poisoning achieves is to accidentally give you denial-of-service CVEs, and is actively harmful when it comes to producing reliable software.
JoshTriplett 11mo ago
We're currently working on separating poison from mutexes, such that the default mutexes won't have poisoning (no more `.lock().unwrap()`), and if you want poisoning you can use something like `Mutex<Poison<T>>`.
- kouteiheika 11mo ago
  Yeah, I'm looking forward to it! While we're at it, another thing that'd be nice to get rid of is `AssertUnwindSafe`, which I find even more pointless.
  - JoshTriplett 11mo ago
    Speaking only for myself (though several other people have expressed the same sentiment), I wish we could get rid of unwinding. That would be a massive challenge to do while preserving capabilities people care about, such as the ability to handle panics in http request handlers without exiting. I think it would be possible, though.
    - questioner8216 11mo ago
      That sounds really interesting, whether it is done in Rust, some Rust 2.0, or a successor or experimental language. I do not know whether it is possible, though. If one does not unwind, what should actually happen instead? How would for instance partial computations, and resources on the stack, be handled? Some partial or constrained unwinding? I have not given it a lot of thought, though. How do languages without exceptions handle it? How does C handle it? Error codes all the way? Maybe something with arenas or regions? I do not have a good grasp on panics in Rust, but panics in Rust being able to either unwind or abort dependent on configuration, seems complex, and that design happened for historical reasons, from what I have read elsewhere.
      - JoshTriplett 11mo ago
        Vague sketch: imagine if we had scoped panic hooks, unhooked via RAII. So, for use cases that today use unwinding for cleanup (e.g. "switch the terminal back out of curses mode"), you do that cleanup in a panic hook instead. The hard use case to handle without unwinding is an HTTP server that wants to allow for panics in a request handler without panicking the entire process. Unwinding is a janky way to handle that, and creates issues in code that doesn't expect unwinding (e.g. half-modified states), and poisoning in particular seems likely to cascade and bring down other parts of the process anyway if some needed resource gets poisoned. But we need a reasonable alternative to propose for that use case, in order to seriously evaluate eliminating unwinding.
questioner8216 11mo ago
Questions for anyone who is an expert on poisoning in Rust: Is it safe to ignore poisoned mutexes if and only if the relevant pieces of code are unwind-safe, similar to exception safety in C++? As in, if a panic happens, the relevant pieces of code handles the unwinding safely, thus data is not corrupted, and thus ignoring the poison is fine?
lmm 11mo ago
> poisoning a Mutex is a very convenient avenue for a denial-of-service attack, since a poisoned Mutex will just completely brick a given critical section. There's a tension between making DoS hard and avoiding RCE vulnerabilities, since the way to avoid an unplanned/bad code state becoming an RCE vulnerability is to crash as quickly and thoroughly as possible when you get into that state.
juliangmp 11mo ago
I've dug into this topic in the past and my takeaway for this entire thing was “cool idea, but don't use it practice ”. I.e. just unrwap the lock call's result. If a worker thread panics you should assume your applications done for. Some people even recommend setting panic=abort for release builds, in which case you won't even be able to catch those panics to begin with. I mean, think about the actual use cases here. On of my threads just panicked. Does it make sense to continue running the application? And if you answer yes, this is an error condition that can occur, then it shouldn't panick to begin with and instead handle errors gracefully, leaving the mutex unpoisoned.
exDM69 11mo ago
I disagree, lock poisoning is a good way of improving correctness of concurrent code in case of fatal errors. As demonstrated by the benchmarks in this article, it's not very expensive for typical use cases. In 99% of the cases where one thread has panic'd while holding a lock, you want to panic the thread that attempts to grab the lock. The contents of anything inside the lock is very much undefined and continuing will lead to unpredictable results. So most of the time you just want: let guard = mutex.lock().expect("poisoned"); The last 1% is when you want to clean up something even if a panic has occured. This is usually in a impl Drop situation. It's not much more verbose either, just: let guard = mutex.lock().unwrap_or_else(|poison| poison.into_inner()); What is painful is trying to propagate the poison value as an error using `?`. In that case you're probably better off using a match expression because the usual `.into()` will not play nice with common error handling crates (thiserror, anyhow) or need to implement `From` manually for the error types and drop the contents of the poison error before propagating. This might be the case for long running server processes where you have n:m threading with long running threads and want to keep processing other requests even if one request fails. Although in that case you probably want (or your framework provides) some kind of robustness with `catch_unwind` that will log the errors, respond with HTTP 500 or whatever and then resume. Because that's needed to catch panics from non-mutex related code.

thayne 11mo ago

There are cases where it is useful. I had a case where if the mutex was poisened it was possible to reset the lock to a safe state (by writing a new value to the locked content). Or you may want to drop some resource or restart some operation instead of panicing if it is poisoned. But I agree that the default behavior should be that the user doesn't have to worry about it.

LegionMammal978 11mo ago

> It's always either an `unwrap` (and we know how well that can go [2]) If a mutex has been poisoned, then something must have already panicked, likely in some other thread, so you're already in trouble at that point. It's fine to panic in a critical section if something's horribly wrong, the problem comes with blindly continuing after a panic in other threads that operate on the same data. In general, you're unlikely to know what that panic was, so you have no clue if the shared data might be incompletely modified or otherwise logically corrupted. In general, unless I were being careful to maintain fault boundaries between threads or tasks (the archetypical example being an HTTP server handling independent requests), I'd want a panic in one thread to cascade into stopping the program as soon as possible. I wouldn't want to swallow it up and keep using the same data like nothing's wrong.

kouteiheika 11mo ago
> If a mutex has been poisoned, then something must have already panicked, likely in some other thread, so you're already in trouble at that point. I find that in the majority of cases you're essentially dealing with one of two cases: 1) Your critical sections are tiny and you know you can't panic, in which case dealing with poisoning is just useless busywork. 2) You use a Mutex to get around Rust's "shared xor mutable" requirement. That is, you just want to temporarily grab a mutable reference and modify an object, but you don't have any particular atomicity requirements. In this case panicking is no different than if you would panic on a single thread while modifying an object through a plain old `&mut`. Here too dealing with poisoning is just useless busywork. > I'd want a panic in one thread to cascade into stopping the program as soon as possible. Sure, but you don't need mutex poisoning for this.
- LegionMammal978 11mo ago
  > 1) Your critical sections are tiny and you know you can't panic, in which case dealing with poisoning is just useless busywork. Many people underestimate how many things can panic in corner cases. I've found quite a few unsafe functions in various crates that were unsound due to integer-overflow panics that the author hadn't noticed. Knowing for a fact that your operation cannot panic is the exception rather than the rule, and while it's unfortunate that the std Mutex doesn't accomodate non-poisoning mutexes, I see poisoning as a reasonable default. (If Mutex::lock() unwrapped the error automatically, then very few people would even think about the "useless busywork" of the poison bit. For a similar example, the future types generated for async functions contain panic statements in case they are polled after completion, and no one complains about those.) > 2) You use a Mutex to get around Rust's "shared xor mutable" requirement. That is, you just want to temporarily grab a mutable reference and modify an object, but you don't have any particular atomicity requirements. Then I'd stick to a RefCell. Unless it's a static variable in a single-threaded program, in which case I usually just write some short wrapper functions if I find the manipulation too tedious.
kprotty 11mo ago
> so you have no clue if the shared data might be incompletely modified or otherwise logically corrupted. One can make a panic wrapper type if they cared: It's what the stdlib Mutex currently does: MutexGuard checks if its panicking during drop using `std::thread::panicking()`, and if so, sets a bool on the Mutex. The next acquirer checks for that bool & knows state may be corrupted. No need to bake this into the Mutex itself.

sunshowers 11mo ago

To the contrary, the projects I've been part of have had no end of issues related to being cancelled in the middle of a critical section [1]. I consider poisoning to be table stakes for a mutex. [1] https://sunshowers.io/posts/cancelling-async-rust/#the-pain-of-tokio-mutexes https://sunshowers.io/posts/cancelling-async-rust/#the-pain-...

kouteiheika 11mo ago
Well, I mean, if you've made the unfortunate decision to hold a Mutex across await points...? This is completely banned in all of my projects. I have a 100k+ LOC project running in production, that is heavily async and with pervasive usage of threads and mutexes, and I never had a problem, precisely because I never hold a mutex across an await point. Hell, I don't even use async mutexes - I just use normal synchronous parking lot mutexes (since I find the async ones somewhat pointless). I just never hold them across await points.
- sunshowers 11mo ago
  As I said in the article, we avoid Tokio mutexes entirely for the exact reason that being cancelled in the middle of a critical section is bad. In Rust, there are two sources of cancellations in the middle of a critical section: async cancellations and panics. Ergo, panicking in the middle of a critical section is also bad, and mutexes ought to detect that and mark their internal state as corrupted as a result.
  - kouteiheika 11mo ago
    > Ergo, panicking in the middle of a critical section is also bad, and mutexes ought to detect that and mark their internal state as corrupted as a result. I fundamentally disagree with this. Panicking in the middle of an operation that is supposed to be atomic is bad. If it's not supposed to be atomic then it's totally fine, just as panicking when you hold a plain old `&mut` is fine. Not every use of a `Mutex` is protecting an atomic operation that depends on not being cancelled for its correctness, and even for those situations where you do it's a better idea to prove that a panic cannot happen (if possible) or gracefully handle the panic. I really don't see a point of mutex poisoning in most cases. You can either safely panic while you're holding a mutex (because your code doesn't care about atomicity), or you simply write your code in such a way that it's still correct even if you panic (e.g. if you temporarily `.take()` something in your critical section then you write a wrapper which restores it on `Drop` in case of a panic). The only thing poisoning achieves is to accidentally give you denial-of-service CVEs, and is actively harmful when it comes to producing reliable software.

JoshTriplett 11mo ago

We're currently working on separating poison from mutexes, such that the default mutexes won't have poisoning (no more `.lock().unwrap()`), and if you want poisoning you can use something like `Mutex<Poison<T>>`.

kouteiheika 11mo ago
Yeah, I'm looking forward to it! While we're at it, another thing that'd be nice to get rid of is `AssertUnwindSafe`, which I find even more pointless.
- JoshTriplett 11mo ago
  Speaking only for myself (though several other people have expressed the same sentiment), I wish we could get rid of unwinding. That would be a massive challenge to do while preserving capabilities people care about, such as the ability to handle panics in http request handlers without exiting. I think it would be possible, though.
  - questioner8216 11mo ago
    That sounds really interesting, whether it is done in Rust, some Rust 2.0, or a successor or experimental language. I do not know whether it is possible, though. If one does not unwind, what should actually happen instead? How would for instance partial computations, and resources on the stack, be handled? Some partial or constrained unwinding? I have not given it a lot of thought, though. How do languages without exceptions handle it? How does C handle it? Error codes all the way? Maybe something with arenas or regions? I do not have a good grasp on panics in Rust, but panics in Rust being able to either unwind or abort dependent on configuration, seems complex, and that design happened for historical reasons, from what I have read elsewhere.
    - JoshTriplett 11mo ago
      Vague sketch: imagine if we had scoped panic hooks, unhooked via RAII. So, for use cases that today use unwinding for cleanup (e.g. "switch the terminal back out of curses mode"), you do that cleanup in a panic hook instead. The hard use case to handle without unwinding is an HTTP server that wants to allow for panics in a request handler without panicking the entire process. Unwinding is a janky way to handle that, and creates issues in code that doesn't expect unwinding (e.g. half-modified states), and poisoning in particular seems likely to cascade and bring down other parts of the process anyway if some needed resource gets poisoned. But we need a reasonable alternative to propose for that use case, in order to seriously evaluate eliminating unwinding.

questioner8216 11mo ago

Questions for anyone who is an expert on poisoning in Rust: Is it safe to ignore poisoned mutexes if and only if the relevant pieces of code are unwind-safe, similar to exception safety in C++? As in, if a panic happens, the relevant pieces of code handles the unwinding safely, thus data is not corrupted, and thus ignoring the poison is fine?

lmm 11mo ago

> poisoning a Mutex is a very convenient avenue for a denial-of-service attack, since a poisoned Mutex will just completely brick a given critical section. There's a tension between making DoS hard and avoiding RCE vulnerabilities, since the way to avoid an unplanned/bad code state becoming an RCE vulnerability is to crash as quickly and thoroughly as possible when you get into that state.

juliangmp 11mo ago

I've dug into this topic in the past and my takeaway for this entire thing was “cool idea, but don't use it practice ”. I.e. just unrwap the lock call's result. If a worker thread panics you should assume your applications done for. Some people even recommend setting panic=abort for release builds, in which case you won't even be able to catch those panics to begin with. I mean, think about the actual use cases here. On of my threads just panicked. Does it make sense to continue running the application? And if you answer yes, this is an error condition that can occur, then it shouldn't panick to begin with and instead handle errors gracefully, leaving the mutex unpoisoned.

exDM69 11mo ago

I disagree, lock poisoning is a good way of improving correctness of concurrent code in case of fatal errors. As demonstrated by the benchmarks in this article, it's not very expensive for typical use cases. In 99% of the cases where one thread has panic'd while holding a lock, you want to panic the thread that attempts to grab the lock. The contents of anything inside the lock is very much undefined and continuing will lead to unpredictable results. So most of the time you just want: let guard = mutex.lock().expect("poisoned"); The last 1% is when you want to clean up something even if a panic has occured. This is usually in a impl Drop situation. It's not much more verbose either, just: let guard = mutex.lock().unwrap_or_else(|poison| poison.into_inner()); What is painful is trying to propagate the poison value as an error using `?`. In that case you're probably better off using a match expression because the usual `.into()` will not play nice with common error handling crates (thiserror, anyhow) or need to implement `From` manually for the error types and drop the contents of the poison error before propagating. This might be the case for long running server processes where you have n:m threading with long running threads and want to keep processing other requests even if one request fails. Although in that case you probably want (or your framework provides) some kind of robustness with `catch_unwind` that will log the errors, respond with HTTP 500 or whatever and then resume. Because that's needed to catch panics from non-mutex related code.