4 ms·

Yeah, you can't enable priority inheritance for mutexes in std of either C++ or Rust. Which is a show stopper for hard realtime (my dayjob). And then you have

Yeah, you can't enable priority inheritance for mutexes in std of either C++ or Rust. Which is a show stopper for hard realtime (my dayjob). And then you have mutexes internally inside some dependency still (e.g. grpc or what have you). What I would really like is the ability to change defaults for all mutexes created in the program, and have everyone use the same std mutexes. By the way: rwlocks are often a bad idea, since you still get cache contention between readers on the counter for number of active readers. Unless the time you hold the lock for is really long (several milliseconds at least) it usually doesn't improve performance compared to mutexes. Consider alternatives like seqlocks, RCU, hazard pointers etc instead, depending on the specifics of your situation (there is no silver bullet when it comes to performance in concurrent primtitves).

jcalvinowens 8mo ago
> What I would really like is the ability to change defaults for all mutexes created in the program, and have everyone use the same std mutexes. Assuming you're building the whole userspace at once with something like yocto... you can just patch pthread to change the default to PTHREAD_PRIO_INHERIT and silently ignore attempts to set it to PTHREAD_PRIO_NONE. It's a little evil though. > By the way: rwlocks are often a bad idea +1
- i_am_a_peasant 8mo ago
  i think both you guys have the same job as me lol
- VorpalWay 8mo ago
  That is a great terrible idea (I really have to think a bit more on that). Won't help for Rust, since the mutexes there use futex directly, so you would have to patch the standard library itself (and for futex it is more complex than just enabling a flag). Seems plausible that other libraries and language runtimes might do similar things.
  - surajrmal 8mo ago
    The implementation for the rust std mutex when targeting fuchsia does implement priority inheritance by default, but the zircon kernel scheduler and futex implementation are written with priority inheritance in mind as the default approach rather than something ad hoc tacked on. Unfortunately on Linux it seems like there is a large performance tradeoff which may not be worthwhile for the common case. It does seem like it would be nice to set an env variable to change the behavior through rather than require a recompile of libstd. A lot of programs use alternatives to the std library as well like parking_lot, which is indeed a pain. Sometimes I feel like trying to use Linux for realtime is an effort in futility. The ecosystem is optimized for throughput over fairness, predictability, and latency.
    - jcalvinowens 8mo ago
      > Unfortunately on Linux it seems like there is a large performance tradeoff Implementing transitive priority inheritance is just inherently algorithmically more expensive: there's no avoiding that. > Sometimes I feel like trying to use Linux for realtime is an effort in futility. If you're not actually using an RT kernel, yeah, it's futile. But if you are, the guarantees are pretty strong... on x86 PCs, the hardware gets in the way much more than the software in my experience. There's a lot of active work upstream.
      - VorpalWay 8mo ago
        The Linux priority inheritance futexes are also fair, which adds unnecessary overhead if you only care about PI, not fairness.
loeg 8mo ago
> What I would really like is the ability to change defaults for all mutexes created in the program, and have everyone use the same std mutexes. Yes. I imagine subbing out a debug mutex implementation that tracks lock ordering and warns about order inversion (similar to things like witness(4)): https://man.freebsd.org/cgi/man.cgi?witness(4) https://man.freebsd.org/cgi/man.cgi?witness(4) > rwlocks are often a bad idea Yes. > since you still get cache contention between readers on the counter for number of active readers There are rwlock impls that put the reader counts on distinct cache lines per core, or something like that (e.g., folly::SharedMutex), mitigating this particular problem. But it isn't the only problem with rwlocks. > Consider alternatives like seqlocks, RCU, hazard pointers etc instead, depending on the specifics of your situation (there is no silver bullet when it comes to performance in concurrent primtitves). Yes. :)
- menaerus 8mo ago
  What are some of the other problems with rwlocks? I genuinely ask because I used them with quite good success for pretty complicated use-case at large scale and very high concurrency.
  - ethin 8mo ago
    I have as well. I find RW locks much easier to use than, say, a recursive mutex. Mainly since it took me a long time to actually understand how a recursive mutex actually works in the first place. When you want to use only the stdlib, you aren't left with many choices. At least in the STL.
  - loeg 8mo ago
    Mostly that if you actually have both readers and writers, they obstruct each other; this is often undesirable. And you have to pick some bias in advance. You can get priority inversion because readers are anonymous.
    - menaerus 8mo ago
      Sure, the workload was with both the readers and the writers, and it was a pretty "bursty" one with high volume of data which had to scale across all the cores. So, not a particularly light workload. It was basically a hi-concurrency cache which was write-mostly in the first-phase (ingestion), and then read-mostly in the second-phase (crunching). It had to support multiple sessions simultaneously at the same time so in the end it was about supporting heavily mixed read-write workloads, e.g. second-phase from session nr. 1 could overlap with the first-phase from session nr. 2. To avoid the lock contention I managed to get away with sharding across the array of shared-mutexes and load-balancing the sessions by their UUIDs. And this worked pretty well - after almost ~10 years it's still rock solid, and workloads are basically ever changing. I considered the RCU for this use-case too but I figured that it wouldn't be as good fit because the workload is essentially heavily mixed so I thought it would result with a lot of strain to the memory subsystem by having to handle multiple copies of the data (which was not small). One thing I don't understand is the priority inversion and how that may happen. I'll think about it, thanks.
nly 8mo ago
There are rw lock implementations where waiters (whether or readers or writers) don't contend on a shared cache line (they only touch it once to enqueue themselves, not to spin/wait) These are usually called "scaleable locks" and the algorithms for them have been out there for decades. They are optimal from a cache coherence point of view. The issue with them is it's impossible to support the same API as you're used to with std::shared_mutex, as every thread needs it's own line.
- gpderetta 8mo ago
  > it's impossible to support the same API as you're used to with std::shared_mutex If I understand the problem correctly, the standard library could specialize the scoped lock objects for the shared mutex, online allocate the waiter object in the lock (so it will be in the stack) and use an internal dedicated API. It might be harder (but not impossible) to interoperate threads using direct lock calls with threads using the scoped API. In any case it is a moot point, as I don't think any std library does it nor will never do as it would probably be ABI breaking.
  - nly 8mo ago
    Yes, The RAII lock_guard pattern is the correct and proper one. But efficient FIFO/scalable locks can't be created using the C POSIX locking API.

jcalvinowens 8mo ago

> What I would really like is the ability to change defaults for all mutexes created in the program, and have everyone use the same std mutexes. Assuming you're building the whole userspace at once with something like yocto... you can just patch pthread to change the default to PTHREAD_PRIO_INHERIT and silently ignore attempts to set it to PTHREAD_PRIO_NONE. It's a little evil though. > By the way: rwlocks are often a bad idea +1

i_am_a_peasant 8mo ago
i think both you guys have the same job as me lol
VorpalWay 8mo ago
That is a great terrible idea (I really have to think a bit more on that). Won't help for Rust, since the mutexes there use futex directly, so you would have to patch the standard library itself (and for futex it is more complex than just enabling a flag). Seems plausible that other libraries and language runtimes might do similar things.
- surajrmal 8mo ago
  The implementation for the rust std mutex when targeting fuchsia does implement priority inheritance by default, but the zircon kernel scheduler and futex implementation are written with priority inheritance in mind as the default approach rather than something ad hoc tacked on. Unfortunately on Linux it seems like there is a large performance tradeoff which may not be worthwhile for the common case. It does seem like it would be nice to set an env variable to change the behavior through rather than require a recompile of libstd. A lot of programs use alternatives to the std library as well like parking_lot, which is indeed a pain. Sometimes I feel like trying to use Linux for realtime is an effort in futility. The ecosystem is optimized for throughput over fairness, predictability, and latency.
  - jcalvinowens 8mo ago
    > Unfortunately on Linux it seems like there is a large performance tradeoff Implementing transitive priority inheritance is just inherently algorithmically more expensive: there's no avoiding that. > Sometimes I feel like trying to use Linux for realtime is an effort in futility. If you're not actually using an RT kernel, yeah, it's futile. But if you are, the guarantees are pretty strong... on x86 PCs, the hardware gets in the way much more than the software in my experience. There's a lot of active work upstream.
    - VorpalWay 8mo ago
      The Linux priority inheritance futexes are also fair, which adds unnecessary overhead if you only care about PI, not fairness.

loeg 8mo ago

> What I would really like is the ability to change defaults for all mutexes created in the program, and have everyone use the same std mutexes. Yes. I imagine subbing out a debug mutex implementation that tracks lock ordering and warns about order inversion (similar to things like witness(4)): https://man.freebsd.org/cgi/man.cgi?witness(4) https://man.freebsd.org/cgi/man.cgi?witness(4) > rwlocks are often a bad idea Yes. > since you still get cache contention between readers on the counter for number of active readers There are rwlock impls that put the reader counts on distinct cache lines per core, or something like that (e.g., folly::SharedMutex), mitigating this particular problem. But it isn't the only problem with rwlocks. > Consider alternatives like seqlocks, RCU, hazard pointers etc instead, depending on the specifics of your situation (there is no silver bullet when it comes to performance in concurrent primtitves). Yes. :)

menaerus 8mo ago
What are some of the other problems with rwlocks? I genuinely ask because I used them with quite good success for pretty complicated use-case at large scale and very high concurrency.
- ethin 8mo ago
  I have as well. I find RW locks much easier to use than, say, a recursive mutex. Mainly since it took me a long time to actually understand how a recursive mutex actually works in the first place. When you want to use only the stdlib, you aren't left with many choices. At least in the STL.
- loeg 8mo ago
  Mostly that if you actually have both readers and writers, they obstruct each other; this is often undesirable. And you have to pick some bias in advance. You can get priority inversion because readers are anonymous.
  - menaerus 8mo ago
    Sure, the workload was with both the readers and the writers, and it was a pretty "bursty" one with high volume of data which had to scale across all the cores. So, not a particularly light workload. It was basically a hi-concurrency cache which was write-mostly in the first-phase (ingestion), and then read-mostly in the second-phase (crunching). It had to support multiple sessions simultaneously at the same time so in the end it was about supporting heavily mixed read-write workloads, e.g. second-phase from session nr. 1 could overlap with the first-phase from session nr. 2. To avoid the lock contention I managed to get away with sharding across the array of shared-mutexes and load-balancing the sessions by their UUIDs. And this worked pretty well - after almost ~10 years it's still rock solid, and workloads are basically ever changing. I considered the RCU for this use-case too but I figured that it wouldn't be as good fit because the workload is essentially heavily mixed so I thought it would result with a lot of strain to the memory subsystem by having to handle multiple copies of the data (which was not small). One thing I don't understand is the priority inversion and how that may happen. I'll think about it, thanks.

nly 8mo ago

There are rw lock implementations where waiters (whether or readers or writers) don't contend on a shared cache line (they only touch it once to enqueue themselves, not to spin/wait) These are usually called "scaleable locks" and the algorithms for them have been out there for decades. They are optimal from a cache coherence point of view. The issue with them is it's impossible to support the same API as you're used to with std::shared_mutex, as every thread needs it's own line.

gpderetta 8mo ago
> it's impossible to support the same API as you're used to with std::shared_mutex If I understand the problem correctly, the standard library could specialize the scoped lock objects for the shared mutex, online allocate the waiter object in the lock (so it will be in the stack) and use an internal dedicated API. It might be harder (but not impossible) to interoperate threads using direct lock calls with threads using the scoped API. In any case it is a moot point, as I don't think any std library does it nor will never do as it would probably be ABI breaking.
- nly 8mo ago
  Yes, The RAII lock_guard pattern is the correct and proper one. But efficient FIFO/scalable locks can't be created using the C POSIX locking API.