20 ms·

MutexProtected: A C++ Pattern for Easier Concurrency

118 points by soheilpro 4y ago

chombier 4y ago
Or you could give `MutexLocker` pointer semantics to access the underlying object only through the lock, which would probably be more idiomatic c++ anyways.
- Yoric 4y ago
  Lifetimes might be a bit trickier, though.
  - chombier 4y ago
    Indeed, with lambdas you don't need to deal with locks outliving the underlying object (though I think std::unique_lock has the same issue with the underlying mutex?) The downside with lambdas is that you cannot lock across function boundaries e.g. give ownership of a lock to the caller. There's a similar tradeoff for all "context managers" (in Python parlance) where you could either CPS your way around the control flow (the lambda approach) or use c++'s RAII mechanism. My impression is that RAII feels a bit more idiomatic and puts the burden on library writers instead of users.
- saagarjha 4y ago
  Individual accesses are typically the wrong level of abstraction for synchronization. You’ll avoid simple data races but it’s easy to have accidental reentrancy this way.
vore 4y ago
Clang’s static thread safety analysis is pretty good for this too: https://clang.llvm.org/docs/ThreadSafetyAnalysis.html https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
ajtulloch 4y ago
https://github.com/facebook/folly/blob/main/folly/docs/Synchronized.md https://github.com/facebook/folly/blob/main/folly/docs/Synch...
afc 4y ago
I also use this pattern: https://github.com/alefore/edge/blob/master/src/concurrent/protected.h https://github.com/alefore/edge/blob/master/src/concurrent/p... I added an optional template Validator parameter that can be used to double-check invariants on the protected fields. I also declared a Protected with condition subclass. Here is one example of how I use it to implement a "barrier" of sorts (the operation instance blocks in the destructor until execution of all callables passed to "Add" has completed): https://github.com/alefore/edge/blob/master/src/concurrent/operation.h https://github.com/alefore/edge/blob/master/src/concurrent/o... I found it slightly preferable over Abseil annotations; I think it's slightly more robust (i.e., make errors, like forgetting an annotation, less likely) and I like making this explicit in the type system.
Blackthorn 4y ago
Neat idea. Though I think having to pass a lambda for anything you want to do with the fields is awful ergonomics. Maybe instead combine the two ideas. MutexProtected<T> but can be locked with MutexProtected<T>::lock() which returns a MutexLocked<T> object. That object then cleans up the lock when it goes out of scope, and also provides direct access to the enclosed type.
- akling 4y ago
  Yes, that's a great way to improve the ergonomics of MutexProtected in simple cases! We should totally add that as a complement to the lambda API. :)
  - akling 4y ago
    After sketching this out, it's nice that this: auto x = state.with([](auto& state) { return state.x; }); Becomes this: auto x = state.locked()->x; But it also creates this very accessible footgun: auto& x = state.locked()->x; Where it's way too easy to bind a reference to something that should be protected by the lock, but now isn't. So I'm not sure this is a great idea anymore.
    - gpderetta 4y ago
      At the end of the day, this is C++, there are no compiler tracked lifetimes and it is easy to leak references out of the lambda as well. At least the synchronized pattern makes it easy to document which state is protected by which mutex.
    - kajaktum 4y ago
      Wait why is this a footgun?
      - akling 4y ago
        Because now you're holding a reference to `x` which is supposed to be protected by a mutex, even after the mutex is unlocked. With the lambda-only API, it's much harder to make this mistake, since a temporary reference like this will still go out of scope at the end of the lambda expression.
comex 4y ago
This is how Rust’s mutexes work as well - albeit using RAII rather than a callback, but with the same fundamental innovation of putting the data inside the mutex object. I’ve used it in my C++ code as well; it’s a great pattern.
- nextaccountic 4y ago
  But is there a way to use Rust's approach of RAII guards in C++, rather than accessing the mutex inside a closure?
  - gpderetta 4y ago
    Of course. Boost.synchronized and the folly variant have been mentioned elsethread.
  - ReactiveJelly 4y ago
    Adding my favorite to the pile: https://github.com/dragazo/rustex https://github.com/dragazo/rustex
- tialaramex 4y ago
  Crucially the Rust borrow checker means using this wrong from (safe) Rust is impossible, whereas it's relatively easy to use either C++ approach wrongly, especially if nobody has actually sat down and explained why you're doing it. In C++ we can "just" leak the reference to the protected data, in (safe) Rust of course the borrow checker means that programs with this mistake won't build. Rust's ZSTs (~ unit types have zero storage size) make it reasonable to write Mutex<MyMarker> and then require that people show you their MyMarker (which they can only get at by holding the lock) to do stuff thus enforcing locking even when it's not really data you're protecting. Because these are zero size, the resulting machine code is unchanged but the type checking means if you forget to lock first your program won't even build, which shifts locking mistakes very hard left.
  - chrisshroba 4y ago
    Can you give an example of using a ZST as a marker in a Mutex? I think I understand that you're suggesting this as a way of locking some section of code even if it's not a specific piece of data that you're locking, but I'm wondering how this could "enforce locking" then, since anything you do while locked could just as easily be accidentally done without the lock, right?
amadvance 4y ago
In truth, I prefer using explicit lock and unlock operations like in C programming. The visibility of these operations aids in understanding the interdependencies within the code.
- pavlov 4y ago
  I could understand this argument against the RAII-style destructor unlock, but this lambda approach seems actually better in the visibility sense since the mutex access is a separately indented block.
  - amadvance 4y ago
    It's obviously subjective, but it looks like as any other indented block. Named "lock" and "unlock" operations are more explicit to me. Another potential disadvantage, is that you lose the possibility to control the unlock order, when you have multiple nested locks.
    - gpderetta 4y ago
      An indented scope (that starts with a scoped_lock or some appropriately named function) is more explicit than lock/unlock calls interspersed in the code. std::scoped_lock allows locking multiple locks.
  - snozolli 4y ago
    this lambda approach seems actually better in the visibility sense since the mutex access is a separately indented block. When I'm writing multithreaded code, the first line of any block containing a RAII lock is the lock. Any subsection of code that needs another lock gets its own block.
- gpderetta 4y ago
  I don't trust explicit lock and un lock operations abstractions in C, I always use inline asm to acquire and release my locks.
eps 4y ago
A pattern for easier slower concurrency as lambdas aren't free. If the code under the mutex is trivial, this cost of using lambdas will be noticeable, especially in fast paths. Might be worth looking at the resulting assembly first if you are looking to adapt this in your codebase.
- gpderetta 4y ago
  Aside from lambdas being cheap abstractions in C++, any code under a mutex is likely not the fast path anyway.
  - saagarjha 4y ago
    I mean, some code is all slow paths ;)
- ot 4y ago
  Lambdas are free. Unless the class type-erases the callback (and MutexProtected doesn't), and the callback is not huge, the compiler is virtually guaranteed to inline it.
  - huhtenberg 4y ago
    It can go both ways. Clang will inline everything - https://godbolt.org/z/v19P1W9sj https://godbolt.org/z/v19P1W9sj MSVC will not, even with O2 - https://godbolt.org/z/naovTxncs https://godbolt.org/z/naovTxncs > Might be worth looking at the resulting assembly first if you are looking to adapt this in your codebase.
- akling 4y ago
  AFAIK if you use a templated type for the callback (as you should), it will be inlined by modern compilers at any optimization level other than -O0. :)
lunfard000 4y ago
maybe add SerenityOS to the title? Almost sure that lock_guard is the ideomatic way to do this on cpp, not buying the "forget to lock".
- Yoric 4y ago
  FWIW, I've encountered "forget to lock" in production C++.
- saagarjha 4y ago
  It’s a general C++ trick.
secondcoming 4y ago
We use something quite similar to this. It won’t save you from accidental recursive locking though, but I’m not sure any solution will.
- pavlov 4y ago
  If the problem is really just recursive lock calls on the same thread, then PTHREAD_MUTEX_RECURSIVE should do the trick?
  - saagarjha 4y ago
    Generally PTHREAD_MUTEX_RECURSIVE is a band-aid, not a solution.
    - AnimalMuppet 4y ago
      Could you explain? A simple drive-by condemnation doesn't tell us why, or give us any reason to agree with you.
      - gpderetta 4y ago
        This gets shared periodically here: https://groups.google.com/g/comp.programming.threads/c/tcrTKnfP8HI/m/me2K7_byNdgJ https://groups.google.com/g/comp.programming.threads/c/tcrTK...
      - secondcoming 4y ago
        If you find yourself needing recursive locks your design is wrong. They were invented as a dare.
      - loeg 4y ago
        They mean you don't know and can't reason about lock state. To design concurrent systems with good performance and correctness, you need to be able to reason about lock state statically. Here's some random article I found in 30 seconds of googling that seems to go into more depth on the subject: https://blog.stephencleary.com/2013/04/recursive-re-entrant-locks.html https://blog.stephencleary.com/2013/04/recursive-re-entrant-...
nly 4y ago
Or...you could use Clangs GUARDED_BY annotations and thread safety analysis... https://clang.llvm.org/docs/ThreadSafetyAnalysis.html https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
pjmlp 4y ago
This is basically one of the ways to do RAII in GC based languages, only provide the resources via lambdas, thus kind of simulating arena like resource managment, the optimizer will take care of inlining them anyway.
gpderetta 4y ago
Another common addition to these lockable objects is having a free function variant of 'with' that allows visiting multiple objects: MutexProtected<X> x; MutexProtected<Y> y; ... with(x, y, [](X& x, Y& y) {....} ); Using the scoped_lock protocol this can avoid deadlocks.
franke2 4y ago
Even if very similar, I prefer the MutexedObj class described in those two articles https://fekir.info/post/extend-generic-thread-safe-mutexed_obj-class/ https://fekir.info/post/extend-generic-thread-safe-mutexed_o... https://fekir.info/post/sharing-data-between-threads/#_bind-the-data-and-mutex-together https://fekir.info/post/sharing-data-between-threads/#_bind-... In particular it explains why providing getters or operator* like some other implementation do (not in this case) is not a feature, and it does not use multiple classes (like the linked implementation of SerenityOS does) making the implementation simpler. Also the "simple" implementation is just 20 lines of code...
einpoklum 4y ago
How does this compare with: * Using std::atomic<T> ? https://en.cppreference.com/w/cpp/atomic/atomic https://en.cppreference.com/w/cpp/atomic/atomic * folly::Synchronized, mentioned by ot57? https://github.com/facebook/folly/blob/main/folly/docs/Synchronized.md https://github.com/facebook/folly/blob/main/folly/docs/Synch... * Boost synchronized data structures, -mentioned by mchicken 53? https://www.boost.org/doc/libs/1_81_0/doc/html/thread/sds.html https://www.boost.org/doc/libs/1_81_0/doc/html/thread/sds.ht... ? ... it looks like the SerenityOS people developed this without considering the C++ ecosystem of today.
- gpderetta 4y ago
  std::atomic really has different use cases. It is meant to be used for objects that can be modified via atomic RMW primitives and require T to be trivially copyable. While you can in principle implement arbitrary operations on top of those RMWs, it might not be the best fit. For large Ts, is indeed implemented using a mutex or, more likely, a lock pool, so I guess if you squint hard enough it can be considered related.
- MontagFTB 4y ago
  My understanding is that Serenity has a philosophy of not using dependencies outside of itself. Perhaps this type drew from external inspiration, but was ultimately written in light of that philosophy?
- loeg 4y ago
  Well, std::atomic is wholly unrelated. MutexProtected seems fundamentally similar to boost and folly synchronized objects, just without the ergonomics and battle-testing the other major libraries have already gotten.
mgaunard 4y ago
If you want easier concurrency, don't use mutexes, they're a bad pattern. It's well-established that the safe way to do concurrency is to use channels.
- jstx1 4y ago
  What's the problem with mutexes?
  - mgaunard 4y ago
    If you have to ask, I suppose you shouldn't touch anything that involves concurrent or parallel execution.
  - paulmd 4y ago
    mutexes are inherently single-threaded code, it's exclusive. It's a point where all of your threads have to serialize and run one-at-a-time to perform a certain operation. additionally, common implementations often have bad behavior as the number of threads increases. spinlocks don't work well if you have 32 threads trying to take the same lock, they can consume quite a lot of CPU time just trying to take the lock. they don't work at all if you have thousands of threads (which is a situation that occurs regularly in GPGPU programming!). you can of course increase the number of locks but that's basically what SQL/RDBMS does with row locks - and now you have the problem of deadlock too. obviously it all depends on specifics - how many threads doing how much work on their own, compared to how many locks they're trying to take. it's not that they are inherently bad, they are in fact one of the basic primitives in concurrent programming really. they just don't really scale well with increased concurrency. unfortunately, the deeper solutions involve restructuring your program and your data processing (or the data itself) to expose higher levels of concurrency and less interdependency, which is its own can of worms.
- tlb 4y ago
  That can work well for many programs, but no one has succeeded in writing a multithreaded kernel that way.
unwind 4y ago
I thought this was cool, and I really appreciate the low-level bottom-up description/introduction of how MutexProtected helps keep things safe. I would have loved also getting a disassembly showing the resulting code, it's very hard for me (as an only-sometimes C++ programmer) to guess how the lambda is compiled.
samsquire 4y ago
I am working on a left-right concurrency control with multiple threads. You shard your data and define a merge operation that is commutative so you can always merge data of different threads. By sharding data by thread each thread can always read and write to its own shard without synchronization or contention. I also have a global snapshot per thread which is the aggregated data across all threads that is local to a thread. So it requires 4× the memory per thread but it is extremely performant due to each thread is independent of every other thread
boricj 4y ago
Another feature of the MutexProtected implementation in SerenityOS is the ability to either obtain an exclusive lock (writable) or a shared lock (read-only). A shared lock provides a const reference to the protected value and therefore enforces read-only access semantics at compile-time by leveraging the C++ type system (specifically, const-correctness). The initial PR introducing the ancestor to MutexProtected has some details on the motivation behind it and also unearthed a bunch of incorrect locking that the C++ compiler caught when introducing it: https://github.com/SerenityOS/serenity/pull/8851 https://github.com/SerenityOS/serenity/pull/8851
mike_hearn 4y ago
Yep, it's a useful pattern albeit not a game changer or anything. I've been using it in Kotlin for a while. Here's an implementation: https://gist.github.com/mikehearn/1913202829403f65331123f0473a7901 https://gist.github.com/mikehearn/1913202829403f65331123f047... One of the nice things about doing it in Kotlin rather than C++ is the clean syntax using trailing lambda blocks and anonymous objects: private val state = Locker(object { var value = 1 var another = "value" }) val nextValue = state.locked { ++value } This works because Kotlin lets you define anonymous types that can take part in type inference even though they're not denotable, and then use them as lambda receivers. So "this" inside the final block points to the unnamed singleton. It has a few nice features: • It's runtime overhead free. Kotlin frontend can inline the "locked" call so you end up with the equivalent of manual locking. • There's no way to get to the state without holding the lock unless you deliberately leak it outside the lambda. • You can control re-entrancy. • It tells the compiler the lambda is only invoked once, which has various benefits for allowing more natural code constructs. These days I'd probably code it with an explicit ReadWriteLock so it's Loom compatible and to allow explicit read/write blocks, but the JVM optimizes locks pretty well so if there's never any contention the memory for the lock is never allocated. I'd also experiment with making it value type so the overhead of the Locker object goes away. But it was never necessary so far.
throwbadubadu 4y ago
> A C++ Pattern for Easier Concurrency Shouldn't downplay sharing that idea, but come on: Realizing Python-like-with-contexts (and yes, even Python isn't the inventor of that..) in C++ with lambdas is a general pattern (think of scenarios like db transactions, or also stuff where RAII pattern would be useful, except that you need a result from the destructor, or it could even fail..) that shouldn't be too new.. Since C++11 when lambdas where introduced, to be precise?
quietbritishjim 4y ago
[edit: deleted first half of my comment - it was just duplication of existing top comment] ----- Even better, when possible (it isn't always), is to avoid using mutexes at all, and instead have the data structure owned by a relevant thread. When you need to access a data structure, pass a message to that thread, and then just access the data structure freely when you get there. (Of course the inter thread queue uses a mutex or some other synchronisation mechanism, and the queue itself effectively acts as a synchronisation mechanism.) struct SomeMsg { SomeMsg() { thread_foo->pass_message(std::bind_front(this, &SomeMsg::handle_foo)); } void handle_foo(Foo& foo) { // ... use foo ... thread_bar->pass_message(std::bind_front(this, &SomeMsg::handle_bar)); } void handle_bar(Bar& bar) { // ... etc. ... } };
- smallstepforman 4y ago
  You’ve just described Actors. Now take the next step and add a work stealing schedular for a fuller Actor system.
  - MontagFTB 4y ago
    I’ve been toying with these lately, too. My solution was to give each actor an executor where it will run: actor<int> foo(stlab::default_executor, “name”, 42); auto f = foo.send([](auto& x){ std::cout << x << ‘\n’; // prints 42 }); Each call to ‘send’ returns a stlab::future which can have a continuation chained, etc. The nice thing about these is the executor based interface is generic. The actor could be dedicated to a thread, run on an OS-provided thread pool, etc. At the time the actor is running, the thread it is on is temporarily given the name of the actor to ease debugging.
  - quietbritishjim 4y ago
    Almost. If each thread were an Actor (I think that's what you're saying?) then that would structure the code around each type of work e.g. here is the code for all the database operations. That's not a very useful way to group things for the developer. The way I wrote it above, the code is structured around a single activity across all the types of work it has to do.
planede 4y ago
See also: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p0290r4.html https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p02... This got accepted in C++23.
- tialaramex 4y ago
  It got added to the Concurrency TS during the C++ 23 process. It's not in standard C++ itself.
  - planede 4y ago
    Ah, OK.
pelorat 4y ago
That's not any easier than RAII, although I wish C++ had the lock syntax from C# lock (object) { // do stuff }
- gpderetta 4y ago
  That's commonly done in C++ with macro, but the lock(object) syntax doesn't prevent doing stuff with object without locking the mutex, while the synchronized pattern does.
thesuperbigfrog 4y ago
MutexProtected is almost Ada protected objects for C++: https://learn.adacore.com/courses/Ada_For_The_CPP_Java_Developer/chapters/11_Concurrency.html#protected-objects https://learn.adacore.com/courses/Ada_For_The_CPP_Java_Devel... This IS a very useful concurrency pattern. Any language that implements concurrent access to shared data should have it or something very similar.
HarHarVeryFunny 4y ago
If you want an mt-safe data structure, then may as well just build it that way in the first place, with a private std::mutex and getters/setters that use std::lock_guard.
29athrowaway 4y ago
How does it handle exceptions?
- loeg 4y ago
  Internally it uses a MutexLocker which will get destructed as an exception unwinds the stack. The destructor unlocks the mutex. https://github.com/SerenityOS/serenity/blob/master/Kernel/Locking/MutexProtected.h#L55 https://github.com/SerenityOS/serenity/blob/master/Kernel/Lo... https://github.com/SerenityOS/serenity/blob/master/Kernel/Locking/Mutex.h#L146 https://github.com/SerenityOS/serenity/blob/master/Kernel/Lo...
loeg 4y ago
folly::Synchronized<T> provides a similar pattern. It works well for very simplistic concurrency schemes. struct { int a; int b; } A; folly::Synchronized<A> syncA; int c; { auto aReader = syncA.rlock(); c = aReader->a; // aReader going out of scope drops the rlock } { auto aWriter = syncA.wlock(); aWriter->b = 15; // aWriter going out of scope drops the wlock } I would argue that the pseudo-pointer accessor objects are more usable than passing lambdas to a `with()` method, but if you want that, folly::Synchronized provides similar `withWLock()` and `withRLock()` methods. And in fact, MutexProtected seems to provide APIs like Synchronized's `rlock()`/`wlock()`: `lock_shared()` and `lock_exclusive()`: https://github.com/SerenityOS/serenity/blob/master/Kernel/Locking/MutexProtected.h#L58-L59 https://github.com/SerenityOS/serenity/blob/master/Kernel/Lo... I don't know why the article doesn't touch on these. https://github.com/facebook/folly/blob/main/folly/docs/Synchronized.md https://github.com/facebook/folly/blob/main/folly/docs/Synch... https://github.com/facebook/folly/blob/main/folly/Synchronized.h https://github.com/facebook/folly/blob/main/folly/Synchroniz...
PointyFluff 4y ago
[dead]
nayuki 4y ago
The "Barebones example in C" is enshrined in Java at the language level as every object has a built-in lock and condition variable, and locking is done through the synchronized keyword. https://en.wikipedia.org/wiki/Monitor_(synchronization)#Implicit_condition_variable_monitors https://en.wikipedia.org/wiki/Monitor_(synchronization)#Impl...
deleted 4y ago
[deleted]
axilmar 4y ago
Nice pattern, but it only solves the problem of having to use a mutex, which is almost never the problem. Deadlocks are way more nastier to handle. In order to solve deadlocks, I created a mutex that first does a try lock, then if that fails it unlocks all mutexes of the thread and then relocks them in memory order. Not very efficient some times, especially if there are many mutexes involved (i.e maybe it does not scale up), but it allows me to have algorithms not deadlock. Perhaps it could be made more efficient with various tricks, I may have to do research on that idea...
kazinator 4y ago
"Let's completely violate encapsulation by operating on Thing with random lambdas instead of Thing methods---oh but look how nicely the lambdas are forced to run under Thing's lock!" > However, you can’t access the T directly! The only way we’ll let you access the T is by calling with() and passing it a callback that takes a T& parameter. OK, but then, given this: thing->field.with([&](Field& field) { use(field); }); what prevents us from just replacing it with: use(thing->field); and, secondly, how does use() acctually use the field? If access is not allowed without using with, then use() has to use with also, and another callback, leading to infinite regression.

118 points by soheilpro 4y ago

Or you could give `MutexLocker` pointer semantics to access the underlying object only through the lock, which would probably be more idiomatic c++ anyways.

Yoric 4y ago
Lifetimes might be a bit trickier, though.
- chombier 4y ago
  Indeed, with lambdas you don't need to deal with locks outliving the underlying object (though I think std::unique_lock has the same issue with the underlying mutex?) The downside with lambdas is that you cannot lock across function boundaries e.g. give ownership of a lock to the caller. There's a similar tradeoff for all "context managers" (in Python parlance) where you could either CPS your way around the control flow (the lambda approach) or use c++'s RAII mechanism. My impression is that RAII feels a bit more idiomatic and puts the burden on library writers instead of users.
saagarjha 4y ago
Individual accesses are typically the wrong level of abstraction for synchronization. You’ll avoid simple data races but it’s easy to have accidental reentrancy this way.

vore 4y ago

Clang’s static thread safety analysis is pretty good for this too: https://clang.llvm.org/docs/ThreadSafetyAnalysis.html https://clang.llvm.org/docs/ThreadSafetyAnalysis.html

ajtulloch 4y ago

https://github.com/facebook/folly/blob/main/folly/docs/Synchronized.md https://github.com/facebook/folly/blob/main/folly/docs/Synch...

afc 4y ago

I also use this pattern: https://github.com/alefore/edge/blob/master/src/concurrent/protected.h https://github.com/alefore/edge/blob/master/src/concurrent/p... I added an optional template Validator parameter that can be used to double-check invariants on the protected fields. I also declared a Protected with condition subclass. Here is one example of how I use it to implement a "barrier" of sorts (the operation instance blocks in the destructor until execution of all callables passed to "Add" has completed): https://github.com/alefore/edge/blob/master/src/concurrent/operation.h https://github.com/alefore/edge/blob/master/src/concurrent/o... I found it slightly preferable over Abseil annotations; I think it's slightly more robust (i.e., make errors, like forgetting an annotation, less likely) and I like making this explicit in the type system.

Blackthorn 4y ago

Neat idea. Though I think having to pass a lambda for anything you want to do with the fields is awful ergonomics. Maybe instead combine the two ideas. MutexProtected<T> but can be locked with MutexProtected<T>::lock() which returns a MutexLocked<T> object. That object then cleans up the lock when it goes out of scope, and also provides direct access to the enclosed type.

akling 4y ago
Yes, that's a great way to improve the ergonomics of MutexProtected in simple cases! We should totally add that as a complement to the lambda API. :)
- akling 4y ago
  After sketching this out, it's nice that this: auto x = state.with([](auto& state) { return state.x; }); Becomes this: auto x = state.locked()->x; But it also creates this very accessible footgun: auto& x = state.locked()->x; Where it's way too easy to bind a reference to something that should be protected by the lock, but now isn't. So I'm not sure this is a great idea anymore.
  - gpderetta 4y ago
    At the end of the day, this is C++, there are no compiler tracked lifetimes and it is easy to leak references out of the lambda as well. At least the synchronized pattern makes it easy to document which state is protected by which mutex.
  - kajaktum 4y ago
    Wait why is this a footgun?
    - akling 4y ago
      Because now you're holding a reference to `x` which is supposed to be protected by a mutex, even after the mutex is unlocked. With the lambda-only API, it's much harder to make this mistake, since a temporary reference like this will still go out of scope at the end of the lambda expression.

comex 4y ago

This is how Rust’s mutexes work as well - albeit using RAII rather than a callback, but with the same fundamental innovation of putting the data inside the mutex object. I’ve used it in my C++ code as well; it’s a great pattern.

nextaccountic 4y ago
But is there a way to use Rust's approach of RAII guards in C++, rather than accessing the mutex inside a closure?
- gpderetta 4y ago
  Of course. Boost.synchronized and the folly variant have been mentioned elsethread.
- ReactiveJelly 4y ago
  Adding my favorite to the pile: https://github.com/dragazo/rustex https://github.com/dragazo/rustex
tialaramex 4y ago
Crucially the Rust borrow checker means using this wrong from (safe) Rust is impossible, whereas it's relatively easy to use either C++ approach wrongly, especially if nobody has actually sat down and explained why you're doing it. In C++ we can "just" leak the reference to the protected data, in (safe) Rust of course the borrow checker means that programs with this mistake won't build. Rust's ZSTs (~ unit types have zero storage size) make it reasonable to write Mutex<MyMarker> and then require that people show you their MyMarker (which they can only get at by holding the lock) to do stuff thus enforcing locking even when it's not really data you're protecting. Because these are zero size, the resulting machine code is unchanged but the type checking means if you forget to lock first your program won't even build, which shifts locking mistakes very hard left.
- chrisshroba 4y ago
  Can you give an example of using a ZST as a marker in a Mutex? I think I understand that you're suggesting this as a way of locking some section of code even if it's not a specific piece of data that you're locking, but I'm wondering how this could "enforce locking" then, since anything you do while locked could just as easily be accidentally done without the lock, right?

amadvance 4y ago

In truth, I prefer using explicit lock and unlock operations like in C programming. The visibility of these operations aids in understanding the interdependencies within the code.

pavlov 4y ago
I could understand this argument against the RAII-style destructor unlock, but this lambda approach seems actually better in the visibility sense since the mutex access is a separately indented block.
- amadvance 4y ago
  It's obviously subjective, but it looks like as any other indented block. Named "lock" and "unlock" operations are more explicit to me. Another potential disadvantage, is that you lose the possibility to control the unlock order, when you have multiple nested locks.
  - gpderetta 4y ago
    An indented scope (that starts with a scoped_lock or some appropriately named function) is more explicit than lock/unlock calls interspersed in the code. std::scoped_lock allows locking multiple locks.
- snozolli 4y ago
  this lambda approach seems actually better in the visibility sense since the mutex access is a separately indented block. When I'm writing multithreaded code, the first line of any block containing a RAII lock is the lock. Any subsection of code that needs another lock gets its own block.
gpderetta 4y ago
I don't trust explicit lock and un lock operations abstractions in C, I always use inline asm to acquire and release my locks.

eps 4y ago

A pattern for easier slower concurrency as lambdas aren't free. If the code under the mutex is trivial, this cost of using lambdas will be noticeable, especially in fast paths. Might be worth looking at the resulting assembly first if you are looking to adapt this in your codebase.

gpderetta 4y ago
Aside from lambdas being cheap abstractions in C++, any code under a mutex is likely not the fast path anyway.
- saagarjha 4y ago
  I mean, some code is all slow paths ;)
ot 4y ago
Lambdas are free. Unless the class type-erases the callback (and MutexProtected doesn't), and the callback is not huge, the compiler is virtually guaranteed to inline it.
- huhtenberg 4y ago
  It can go both ways. Clang will inline everything - https://godbolt.org/z/v19P1W9sj https://godbolt.org/z/v19P1W9sj MSVC will not, even with O2 - https://godbolt.org/z/naovTxncs https://godbolt.org/z/naovTxncs > Might be worth looking at the resulting assembly first if you are looking to adapt this in your codebase.
akling 4y ago
AFAIK if you use a templated type for the callback (as you should), it will be inlined by modern compilers at any optimization level other than -O0. :)

lunfard000 4y ago

maybe add SerenityOS to the title? Almost sure that lock_guard is the ideomatic way to do this on cpp, not buying the "forget to lock".

Yoric 4y ago
FWIW, I've encountered "forget to lock" in production C++.
saagarjha 4y ago
It’s a general C++ trick.

secondcoming 4y ago

We use something quite similar to this. It won’t save you from accidental recursive locking though, but I’m not sure any solution will.

pavlov 4y ago
If the problem is really just recursive lock calls on the same thread, then PTHREAD_MUTEX_RECURSIVE should do the trick?
- saagarjha 4y ago
  Generally PTHREAD_MUTEX_RECURSIVE is a band-aid, not a solution.
  - AnimalMuppet 4y ago
    Could you explain? A simple drive-by condemnation doesn't tell us why, or give us any reason to agree with you.
    - gpderetta 4y ago
      This gets shared periodically here: https://groups.google.com/g/comp.programming.threads/c/tcrTKnfP8HI/m/me2K7_byNdgJ https://groups.google.com/g/comp.programming.threads/c/tcrTK...
    - secondcoming 4y ago
      If you find yourself needing recursive locks your design is wrong. They were invented as a dare.
    - loeg 4y ago
      They mean you don't know and can't reason about lock state. To design concurrent systems with good performance and correctness, you need to be able to reason about lock state statically. Here's some random article I found in 30 seconds of googling that seems to go into more depth on the subject: https://blog.stephencleary.com/2013/04/recursive-re-entrant-locks.html https://blog.stephencleary.com/2013/04/recursive-re-entrant-...

nly 4y ago

Or...you could use Clangs GUARDED_BY annotations and thread safety analysis... https://clang.llvm.org/docs/ThreadSafetyAnalysis.html https://clang.llvm.org/docs/ThreadSafetyAnalysis.html

pjmlp 4y ago

This is basically one of the ways to do RAII in GC based languages, only provide the resources via lambdas, thus kind of simulating arena like resource managment, the optimizer will take care of inlining them anyway.

gpderetta 4y ago

Another common addition to these lockable objects is having a free function variant of 'with' that allows visiting multiple objects: MutexProtected<X> x; MutexProtected<Y> y; ... with(x, y, [](X& x, Y& y) {....} ); Using the scoped_lock protocol this can avoid deadlocks.

franke2 4y ago

Even if very similar, I prefer the MutexedObj class described in those two articles https://fekir.info/post/extend-generic-thread-safe-mutexed_obj-class/ https://fekir.info/post/extend-generic-thread-safe-mutexed_o... https://fekir.info/post/sharing-data-between-threads/#_bind-the-data-and-mutex-together https://fekir.info/post/sharing-data-between-threads/#_bind-... In particular it explains why providing getters or operator* like some other implementation do (not in this case) is not a feature, and it does not use multiple classes (like the linked implementation of SerenityOS does) making the implementation simpler. Also the "simple" implementation is just 20 lines of code...

einpoklum 4y ago

How does this compare with: * Using std::atomic<T> ? https://en.cppreference.com/w/cpp/atomic/atomic https://en.cppreference.com/w/cpp/atomic/atomic * folly::Synchronized, mentioned by ot57? https://github.com/facebook/folly/blob/main/folly/docs/Synchronized.md https://github.com/facebook/folly/blob/main/folly/docs/Synch... * Boost synchronized data structures, -mentioned by mchicken 53? https://www.boost.org/doc/libs/1_81_0/doc/html/thread/sds.html https://www.boost.org/doc/libs/1_81_0/doc/html/thread/sds.ht... ? ... it looks like the SerenityOS people developed this without considering the C++ ecosystem of today.

gpderetta 4y ago
std::atomic really has different use cases. It is meant to be used for objects that can be modified via atomic RMW primitives and require T to be trivially copyable. While you can in principle implement arbitrary operations on top of those RMWs, it might not be the best fit. For large Ts, is indeed implemented using a mutex or, more likely, a lock pool, so I guess if you squint hard enough it can be considered related.
MontagFTB 4y ago
My understanding is that Serenity has a philosophy of not using dependencies outside of itself. Perhaps this type drew from external inspiration, but was ultimately written in light of that philosophy?
loeg 4y ago
Well, std::atomic is wholly unrelated. MutexProtected seems fundamentally similar to boost and folly synchronized objects, just without the ergonomics and battle-testing the other major libraries have already gotten.

mgaunard 4y ago

If you want easier concurrency, don't use mutexes, they're a bad pattern. It's well-established that the safe way to do concurrency is to use channels.

jstx1 4y ago
What's the problem with mutexes?
- mgaunard 4y ago
  If you have to ask, I suppose you shouldn't touch anything that involves concurrent or parallel execution.
- paulmd 4y ago
  mutexes are inherently single-threaded code, it's exclusive. It's a point where all of your threads have to serialize and run one-at-a-time to perform a certain operation. additionally, common implementations often have bad behavior as the number of threads increases. spinlocks don't work well if you have 32 threads trying to take the same lock, they can consume quite a lot of CPU time just trying to take the lock. they don't work at all if you have thousands of threads (which is a situation that occurs regularly in GPGPU programming!). you can of course increase the number of locks but that's basically what SQL/RDBMS does with row locks - and now you have the problem of deadlock too. obviously it all depends on specifics - how many threads doing how much work on their own, compared to how many locks they're trying to take. it's not that they are inherently bad, they are in fact one of the basic primitives in concurrent programming really. they just don't really scale well with increased concurrency. unfortunately, the deeper solutions involve restructuring your program and your data processing (or the data itself) to expose higher levels of concurrency and less interdependency, which is its own can of worms.
tlb 4y ago
That can work well for many programs, but no one has succeeded in writing a multithreaded kernel that way.

unwind 4y ago

I thought this was cool, and I really appreciate the low-level bottom-up description/introduction of how MutexProtected helps keep things safe. I would have loved also getting a disassembly showing the resulting code, it's very hard for me (as an only-sometimes C++ programmer) to guess how the lambda is compiled.

samsquire 4y ago

I am working on a left-right concurrency control with multiple threads. You shard your data and define a merge operation that is commutative so you can always merge data of different threads. By sharding data by thread each thread can always read and write to its own shard without synchronization or contention. I also have a global snapshot per thread which is the aggregated data across all threads that is local to a thread. So it requires 4× the memory per thread but it is extremely performant due to each thread is independent of every other thread

boricj 4y ago

Another feature of the MutexProtected implementation in SerenityOS is the ability to either obtain an exclusive lock (writable) or a shared lock (read-only). A shared lock provides a const reference to the protected value and therefore enforces read-only access semantics at compile-time by leveraging the C++ type system (specifically, const-correctness). The initial PR introducing the ancestor to MutexProtected has some details on the motivation behind it and also unearthed a bunch of incorrect locking that the C++ compiler caught when introducing it: https://github.com/SerenityOS/serenity/pull/8851 https://github.com/SerenityOS/serenity/pull/8851

mike_hearn 4y ago

Yep, it's a useful pattern albeit not a game changer or anything. I've been using it in Kotlin for a while. Here's an implementation: https://gist.github.com/mikehearn/1913202829403f65331123f0473a7901 https://gist.github.com/mikehearn/1913202829403f65331123f047... One of the nice things about doing it in Kotlin rather than C++ is the clean syntax using trailing lambda blocks and anonymous objects: private val state = Locker(object { var value = 1 var another = "value" }) val nextValue = state.locked { ++value } This works because Kotlin lets you define anonymous types that can take part in type inference even though they're not denotable, and then use them as lambda receivers. So "this" inside the final block points to the unnamed singleton. It has a few nice features: • It's runtime overhead free. Kotlin frontend can inline the "locked" call so you end up with the equivalent of manual locking. • There's no way to get to the state without holding the lock unless you deliberately leak it outside the lambda. • You can control re-entrancy. • It tells the compiler the lambda is only invoked once, which has various benefits for allowing more natural code constructs. These days I'd probably code it with an explicit ReadWriteLock so it's Loom compatible and to allow explicit read/write blocks, but the JVM optimizes locks pretty well so if there's never any contention the memory for the lock is never allocated. I'd also experiment with making it value type so the overhead of the Locker object goes away. But it was never necessary so far.

throwbadubadu 4y ago

> A C++ Pattern for Easier Concurrency Shouldn't downplay sharing that idea, but come on: Realizing Python-like-with-contexts (and yes, even Python isn't the inventor of that..) in C++ with lambdas is a general pattern (think of scenarios like db transactions, or also stuff where RAII pattern would be useful, except that you need a result from the destructor, or it could even fail..) that shouldn't be too new.. Since C++11 when lambdas where introduced, to be precise?

quietbritishjim 4y ago

[edit: deleted first half of my comment - it was just duplication of existing top comment] ----- Even better, when possible (it isn't always), is to avoid using mutexes at all, and instead have the data structure owned by a relevant thread. When you need to access a data structure, pass a message to that thread, and then just access the data structure freely when you get there. (Of course the inter thread queue uses a mutex or some other synchronisation mechanism, and the queue itself effectively acts as a synchronisation mechanism.) struct SomeMsg { SomeMsg() { thread_foo->pass_message(std::bind_front(this, &SomeMsg::handle_foo)); } void handle_foo(Foo& foo) { // ... use foo ... thread_bar->pass_message(std::bind_front(this, &SomeMsg::handle_bar)); } void handle_bar(Bar& bar) { // ... etc. ... } };

smallstepforman 4y ago
You’ve just described Actors. Now take the next step and add a work stealing schedular for a fuller Actor system.
- MontagFTB 4y ago
  I’ve been toying with these lately, too. My solution was to give each actor an executor where it will run: actor<int> foo(stlab::default_executor, “name”, 42); auto f = foo.send([](auto& x){ std::cout << x << ‘\n’; // prints 42 }); Each call to ‘send’ returns a stlab::future which can have a continuation chained, etc. The nice thing about these is the executor based interface is generic. The actor could be dedicated to a thread, run on an OS-provided thread pool, etc. At the time the actor is running, the thread it is on is temporarily given the name of the actor to ease debugging.
- quietbritishjim 4y ago
  Almost. If each thread were an Actor (I think that's what you're saying?) then that would structure the code around each type of work e.g. here is the code for all the database operations. That's not a very useful way to group things for the developer. The way I wrote it above, the code is structured around a single activity across all the types of work it has to do.

planede 4y ago

See also: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p0290r4.html https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p02... This got accepted in C++23.

tialaramex 4y ago
It got added to the Concurrency TS during the C++ 23 process. It's not in standard C++ itself.
- planede 4y ago
  Ah, OK.

pelorat 4y ago

That's not any easier than RAII, although I wish C++ had the lock syntax from C# lock (object) { // do stuff }

gpderetta 4y ago
That's commonly done in C++ with macro, but the lock(object) syntax doesn't prevent doing stuff with object without locking the mutex, while the synchronized pattern does.

thesuperbigfrog 4y ago

MutexProtected is almost Ada protected objects for C++: https://learn.adacore.com/courses/Ada_For_The_CPP_Java_Developer/chapters/11_Concurrency.html#protected-objects https://learn.adacore.com/courses/Ada_For_The_CPP_Java_Devel... This IS a very useful concurrency pattern. Any language that implements concurrent access to shared data should have it or something very similar.

HarHarVeryFunny 4y ago

If you want an mt-safe data structure, then may as well just build it that way in the first place, with a private std::mutex and getters/setters that use std::lock_guard.

29athrowaway 4y ago

How does it handle exceptions?

loeg 4y ago
Internally it uses a MutexLocker which will get destructed as an exception unwinds the stack. The destructor unlocks the mutex. https://github.com/SerenityOS/serenity/blob/master/Kernel/Locking/MutexProtected.h#L55 https://github.com/SerenityOS/serenity/blob/master/Kernel/Lo... https://github.com/SerenityOS/serenity/blob/master/Kernel/Locking/Mutex.h#L146 https://github.com/SerenityOS/serenity/blob/master/Kernel/Lo...

loeg 4y ago

folly::Synchronized<T> provides a similar pattern. It works well for very simplistic concurrency schemes. struct { int a; int b; } A; folly::Synchronized<A> syncA; int c; { auto aReader = syncA.rlock(); c = aReader->a; // aReader going out of scope drops the rlock } { auto aWriter = syncA.wlock(); aWriter->b = 15; // aWriter going out of scope drops the wlock } I would argue that the pseudo-pointer accessor objects are more usable than passing lambdas to a `with()` method, but if you want that, folly::Synchronized provides similar `withWLock()` and `withRLock()` methods. And in fact, MutexProtected seems to provide APIs like Synchronized's `rlock()`/`wlock()`: `lock_shared()` and `lock_exclusive()`: https://github.com/SerenityOS/serenity/blob/master/Kernel/Locking/MutexProtected.h#L58-L59 https://github.com/SerenityOS/serenity/blob/master/Kernel/Lo... I don't know why the article doesn't touch on these. https://github.com/facebook/folly/blob/main/folly/docs/Synchronized.md https://github.com/facebook/folly/blob/main/folly/docs/Synch... https://github.com/facebook/folly/blob/main/folly/Synchronized.h https://github.com/facebook/folly/blob/main/folly/Synchroniz...

PointyFluff 4y ago

[dead]

nayuki 4y ago

The "Barebones example in C" is enshrined in Java at the language level as every object has a built-in lock and condition variable, and locking is done through the synchronized keyword. https://en.wikipedia.org/wiki/Monitor_(synchronization)#Implicit_condition_variable_monitors https://en.wikipedia.org/wiki/Monitor_(synchronization)#Impl...

deleted 4y ago

[deleted]

axilmar 4y ago

Nice pattern, but it only solves the problem of having to use a mutex, which is almost never the problem. Deadlocks are way more nastier to handle. In order to solve deadlocks, I created a mutex that first does a try lock, then if that fails it unlocks all mutexes of the thread and then relocks them in memory order. Not very efficient some times, especially if there are many mutexes involved (i.e maybe it does not scale up), but it allows me to have algorithms not deadlock. Perhaps it could be made more efficient with various tricks, I may have to do research on that idea...

kazinator 4y ago

"Let's completely violate encapsulation by operating on Thing with random lambdas instead of Thing methods---oh but look how nicely the lambdas are forced to run under Thing's lock!" > However, you can’t access the T directly! The only way we’ll let you access the T is by calling with() and passing it a callback that takes a T& parameter. OK, but then, given this: thing->field.with([&](Field& field) { use(field); }); what prevents us from just replacing it with: use(thing->field); and, secondly, how does use() acctually use the field? If access is not allowed without using with, then use() has to use with also, and another callback, leading to infinite regression.