5 ms·

This is an excellent comment that clearly comes from a lot of experience. Can you say any more about why having multiple locks is such a red flag? Is it because

This is an excellent comment that clearly comes from a lot of experience. Can you say any more about why having multiple locks is such a red flag? Is it because you start getting into deadlock/hungry philosophers territory? Are there rules of thumb for using multiple locks that make it more tractable or is it just always a bad idea?

ntoskrnl 4y ago
The rule of thumb for multiple locks is always acquire them in the same order. Lock A, lock B, unlock B, unlock A. Otherwise you risk a deadlock. If another thread locks B, then tries to lock A, you're in trouble.
- mandevil 4y ago
  And keeping that discipline over many developers, over many years, is in practice impossible, which is why multiple locks seem attractive but are actually a major problem for complex systems.
  - layer8 4y ago
    You can, of course, encapsulate such a lock order with dedicated classes (or functions/closures) modeling linear types enforcing the sequence of lock A > lock B > unlock B > unlock A. The resulting objects can still be misapplied, but should fail immediately with a runtime error, making it much harder to inadvertently misuse.
    - jerf 4y ago
      But that still doesn't scale up to more locks of that size very well, because you start trying to take even more of them, and in dynamic orders, and etc etc. I know about the theoretical solution of taking locks always in a defined order, but I don't consider it a practical solution. If taking locks willy-nilly lets you scale to program size X, this might give you 2X or 3X, but that's not nearly large enough in practice. & dandelany, the community has chipped in and answered for me. I endorse this subthread, at least, as I write this comment. :)
      - dandelany 4y ago
        love when that happens :) Thanks!
      - anarazel 4y ago
        IME there's plenty problems where nested locks will lead to a simpler solution than trying to avoid them. It's important to avoid when reasonably doable, but it's also not uncommon to take it too far and end up with a slower and way more complicated solution.
    - mandevil 4y ago
      The problem here is when you have more than 2, and especially as it combinatorially expands as you get more and more locks. Now I need Lock A, Lock C, and Lock F, but then this other critical section needs Lock A, Lock D, and Lock F, and if you are encapsulating then you start needing a whole lot of different functions for all the different combinations, and have to make sure that all are consistent. I've got multi threaded C++ in production right now, and it is because the code is small and avoids multiple locks that I have any faith in it. As a code base expands, in kloc and in time, this gets harder and harder to reason about.
- astrange 4y ago
  You can break this on a live system if you're in a debugger and manually lock one of the inner locks. danluu claims there's other reasons it's not good enough but I was never able to figure out what he meant. Exception safety maybe.
dgb23 4y ago
Not OP, but that might be an implication. Another, more general reason is that this kind of concurrency requires design. It can’t be thought of as a defensive programming technique. Locks are too powerful and they are by definition a point of contention. I think providing a lock is like giving up control to someone else. It’s inherent coupling. To contrast, when I first learned about them I thought of them as a protection mechanism of an isolated part, but they are an execution contract that everyone participates in.
- jerf 4y ago
  "It’s inherent coupling." This matches well with one of the beliefs I've been polishing up over the past few years which I don't think is well understood, which is that structured programming tends to create more coupling than we realize in deep call stacks. When you end up with a 2000-line call stack in Java, that's 2000 function invocations coupled together by the way structured programming works; for instance, at a bare minimum, that's 1999 functions that can't progress until the innermost one does, and that's not the only form of coupling. I think the coupling induced by structured programming is fairly light per stack frame on average, but they tend to add up quickly because they're just so darned easy to add more of. Locks are another thing that makes this really come to light. The more stack frames between some function and something above it in the stack that has taken a lock, the more likely it is for execution to eventually wander back into something trying to take a lock inadvisably. I mean, I've had a number of deadlocks just within an object where it has some internal lock, and a method takes that lock, then tries to call a method that takes the same lock. Whoops. Easy to fix, of course, but that's the easy case. The cases get harder in real code.
  - dgb23 4y ago
    So the stack is a 'lie'? Or the 'wrong' data structure? It isn't just caller knows about callee, there's cross frame and cross stack coupling going on. There is something interesting in your idea.
    - jerf 4y ago
      I don't think it's a lie. It's very, very useful. It did not survive as long as it did without providing value. It killed its competition stone dead for a reason. I just think we underestimate the coupling it introduces. Doesn't mean the solution is to throw it all out. I'm not even sure there is "a solution". Just something to keep in mind.

ntoskrnl 4y ago

The rule of thumb for multiple locks is always acquire them in the same order. Lock A, lock B, unlock B, unlock A. Otherwise you risk a deadlock. If another thread locks B, then tries to lock A, you're in trouble.

mandevil 4y ago
And keeping that discipline over many developers, over many years, is in practice impossible, which is why multiple locks seem attractive but are actually a major problem for complex systems.
- layer8 4y ago
  You can, of course, encapsulate such a lock order with dedicated classes (or functions/closures) modeling linear types enforcing the sequence of lock A > lock B > unlock B > unlock A. The resulting objects can still be misapplied, but should fail immediately with a runtime error, making it much harder to inadvertently misuse.
  - jerf 4y ago
    But that still doesn't scale up to more locks of that size very well, because you start trying to take even more of them, and in dynamic orders, and etc etc. I know about the theoretical solution of taking locks always in a defined order, but I don't consider it a practical solution. If taking locks willy-nilly lets you scale to program size X, this might give you 2X or 3X, but that's not nearly large enough in practice. & dandelany, the community has chipped in and answered for me. I endorse this subthread, at least, as I write this comment. :)
    - dandelany 4y ago
      love when that happens :) Thanks!
    - anarazel 4y ago
      IME there's plenty problems where nested locks will lead to a simpler solution than trying to avoid them. It's important to avoid when reasonably doable, but it's also not uncommon to take it too far and end up with a slower and way more complicated solution.
  - mandevil 4y ago
    The problem here is when you have more than 2, and especially as it combinatorially expands as you get more and more locks. Now I need Lock A, Lock C, and Lock F, but then this other critical section needs Lock A, Lock D, and Lock F, and if you are encapsulating then you start needing a whole lot of different functions for all the different combinations, and have to make sure that all are consistent. I've got multi threaded C++ in production right now, and it is because the code is small and avoids multiple locks that I have any faith in it. As a code base expands, in kloc and in time, this gets harder and harder to reason about.
astrange 4y ago
You can break this on a live system if you're in a debugger and manually lock one of the inner locks. danluu claims there's other reasons it's not good enough but I was never able to figure out what he meant. Exception safety maybe.

dgb23 4y ago

Not OP, but that might be an implication. Another, more general reason is that this kind of concurrency requires design. It can’t be thought of as a defensive programming technique. Locks are too powerful and they are by definition a point of contention. I think providing a lock is like giving up control to someone else. It’s inherent coupling. To contrast, when I first learned about them I thought of them as a protection mechanism of an isolated part, but they are an execution contract that everyone participates in.

jerf 4y ago
"It’s inherent coupling." This matches well with one of the beliefs I've been polishing up over the past few years which I don't think is well understood, which is that structured programming tends to create more coupling than we realize in deep call stacks. When you end up with a 2000-line call stack in Java, that's 2000 function invocations coupled together by the way structured programming works; for instance, at a bare minimum, that's 1999 functions that can't progress until the innermost one does, and that's not the only form of coupling. I think the coupling induced by structured programming is fairly light per stack frame on average, but they tend to add up quickly because they're just so darned easy to add more of. Locks are another thing that makes this really come to light. The more stack frames between some function and something above it in the stack that has taken a lock, the more likely it is for execution to eventually wander back into something trying to take a lock inadvisably. I mean, I've had a number of deadlocks just within an object where it has some internal lock, and a method takes that lock, then tries to call a method that takes the same lock. Whoops. Easy to fix, of course, but that's the easy case. The cases get harder in real code.
- dgb23 4y ago
  So the stack is a 'lie'? Or the 'wrong' data structure? It isn't just caller knows about callee, there's cross frame and cross stack coupling going on. There is something interesting in your idea.
  - jerf 4y ago
    I don't think it's a lie. It's very, very useful. It did not survive as long as it did without providing value. It killed its competition stone dead for a reason. I just think we underestimate the coupling it introduces. Doesn't mean the solution is to throw it all out. I'm not even sure there is "a solution". Just something to keep in mind.