4 ms·

I've found message passing and thinking in transactions are pretty good architectural patterns for maintaining ones sanity. Message queues with spinlocks for pe

by fsloth 9y ago

I've found message passing and thinking in transactions are pretty good architectural patterns for maintaining ones sanity. Message queues with spinlocks for performance critical code (mutexes are slow). The biggest sin generally is to disregard the overhead caused by thread management and thinking more threads makes the sofrware run faster. I've seen people try to parallellize a sequential program by just spawning mutexes everywhere and then thinking now any number of threads can do whatever they please. Of course when tested the system was quite a bit slower as when it ran on a single thread (the system was quite large so quite a lot of work was needed before reaching this state).

kbwt 9y ago
> mutexes are slow Be careful with that. First off, what people refer to as "mutex" is usually a spinlock that falls back to a kernel wait queue when the spin count is exceeded. There are even adaptive mutexes that figure out at runtime how long the lock is typically held and base their spin count limit on that. Secondly, busy-waiting is often worse than a single slow program, because you actively slow down all of the other running programs.
ajross 9y ago
> spinlocks for performance critical code (mutexes are slow). Gah, no. Userspace spinlocks are the deepest of voodoo and something to be used only by people who know exactly what they are doing and would have no difficulty writing "traditional" threaded code in a C/C++ environment. Among other problems: what happens when the thread holding the spinlock gets preempted and something else runs on the core? How can you prevent that from happening, and how does that collision probability scale with thread count and lock behavior? Traditional locking (e.g. pthread mutexes, windows CriticalSections) in a shared memory environment can be done with atomic operations only for the uncontended case, and will fall back to the kernel to provide blocking/wakeup in a clean and scalable way. Use that. Don't go further unless you're trying to do full-system optimization on known hardware and have a team full of benchmark analysis experts to support the effort.
- jcelerier 9y ago
  > Gah, no. Userspace spinlocks are the deepest of voodoo and something to be used only by people who know exactly what they are doing and would have no difficulty writing "traditional" threaded code in a C/C++ environment. If you ever used pthread_mutex with glibc then you use spinlocks without knowing it. The implementation spins for some time before going for a full kernel mutex.
  - ajross 9y ago
    Sure, and C library authors qualify. App developers don't.
  - mannykannot 9y ago
    I think the warning is directed against rolling your own spinlocks without careful consideration, including a realistic evaluation of whether you know every issue involved in doing so.
  - whatidonteven 9y ago
    There's a bit of a terminology confusion here. When they say "spinlock", most people mean a pure spinlock that spins forever until it succeeds. "Mutex" on the other hand might have a fast-path that spins a few times before inserting the thread onto a wait list.
noobermin 9y ago
Is there a good message passing library for C++? Sounds like something for the stl to include.
- danbolt 9y ago
  I know coworkers that speak well of ZeroMQ: http://zeromq.org http://zeromq.org
- shaklee3 9y ago
  Boost has one, but not the stl
- jcelerier 9y ago
  > Is there a good message passing library for C++? Sounds like something for the stl to include. Qt does it with signals-slots. What I generally do is that I have a queue of std::function and just pass lambdas with the capture being copied.
exDM69 9y ago
An uncontended mutex is just as fast as a spinlock (on modern operating systems using a futex). It takes about 25 nanoseconds to lock it. The difference is when there's contention. A spinlock will burn CPU cycles but a mutex will yield to another thread or process (with some context switch overhead). A spinlock should only be used when you know you're going to get it in the next microsecond or so. Or in kernel space when you don't have other options (e.g. interrupt handler). Anything else is just burning CPU cycles for nothing. Mutex and condition variables (emphasis on the latter) are much more useful than spinlocks and atomics for general multithreaded programming.
- oppositelock 9y ago
  You have to be careful with those things - for instance, you have to special case whether you'r on a single CPU or multiple CPU system, because a spin lock will block forever in a non-preemptive context, such as the kernel. Outside of hard realtime code, there's zero reason to use spin locks.
  - kbwt 9y ago
    In hard realtime code, a FIFO mutex gives you bounded wait time because you have a fixed number of threads. Interesting read: http://www2.rdrop.com/~paulmck/realtime/SMPembedded.2006.10.09b.pdf http://www2.rdrop.com/~paulmck/realtime/SMPembedded.2006.10....
  - gpderetta 9y ago
    > Outside of hard realtime code, there's zero reason to use spin locks. On some common architectures, releasing a spin lock is cheaper than releasing a mutex.
    - comex 9y ago
      On all architectures, releasing a mutex requires at least a branch (to see if you need to wake up sleeping threads) that you don’t need with a pure spinlock. But if you don’t have a guarantee the lock owner won’t be preempted, well, spinning for a whole timeslot is quite a bit more expensive…
LeoNatan25 9y ago
Using spinlocks can cause livelocks under certain conditions due to priority inversion.

by fsloth 9y ago

kbwt 9y ago

> mutexes are slow Be careful with that. First off, what people refer to as "mutex" is usually a spinlock that falls back to a kernel wait queue when the spin count is exceeded. There are even adaptive mutexes that figure out at runtime how long the lock is typically held and base their spin count limit on that. Secondly, busy-waiting is often worse than a single slow program, because you actively slow down all of the other running programs.

ajross 9y ago

> spinlocks for performance critical code (mutexes are slow). Gah, no. Userspace spinlocks are the deepest of voodoo and something to be used only by people who know exactly what they are doing and would have no difficulty writing "traditional" threaded code in a C/C++ environment. Among other problems: what happens when the thread holding the spinlock gets preempted and something else runs on the core? How can you prevent that from happening, and how does that collision probability scale with thread count and lock behavior? Traditional locking (e.g. pthread mutexes, windows CriticalSections) in a shared memory environment can be done with atomic operations only for the uncontended case, and will fall back to the kernel to provide blocking/wakeup in a clean and scalable way. Use that. Don't go further unless you're trying to do full-system optimization on known hardware and have a team full of benchmark analysis experts to support the effort.

jcelerier 9y ago
> Gah, no. Userspace spinlocks are the deepest of voodoo and something to be used only by people who know exactly what they are doing and would have no difficulty writing "traditional" threaded code in a C/C++ environment. If you ever used pthread_mutex with glibc then you use spinlocks without knowing it. The implementation spins for some time before going for a full kernel mutex.
- ajross 9y ago
  Sure, and C library authors qualify. App developers don't.
- mannykannot 9y ago
  I think the warning is directed against rolling your own spinlocks without careful consideration, including a realistic evaluation of whether you know every issue involved in doing so.
- whatidonteven 9y ago
  There's a bit of a terminology confusion here. When they say "spinlock", most people mean a pure spinlock that spins forever until it succeeds. "Mutex" on the other hand might have a fast-path that spins a few times before inserting the thread onto a wait list.

noobermin 9y ago

Is there a good message passing library for C++? Sounds like something for the stl to include.

danbolt 9y ago
I know coworkers that speak well of ZeroMQ: http://zeromq.org http://zeromq.org
shaklee3 9y ago
Boost has one, but not the stl
jcelerier 9y ago
> Is there a good message passing library for C++? Sounds like something for the stl to include. Qt does it with signals-slots. What I generally do is that I have a queue of std::function and just pass lambdas with the capture being copied.

exDM69 9y ago

An uncontended mutex is just as fast as a spinlock (on modern operating systems using a futex). It takes about 25 nanoseconds to lock it. The difference is when there's contention. A spinlock will burn CPU cycles but a mutex will yield to another thread or process (with some context switch overhead). A spinlock should only be used when you know you're going to get it in the next microsecond or so. Or in kernel space when you don't have other options (e.g. interrupt handler). Anything else is just burning CPU cycles for nothing. Mutex and condition variables (emphasis on the latter) are much more useful than spinlocks and atomics for general multithreaded programming.

oppositelock 9y ago
You have to be careful with those things - for instance, you have to special case whether you'r on a single CPU or multiple CPU system, because a spin lock will block forever in a non-preemptive context, such as the kernel. Outside of hard realtime code, there's zero reason to use spin locks.
- kbwt 9y ago
  In hard realtime code, a FIFO mutex gives you bounded wait time because you have a fixed number of threads. Interesting read: http://www2.rdrop.com/~paulmck/realtime/SMPembedded.2006.10.09b.pdf http://www2.rdrop.com/~paulmck/realtime/SMPembedded.2006.10....
- gpderetta 9y ago
  > Outside of hard realtime code, there's zero reason to use spin locks. On some common architectures, releasing a spin lock is cheaper than releasing a mutex.
  - comex 9y ago
    On all architectures, releasing a mutex requires at least a branch (to see if you need to wake up sleeping threads) that you don’t need with a pure spinlock. But if you don’t have a guarantee the lock owner won’t be preempted, well, spinning for a whole timeslot is quite a bit more expensive…

LeoNatan25 9y ago

Using spinlocks can cause livelocks under certain conditions due to priority inversion.