3 ms·

I'm curious what is going on with the "Idle Time" measurements. Does anyone have any theories? My guess is that if the thread gets descheduled inside the lock(

I'm curious what is going on with the "Idle Time" measurements. Does anyone have any theories? My guess is that if the thread gets descheduled inside the lock() spin loop (either by an explicit call to yield() or otherwise) then the "Four longest idle times" reflect the fact that all of the worker threads have remained descheduled for an extended period. But assuming that the threads have elevated priority (why would you use spinlocks on non-high-priority threads?) this seems implausible on an 8 core machine. I would be curious to know what the results are if isolcpus was used and the workers were explicitly assigned to those CPUs.

atq2119 7y ago
The whole "idle time" vs. "longest wait" distinction is suspect and it's surprising they don't go to more effort to justify it. What "longest wait" measures is (b_i - a_i) in: Time a_0 lock Time b_0 unlock Time a_1 lock Time b_1 unlock ... What "idle time" measures is (b_i - a_i) in: lock Time a_0 unlock lock Time b_0 Time a_1 unlock lock Time b_1 ... At least for the spinlock variants, "unlock" should be an extremely cheap operation (just a store for most variants), which means that those two sequences ought to be essentially identical in terms of timing: it ought to be possible to move the "Time a_i"s across the "unlock" operations with a timing impact less than a microsecond, i.e. the two sequences ought to have essentially the same timings. The fact that the timings are supposedly different on a millisecond scale is therefore quite suspicious. Perhaps the chosen clock ends up calling a syscall, which can end up in the scheduler on syscall return occasionally and throw off the measurements that way? Or just the fact that there's one extra syscall in the critical section sometimes causes other threads to make different yielding decisions which throws off the scheduling? More generally, it would have been great to see a deeper investigation into scheduling behavior, which is possible on Linux using the tracing infrastructure and visualizing with different tools like Hotspot. Getting a worst-case delay of ~1.5ms with a naive ticket_spinlock that can yield is really not surprising, to be honest. All it takes is for patterns like: 1. 3 threads pull their tickets in order. 2. The middle thread yields and something else is chosen to run by the scheduler, or perhaps the scheduler decides not to run anything for a short while on the off-chance that something else becomes runnable, which is a fair choice because after all, you yielded. 3. The last thread is then blocked by the middle thread. It just boils down to the fact that "yield" is a bad primitive. You should fall back to futex-based synchronization instead so that the kernel has a better idea of what you're trying to do. The article comes to the same conclusion anyway, in the form of "just use std::mutex".
- RossBencina 7y ago
  It's not entirely clear from the article, but my understanding was that "longest wait" maintains per-thread timestamps, i.e. the time to lock() within a single thread, whereas "idle time" uses a single global atomic timestamp that measures the time from when the last thread unlocks until the next thread locks -- even if the unlock() and subsequent lock() are on different threads.
shaklee3 7y ago
Even with isolcpus, there are some kernel threads that will preempt user threads (kworker/rcuos/softirq) unless you set your priority higher than those with sched_fifo/rr

atq2119 7y ago

The whole "idle time" vs. "longest wait" distinction is suspect and it's surprising they don't go to more effort to justify it. What "longest wait" measures is (b_i - a_i) in: Time a_0 lock Time b_0 unlock Time a_1 lock Time b_1 unlock ... What "idle time" measures is (b_i - a_i) in: lock Time a_0 unlock lock Time b_0 Time a_1 unlock lock Time b_1 ... At least for the spinlock variants, "unlock" should be an extremely cheap operation (just a store for most variants), which means that those two sequences ought to be essentially identical in terms of timing: it ought to be possible to move the "Time a_i"s across the "unlock" operations with a timing impact less than a microsecond, i.e. the two sequences ought to have essentially the same timings. The fact that the timings are supposedly different on a millisecond scale is therefore quite suspicious. Perhaps the chosen clock ends up calling a syscall, which can end up in the scheduler on syscall return occasionally and throw off the measurements that way? Or just the fact that there's one extra syscall in the critical section sometimes causes other threads to make different yielding decisions which throws off the scheduling? More generally, it would have been great to see a deeper investigation into scheduling behavior, which is possible on Linux using the tracing infrastructure and visualizing with different tools like Hotspot. Getting a worst-case delay of ~1.5ms with a naive ticket_spinlock that can yield is really not surprising, to be honest. All it takes is for patterns like: 1. 3 threads pull their tickets in order. 2. The middle thread yields and something else is chosen to run by the scheduler, or perhaps the scheduler decides not to run anything for a short while on the off-chance that something else becomes runnable, which is a fair choice because after all, you yielded. 3. The last thread is then blocked by the middle thread. It just boils down to the fact that "yield" is a bad primitive. You should fall back to futex-based synchronization instead so that the kernel has a better idea of what you're trying to do. The article comes to the same conclusion anyway, in the form of "just use std::mutex".

RossBencina 7y ago
It's not entirely clear from the article, but my understanding was that "longest wait" maintains per-thread timestamps, i.e. the time to lock() within a single thread, whereas "idle time" uses a single global atomic timestamp that measures the time from when the last thread unlocks until the next thread locks -- even if the unlock() and subsequent lock() are on different threads.

shaklee3 7y ago

Even with isolcpus, there are some kernel threads that will preempt user threads (kworker/rcuos/softirq) unless you set your priority higher than those with sched_fifo/rr