3 ms·

Mutex lock/unlock 25ns Main memory reference 100ns Honest question: how can this be right? Surely living a mutex requires synchronising across the c

Mutex lock/unlock 25ns Main memory reference 100ns Honest question: how can this be right? Surely living a mutex requires synchronising across the cores of a CPU, which requires at least as much time – probably quite a bit more – than an uncached access to memory?

dragontamer 6y ago
Mutex lock/unlock is commonly over L3 cache on modern systems. It won't hit main-memory.
chrisseaton 6y ago
> Surely living a mutex requires synchronising across the cores of a CPU Right it's done over cache coherence, which doesn't need to reach all the way out to main memory. And that's only if it's contested.
- dragontamer 6y ago
  A contested mutex lock / unlock is over L3 cache. Uncontested mutex lock/unlock is just a swap instruction followed (or in "unlock": preceded) by a memory barrier. The flush pushes data to L3 cache, where it can be shared between multiple cores. (L1 and L2 cache is local to a core). L3 cache is far closer than main-memory, and roughly 20ns these days. ------- In "actuality", its more of a MESI message, but in the abstract its your L3 caches communicating with each other and synchronizing.
  - chrisseaton 6y ago
    Not sure if you're agreeing with me or contradicting me? Isn't that what I said?
GeneralMayhem 6y ago
Funnily enough, there is another version of the slides these numbers come from in which mutex lock/unlock is listed as the same 100ns as memory access: http://static.googleusercontent.com/media/research.google.com/en/us/people/jeff/stanford-295-talk.pdf http://static.googleusercontent.com/media/research.google.co.... I can't tell which came first, or if one is a typo, or what's going on there. In any case, assuming the mutex in cache, and appears to not currently be locked, the core performing the atomic operation needs to: (1) broadcast an invalidate/exclusive hold on the cache line to the other cores, (2) update its own cache, (3) eventually write back to memory. (1) is faster than main memory access, since CPU cores are closer to each other, physically, than they are to the memory bus, and have hard-wired access specifically for this operation. The slowest part is (3), but with a write-back cache the writer doesn't really pay the latency cost, because it gets amortized into the next cache flush.
- quietbritishjim 6y ago
  Interesting, thanks. The idea of it just hitting the shared cache had occurred to me, but somehow locking a mutex seemed like a complex enough operation that it would surely take more time overall than a full memory access. I'm glad to be corrected about it.

dragontamer 6y ago

Mutex lock/unlock is commonly over L3 cache on modern systems. It won't hit main-memory.

chrisseaton 6y ago

> Surely living a mutex requires synchronising across the cores of a CPU Right it's done over cache coherence, which doesn't need to reach all the way out to main memory. And that's only if it's contested.

dragontamer 6y ago
A contested mutex lock / unlock is over L3 cache. Uncontested mutex lock/unlock is just a swap instruction followed (or in "unlock": preceded) by a memory barrier. The flush pushes data to L3 cache, where it can be shared between multiple cores. (L1 and L2 cache is local to a core). L3 cache is far closer than main-memory, and roughly 20ns these days. ------- In "actuality", its more of a MESI message, but in the abstract its your L3 caches communicating with each other and synchronizing.
- chrisseaton 6y ago
  Not sure if you're agreeing with me or contradicting me? Isn't that what I said?

GeneralMayhem 6y ago

Funnily enough, there is another version of the slides these numbers come from in which mutex lock/unlock is listed as the same 100ns as memory access: http://static.googleusercontent.com/media/research.google.com/en/us/people/jeff/stanford-295-talk.pdf http://static.googleusercontent.com/media/research.google.co.... I can't tell which came first, or if one is a typo, or what's going on there. In any case, assuming the mutex in cache, and appears to not currently be locked, the core performing the atomic operation needs to: (1) broadcast an invalidate/exclusive hold on the cache line to the other cores, (2) update its own cache, (3) eventually write back to memory. (1) is faster than main memory access, since CPU cores are closer to each other, physically, than they are to the memory bus, and have hard-wired access specifically for this operation. The slowest part is (3), but with a write-back cache the writer doesn't really pay the latency cost, because it gets amortized into the next cache flush.

quietbritishjim 6y ago
Interesting, thanks. The idea of it just hitting the shared cache had occurred to me, but somehow locking a mutex seemed like a complex enough operation that it would surely take more time overall than a full memory access. I'm glad to be corrected about it.