5 ms·

This article is pretty old and I suspect if you asked Bartosz now he'd explain it slightly differently. You certainly need to use instructions that impose addit

This article is pretty old and I suspect if you asked Bartosz now he'd explain it slightly differently. You certainly need to use instructions that impose additional ordering guarantees on x86 in these kinds of situations but you don't need an mfence and in general it will be slower than an appropriate locked instruction. The appropriate uses of mfence are actually much more limited, it's only really needed with special memory types (e.g. write combined) or when you need ordering on certain non temporal loads and stores (mostly vector instructions) AFAIK. In regular code you should never require the hammer of mfence for correct synchronization. You can implement C++11 atomics without it.

johnbender 12y ago
It doesn't look like the LOCK prefix applies to MOV (from a quick google)? So how does it address write buffering or OOE for stores in a TSO memory model? [edit] "never require the hammer of mfence for correct synchronization", maybe you're confining this to correct synch. and not recovering sequential consistency (or some other semantic property).
- mattnewport 12y ago
  It's a little while since I looked closely at this and it's easy to get this stuff wrong but here's what I concluded when digging into this in the context of a codebase I was maintaining that made heavy use of lock free techniques: - You don't need mfence (or sfence / lfence) on x64 (which is actually what I care about rather than x86, though they're essentially the same in this respect) to correctly implement C++11 atomics with acquire / release semantics. You can get all the guarantees you need with locked instructions. - Correctly implemented C++11 atomics implemented with locked instructions will be as fast or faster than when implemented with explicit fence instructions. - The obvious way to implement standalone fences on x64 is with fence instructions but you rarely if ever need standalone fences. Generally you are better off using atomic operations with explicit acquire / release semantics. In the codebase I was maintaining, the pre-C++11 atomic library was based around explicit fences rather than C++11 style atomic operations with acquire / release semantics attached to the operations themselves. This code was primarily written for Gen 3 consoles (PS3 / Xbox 360) and so was optimized for PowerPC. On x64 (Gen 4 consoles!) there was measurable performance overhead due to unnecessary/redundant standalone fences. We decided it was too risky to try and rewrite everything in terms of atomic operations with acquire release semantics and remove the standalone fences in the end but it seems to me that if you want to write efficient cross architecture lock free code you want to avoid standalone fences and use a C++11 style atomics library where acquire release semantics are tied to the atomic operations themselves.
- mattnewport 12y ago
  To answer the first part of your question, a locked xchg is the usual way to implement a sequentially consistent store on x86 without an explicit fence. For sequentially consistent loads, plain mov suffices, providing it synchronizes with a locked store. http://stackoverflow.com/questions/4972106/sequentially-consistent-atomic-load-on-x86 http://stackoverflow.com/questions/4972106/sequentially-cons...
amluto 12y ago
> The appropriate uses of mfence are actually much more limited, it's only really needed with special memory types (e.g. write combined) or when you need ordering on certain non temporal loads and stores (mostly vector instructions) AFAIK. You may be thinking of sfence here, not mfence. > In regular code you should never require the hammer of mfence for correct synchronization. You can implement C++11 atomics without it. That's true, but I think that, in many circumstances, mfence is cheaper than atomic RMW instructions. EDIT: Maybe I'm wrong. lock addl to the stack is faster than mfence on my laptop, but not that much faster. This is strange, because in most cases you can replace mfence with a locked access to an unrelated memory location. sfence is much faster than either.
- pkhuong 12y ago
  re sfence: standard x86 memory ordering forbids store/store reordering anyway.
  - amluto 12y ago
    sfence is only useful for WC memory and non-temporal stores. I'm not sure what lfence is for other than use in conjuction with rdtsc and possibly IO.
davidtgoldblatt 12y ago
The relative performance of a lock'd instruction and an mfence will vary across processors and workloads. On my machine, simple peterson locking is ~10% faster when using an mfence as opposed to an xchgl. Maybe if the non-locked code benefited more from pipelining, the results would be different. As an aside, g++ uses mfence for its C++11 atomics while clang uses xchgl; Does anyone know if Intel/AMD has ever guaranteed that all the usual ordering guarantees apply when mixing these? (Of course given their implementations it almost certainly will, but it would be nice to know this for a fact).
redraga 12y ago
>The appropriate uses of mfence are actually much more limited This is true for regular race-free multithreaded programs. However, mfences are used fairly commonly in lock-free, concurrent data structures (http://concurrencykit.org/ http://concurrencykit.org/)
- pkhuong 12y ago
  If you see code compile to a full mfence in CK, please tell us. In my experience, there is no concurrency-related related reason to use mfence on x86. In fact, you'll see that we have code that checks for x86oids and use an atomic RMW instead of non-atomic store/RMW + fence.

johnbender 12y ago

It doesn't look like the LOCK prefix applies to MOV (from a quick google)? So how does it address write buffering or OOE for stores in a TSO memory model? [edit] "never require the hammer of mfence for correct synchronization", maybe you're confining this to correct synch. and not recovering sequential consistency (or some other semantic property).

mattnewport 12y ago
It's a little while since I looked closely at this and it's easy to get this stuff wrong but here's what I concluded when digging into this in the context of a codebase I was maintaining that made heavy use of lock free techniques: - You don't need mfence (or sfence / lfence) on x64 (which is actually what I care about rather than x86, though they're essentially the same in this respect) to correctly implement C++11 atomics with acquire / release semantics. You can get all the guarantees you need with locked instructions. - Correctly implemented C++11 atomics implemented with locked instructions will be as fast or faster than when implemented with explicit fence instructions. - The obvious way to implement standalone fences on x64 is with fence instructions but you rarely if ever need standalone fences. Generally you are better off using atomic operations with explicit acquire / release semantics. In the codebase I was maintaining, the pre-C++11 atomic library was based around explicit fences rather than C++11 style atomic operations with acquire / release semantics attached to the operations themselves. This code was primarily written for Gen 3 consoles (PS3 / Xbox 360) and so was optimized for PowerPC. On x64 (Gen 4 consoles!) there was measurable performance overhead due to unnecessary/redundant standalone fences. We decided it was too risky to try and rewrite everything in terms of atomic operations with acquire release semantics and remove the standalone fences in the end but it seems to me that if you want to write efficient cross architecture lock free code you want to avoid standalone fences and use a C++11 style atomics library where acquire release semantics are tied to the atomic operations themselves.
mattnewport 12y ago
To answer the first part of your question, a locked xchg is the usual way to implement a sequentially consistent store on x86 without an explicit fence. For sequentially consistent loads, plain mov suffices, providing it synchronizes with a locked store. http://stackoverflow.com/questions/4972106/sequentially-consistent-atomic-load-on-x86 http://stackoverflow.com/questions/4972106/sequentially-cons...

amluto 12y ago

> The appropriate uses of mfence are actually much more limited, it's only really needed with special memory types (e.g. write combined) or when you need ordering on certain non temporal loads and stores (mostly vector instructions) AFAIK. You may be thinking of sfence here, not mfence. > In regular code you should never require the hammer of mfence for correct synchronization. You can implement C++11 atomics without it. That's true, but I think that, in many circumstances, mfence is cheaper than atomic RMW instructions. EDIT: Maybe I'm wrong. lock addl to the stack is faster than mfence on my laptop, but not that much faster. This is strange, because in most cases you can replace mfence with a locked access to an unrelated memory location. sfence is much faster than either.

pkhuong 12y ago
re sfence: standard x86 memory ordering forbids store/store reordering anyway.
- amluto 12y ago
  sfence is only useful for WC memory and non-temporal stores. I'm not sure what lfence is for other than use in conjuction with rdtsc and possibly IO.

davidtgoldblatt 12y ago

The relative performance of a lock'd instruction and an mfence will vary across processors and workloads. On my machine, simple peterson locking is ~10% faster when using an mfence as opposed to an xchgl. Maybe if the non-locked code benefited more from pipelining, the results would be different. As an aside, g++ uses mfence for its C++11 atomics while clang uses xchgl; Does anyone know if Intel/AMD has ever guaranteed that all the usual ordering guarantees apply when mixing these? (Of course given their implementations it almost certainly will, but it would be nice to know this for a fact).

redraga 12y ago

>The appropriate uses of mfence are actually much more limited This is true for regular race-free multithreaded programs. However, mfences are used fairly commonly in lock-free, concurrent data structures (http://concurrencykit.org/ http://concurrencykit.org/)

pkhuong 12y ago
If you see code compile to a full mfence in CK, please tell us. In my experience, there is no concurrency-related related reason to use mfence on x86. In fact, you'll see that we have code that checks for x86oids and use an atomic RMW instead of non-atomic store/RMW + fence.