5 ms·
A good, introductory, high-level overview of what is going on with cache coherence... albeit specific to x86. ARM systems are more relaxed, and therefore need
by dragontamer 7y ago
A good, introductory, high-level overview of what is going on with cache coherence... albeit specific to x86.
ARM systems are more relaxed, and therefore need more barriers than on x86. Memory barriers (which also function as "compiler barriers" for the memory / register thing discussed in the article) are handled as long as you properly use locks (or other synchronization primitives like semaphores or mutexes).
Its good to know how things work "under the covers" for performance reasons at least. Especially if you ever write a lock-free data-structure (not allowed to use... well... mutexes or locks), so you need to place the barriers in the appropriate spot.
------
I think the acquire/release model of consistency will become more important in the coming years. PCIe 4.0 is showing signs of supporting acquire/release... ARM and POWER have added acquire/release model instructions, and even CUDA has acquire/release semantics being built.
As high-performance code demands faster-and-faster systems, the more-and-more relaxed our systems will become. Acquire/release is quickly becoming the standard model.
- cwzwarich 7y ago> As high-performance code demands faster-and-faster systems, the more-and-more relaxed our systems will become. Acquire/release is quickly becoming the standard model. Linux relies heavily on performant RCU for scalability, which a pure acquire/release SW programming model can't support.
- dragontamer 7y agoThere must be some kind of communication error going on. I don't know much about RCU, so I just pulled up this webpage: https://www.kernel.org/doc/Documentation/RCU/whatisRCU.txt https://www.kernel.org/doc/Documentation/RCU/whatisRCU.txt In it is: > The rcu_read_lock() and rcu_read_unlock() primitive read-acquire and release a global reader-writer lock. Seems like RCU-operations in the Linux kernel are defined in acquire-barrier and release-barrier terms. I heard a while ago that RCU could be discussed in terms of release-consume semantics (which are slightly faster but harder to understand...) but very few people understand release-consume. As such, release-acquire is probably the memory model of the future. I'm not really aware of anything aside from: Fully Relaxed (unordered), the obscure release-consume, release-acquire, and finally sequentially consistent (too slow for modern systems) --------- Are you perhaps confusing "acquire-release" semantics (which is a memory-barrier / cache coherence principle) with spinlocks perchance? Acquire-release seems to be the "Fastest-practical" memory consistency model. (Since Relaxed doesn't work, and release-consume is too confusing) For more info on acquire-release, Preshing's blogposts are great: https://preshing.com/20130922/acquire-and-release-fences/ https://preshing.com/20130922/acquire-and-release-fences/
- jabl 7y ago> I'm not really aware of anything aside from: Fully Relaxed (unordered), the obscure release-consume, release-acquire, and finally sequentially consistent (too slow for modern systems) What about Total Store Ordering (TSO), which is what e.g. the obscure and rare x86(-64) architecture implements (and SPARC as well)?
- gpderetta 7y agoIn TSO every store is a release and every load is a acquire, so it maps very efficiently to the acquire/release model.
- jabl 7y agoSure, since it's a stronger model, so code which works on weaker acquire/release hw will work on TSO hw as well. You might as well say that sequential consistency maps very efficiently to a acquire/release model too in the same way.
- gpderetta 7y agoIsn't TSO the closest practical implementation to a acquire/release model? What are the practical differences? I know that TSO allows more easily to recover sequential consistency with additional barriers (Intel strengthened their original memory model to TSO for this reason).
- jabl 7y agoNo, acquire/release is a much weaker model than TSO. TSO is basically sequential consistency (SEQCST) + a store buffer. I.e. stores don't go immediately to main memory, but rather via a store buffer. Loads first peek into the store buffer of the local CPU core before going to memory. The practical effect is that in contrast to SEQCST stores may be reordered after loads ("store->load" reordering more formally). Acquire-release consistency allows many more reorderings in addition to store->load (load->load, load->store, store->store). For more info see e.g. Table 5 in http://www.rdrop.com/users/paulmck/scalability/paper/whymb.2010.06.07c.pdf http://www.rdrop.com/users/paulmck/scalability/paper/whymb.2...
- jabl 7y ago> ARM and POWER have added acquire/release model instructions They have implemented the acquire-release consistency model since day one (or, the day they started supporting multi-processors). Yes, there are some subtleties there that have in some cases been tightened later on, e.g. multi-copy atomicity.
- dragontamer 7y agoIIRC, there was a big stink because ARM and POWER historically implemented consume/release semantics, which is very slightly more relaxed than the now de-facto standard acquire/release semantics. ARM and POWERPC CPU devs worked very hard to get consume/release into C++11, but no compiler writer actually implemented that part of the standard. As such, consume/release can be safely forgotten into the annals of computer history (much like DEC Alpha's fully relaxed semantics) Then in ARM8, ARM simply added LDAR (Load-acquire) and STLR (Store-release) instructions. https://developer.arm.com/docs/100941/0100/barriers https://developer.arm.com/docs/100941/0100/barriers . So the ARM CPU how fully supports the acquire/release model. Apparently IBM's POWER instruction set was similarly strengthened to acquire/release (either POWER8 or POWER9). ARM / POWER "normal" loads and stores are still consume/release semantics. But compilers can simply emit LDAR (load-acquire) for the stronger guarantee. ---------- I remember at least one talk that showed that consume/release is ideal for things like Linux's RCU or something like that (that acquire/release is actually "too strong" for RCU, and therefore suboptimal). But because compiler-writers found consume/release too hard to reason about in practice, we're stuck with acquire/release. It seems like the C++ standard continues to evolve to push for memory_order_consume (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0750r1.html http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p075...), but all the details are still up for discussion.
- jabl 7y ago> ARM and POWERPC CPU devs worked very hard to get consume/release into C++11 AFAIR Paul McKenney was the primus motor, and the motivation was largely RCU. Then again, McKenney also worked for IBM at the time and certainly had an interest in pushing a model that mapped well to POWER. But it turned out to be both somewhat mis-specified and hard to implement cleanly, so most compilers just implemented it as an acquire. As you mention, there is ongoing work to fix it. As for ARM, it seems the big thing they've done since the initial release of ARMv8 is to banish non-multicopy atomicity. See https://www.cl.cam.ac.uk/~pes20/armv8-mca/armv8-mca-draft.pdf https://www.cl.cam.ac.uk/~pes20/armv8-mca/armv8-mca-draft.pd...
- gpderetta 7y agoI don't see anything x86 specific on the article. It focuses on cache coherency, which is applicable to ARM and POWER, and there isn't much about memory model. Even the description of MOESI is just an introduction and, as the article mentions, actual systems use more complicated protocols. Edit: if anything, the misconception is that memory barriers have anything to do with cache coherency.