3 ms·

> If you try that on, say, an 80-core ARM CPU [1], CPU #53 might not see a 1 for a long time, if ever.[2] You're no longer guaranteed that cache changes propaga

by rrss 6y ago

> If you try that on, say, an 80-core ARM CPU [1], CPU #53 might not see a 1 for a long time, if ever.[2] You're no longer guaranteed that cache changes propagate unasked. What do you mean by this? ARM requires cache coherency for data caches, which requires that every store is eventually made visible to all cores. Fences are necessary to impose additional requirements on the order which writes to different addresses are made visible, but AFAIK not necessary to ensure that a single write is eventually visible.

Animats 6y ago
Not sure how Load-Acquire (LDAR) and Store-Release (STLR) fit into this. They seem to be intended to be atomic only against other LDAR/STLR pairs, rather than doing a general flush. There's also the whole "shareability domain" thing - full system, outer sharable, inner sharable, and non-sharable. Barrier instructions get to specify how much consistency breadth to request. So there are situations when every store may not be visible to every CPU core, but the ones that don't get the word shouldn't have access to that memory anyway. You can sort of see the architects of the CPU trying hard to prevent the cache system from choking on inter-CPU traffic, and trying to get more help from the software.
- millstone 6y ago
  "Atomic" is probably the wrong term at this level. For a CPU, all loads and stores are atomic (assuming aligned and <= word size). On an out-of-order CPU, loads and stores go into a buffer which may process its entries in a different order. Fences like LDA and STL prevent certain reorderings.
- rigtorp 6y ago
  All loads and stores are atomic. I'm also pretty sure ARM has coherent caches. What ARM does allow is for CPUs to optimize the order it loads and stores from cache as opposed to x86 TSO guarantee. DMB barrier instruction on ARM allows you to control how L/S may be reordered. Both x86 and ARM have special types of memory regions such as uncachable memory, but that's not something you are going to see outside of kernel space.
- rrss 6y ago
  Inner vs outer shareable is not particularly relevant for applications - from the ARM ARM: > This architecture assumes that all PEs that use the same operating system or hypervisor are in the same Inner Shareable shareability domain. [PE = processing element - i.e. core] i.e. if you a running a userspace program, all the cores are in the same shareability domain. (I think your comments about LDAR/STLR are a response to 'adwn's sibling comment). Eventual write propagation is a very basic liveness guarantee for systems with coherent caches. I'm pretty sure that the ARM architecture requires it for Observers with coherent caches, but event if not, it would be very surprising if a mainstream implementation didn't provide this guarantee. Barrier instructions provide control of ordering relationships between accesses and when accesses complete, but I believe there is a guarantee that accesses eventually complete even without barriers. (Also from the ARM ARM (B2.3.7), "complete" here means "visible to other cores": "Any read to the same Location by an Observer within the shareability domain will either Reads-from W1 or Reads-from a write that is Coherence-after W1.") I'd be very interested to learn that ARM (or Ampere Altra) doesn't guarantee eventual write propragation, but that needs a better source than [2].

by rrss 6y ago

Animats 6y ago

Not sure how Load-Acquire (LDAR) and Store-Release (STLR) fit into this. They seem to be intended to be atomic only against other LDAR/STLR pairs, rather than doing a general flush. There's also the whole "shareability domain" thing - full system, outer sharable, inner sharable, and non-sharable. Barrier instructions get to specify how much consistency breadth to request. So there are situations when every store may not be visible to every CPU core, but the ones that don't get the word shouldn't have access to that memory anyway. You can sort of see the architects of the CPU trying hard to prevent the cache system from choking on inter-CPU traffic, and trying to get more help from the software.

millstone 6y ago
"Atomic" is probably the wrong term at this level. For a CPU, all loads and stores are atomic (assuming aligned and <= word size). On an out-of-order CPU, loads and stores go into a buffer which may process its entries in a different order. Fences like LDA and STL prevent certain reorderings.
rigtorp 6y ago
All loads and stores are atomic. I'm also pretty sure ARM has coherent caches. What ARM does allow is for CPUs to optimize the order it loads and stores from cache as opposed to x86 TSO guarantee. DMB barrier instruction on ARM allows you to control how L/S may be reordered. Both x86 and ARM have special types of memory regions such as uncachable memory, but that's not something you are going to see outside of kernel space.
rrss 6y ago
Inner vs outer shareable is not particularly relevant for applications - from the ARM ARM: > This architecture assumes that all PEs that use the same operating system or hypervisor are in the same Inner Shareable shareability domain. [PE = processing element - i.e. core] i.e. if you a running a userspace program, all the cores are in the same shareability domain. (I think your comments about LDAR/STLR are a response to 'adwn's sibling comment). Eventual write propagation is a very basic liveness guarantee for systems with coherent caches. I'm pretty sure that the ARM architecture requires it for Observers with coherent caches, but event if not, it would be very surprising if a mainstream implementation didn't provide this guarantee. Barrier instructions provide control of ordering relationships between accesses and when accesses complete, but I believe there is a guarantee that accesses eventually complete even without barriers. (Also from the ARM ARM (B2.3.7), "complete" here means "visible to other cores": "Any read to the same Location by an Observer within the shareability domain will either Reads-from W1 or Reads-from a write that is Coherence-after W1.") I'd be very interested to learn that ARM (or Ampere Altra) doesn't guarantee eventual write propragation, but that needs a better source than [2].