5 ms·

> You can't trust the hardware coherency protocol to do much for you until you follow the platform-specific rules to tell the CPU to make something act coherent

> You can't trust the hardware coherency protocol to do much for you until you follow the platform-specific rules to tell the CPU to make something act coherent. If software has to perform some operation to maintain coherency then you hardware is not cache coherent. Which some aren't, and especially some sub-sets of operations aren't (instruction cache vs data ops, or CPU vs DMA). But for load/store operations by CPUs, they're all coherent. Barrier/fence instructions are about enforcing ordering on when memory operations to more than one location can be performed or become visible, with respect to one another.

Dylan16807 3y ago
It's coherent behind the scenes but it often presents an incoherent view to the software. It's not acting coherent when the rearrangement of memory operations makes you see different orderings from different cores. When a CPU has a loose memory model and is aggressively making use of the reordering capabilities, the cache being coherent internally is basically just an implementation detail. It's not part of the visible ABI.
- throwawaylinux 3y ago
  It never presents an incoherent view to software. I'm using coherency as in the term of art, not a colloquial meaning. Every agent observes stores to a location in the same order[*]. Cache coherency says nothing about observed ordering of stores to different locations. [*] Although store forwarding throws a bit of a spanner in that definition, there can still be reordering occurring absent that local reordering.
  - Dylan16807 3y ago
    Fine, with that specific term of art meaning then ignore my second post. I stand by my original statement that you can't trust it to "do much for you". Just replace the last word with "act consistent" or "act ordered". Per-address ordering is nearly useless by itself. And if you had a CPU that didn't guarantee that, you'd observe almost no difference.
    - throwawaylinux 3y ago
      Well no, if you don't have a coherent system then your memory operations aren't reliable. You can lose updates or read stale data. Look at what software has to do in incoherent systems, specific flush and invalidate points which is not the same as ordering barriers. Your CPU guarantees a lot, cache coherency to start with. But also a very well defined ordering model and ordering instructions. It's not necessarily trivial to program for, but that doesn't mean you can't trust it.
      - Dylan16807 3y ago
        > You can lose updates A system without cache coherency can still promise that updates won't be lost. There are lots of way to write rules around update propagation, and cache coherency is just one of them. > or read stale data Cache coherency doesn't protect you from stale data unless you only read one memory address ever. > Look at what software has to do in incoherent systems, specific flush and invalidate points which is not the same as ordering barriers. That depends on the memory model. You could have a system that doesn't guarantee cache coherency in general but works fine if you put in ordinary memory barriers.
        gpderetta 3y ago
        > You could have a system that doesn't guarantee cache coherency in general but works fine if you put in ordinary memory barriers. How would that work? In such a system, either you have no caches or memory barriers would need to pessimistically flush all dirty lines to memory and send invalidation and synchronization messages to all other cores. In practice such system, far from being fine, would be so slow to be unusable if barriers had such semantics. Even implementing c++ relaxed semantics would be very expensive.
        Dylan16807 3y ago
        > In such a system, either you have no caches or memory barriers would need to pessimistically flush all dirty lines to memory and send invalidation and synchronization messages to all other cores. Why would you need to avoid caches or flush to memory? And invalidation and synchronization message are already part of a normal CPU's overhead, so I don't see why restricting some of them to memory barriers would increase overhead. In other words, assume you still have a cache coherency protocol, but you're lazy about certain states in the absence of memory barriers, so the default behavior is not always coherent.
  - gpderetta 3y ago
    Hum I don't see how store forwarding breaks the illusion of total order of stores on a single memory location, at least in 5 minutes of thinking I can't come up with a litmus that would demonstrate it. In fact even c++ relaxed stores and loads preserve this ordering. I think your definition is correct without the asterisk. edit: tweaked working
    - throwawaylinux 3y ago
      Oh yes that must be right, I wasn't thinking (or thinking about consistency ordering). Good catch.

Dylan16807 3y ago

It's coherent behind the scenes but it often presents an incoherent view to the software. It's not acting coherent when the rearrangement of memory operations makes you see different orderings from different cores. When a CPU has a loose memory model and is aggressively making use of the reordering capabilities, the cache being coherent internally is basically just an implementation detail. It's not part of the visible ABI.

throwawaylinux 3y ago
It never presents an incoherent view to software. I'm using coherency as in the term of art, not a colloquial meaning. Every agent observes stores to a location in the same order[*]. Cache coherency says nothing about observed ordering of stores to different locations. [*] Although store forwarding throws a bit of a spanner in that definition, there can still be reordering occurring absent that local reordering.
- Dylan16807 3y ago
  Fine, with that specific term of art meaning then ignore my second post. I stand by my original statement that you can't trust it to "do much for you". Just replace the last word with "act consistent" or "act ordered". Per-address ordering is nearly useless by itself. And if you had a CPU that didn't guarantee that, you'd observe almost no difference.
  - throwawaylinux 3y ago
    Well no, if you don't have a coherent system then your memory operations aren't reliable. You can lose updates or read stale data. Look at what software has to do in incoherent systems, specific flush and invalidate points which is not the same as ordering barriers. Your CPU guarantees a lot, cache coherency to start with. But also a very well defined ordering model and ordering instructions. It's not necessarily trivial to program for, but that doesn't mean you can't trust it.
    - Dylan16807 3y ago
      > You can lose updates A system without cache coherency can still promise that updates won't be lost. There are lots of way to write rules around update propagation, and cache coherency is just one of them. > or read stale data Cache coherency doesn't protect you from stale data unless you only read one memory address ever. > Look at what software has to do in incoherent systems, specific flush and invalidate points which is not the same as ordering barriers. That depends on the memory model. You could have a system that doesn't guarantee cache coherency in general but works fine if you put in ordinary memory barriers.
      - gpderetta 3y ago
        > You could have a system that doesn't guarantee cache coherency in general but works fine if you put in ordinary memory barriers. How would that work? In such a system, either you have no caches or memory barriers would need to pessimistically flush all dirty lines to memory and send invalidation and synchronization messages to all other cores. In practice such system, far from being fine, would be so slow to be unusable if barriers had such semantics. Even implementing c++ relaxed semantics would be very expensive.
        Dylan16807 3y ago
        > In such a system, either you have no caches or memory barriers would need to pessimistically flush all dirty lines to memory and send invalidation and synchronization messages to all other cores. Why would you need to avoid caches or flush to memory? And invalidation and synchronization message are already part of a normal CPU's overhead, so I don't see why restricting some of them to memory barriers would increase overhead. In other words, assume you still have a cache coherency protocol, but you're lazy about certain states in the absence of memory barriers, so the default behavior is not always coherent.
- gpderetta 3y ago
  Hum I don't see how store forwarding breaks the illusion of total order of stores on a single memory location, at least in 5 minutes of thinking I can't come up with a litmus that would demonstrate it. In fact even c++ relaxed stores and loads preserve this ordering. I think your definition is correct without the asterisk. edit: tweaked working
  - throwawaylinux 3y ago
    Oh yes that must be right, I wasn't thinking (or thinking about consistency ordering). Good catch.