Caches might be coherent, meaning that the masters might look into each others' content without any flushing. That was cool when I was working on DMA-heavy code

Caches might be coherent, meaning that the masters might look into each others' content without any flushing. That was cool when I was working on DMA-heavy code on a MIPS CPU. However it's not just about caches, but write buffers also. "Modern" GPUs play tricks with memory accesses by combining them and certain CPUs use this feature to combine byte accesses to word accesses, reorder accesses sequentially in a short span, and eliminate redundant stores/loads. Sometimes it helps quite much on performance, but hurts concurrency pretty much. 10% of the code was a call to the au_sync() macro to flush the write buffer... followed by some more instructions to flush the cache, in case it was required.