> >IEEE floating point is deterministic, to the apparent contradiction of many sentences written by Gustafson > "parallel systems, most likely GPUs, where it i

> >IEEE floating point is deterministic, to the apparent contradiction of many sentences written by Gustafson > "parallel systems, most likely GPUs, where it is common to have slightly different results even on the same operation, on the same system. This is due to the fact that several pipelines will do calculations in parallel and depending on which pipeline ends first (which can depend on many factors like current temperature) the sums can happen in a different order, leading to different rounding results." > due to fp associativity? As far as I understand, that's not quite the case (on Fermi+ Nvidia, as previous generations had no real support for proper IEEE single compute). There are pipelines in parallel, but almost all non-determinism in the order of execution (resulting in fp associativity) is exposed to the programmer, except for atomic instructions. Those can even depend on clock skew across PCIe or NVLink, if they operate on remote memory. NVLink supports remote execution in a recent version (I suggest the microbenchmarking-derived microarchitecture analysis papers for Volta and Turing, even if they are less hands-on the Nervana's maxas (with the SGEMM walk-through, which I really recommend)), and thus any hope of reliably tricking synchronization behavior via undefined opcodes/scheduler instructions trickery is deliberately out of the window. On IBM POWER9 CPUs they can apparently delegate execution of atomic operations to the host CPU, which might (I don't know enough about POWER9 cache coherency /NUMA protocols) delegate them over the NUMA fabric to a different rack-mount unit bonded into the same NUMA mesh. TL;DR: Recent Nvidia GPUs do not suffer implicit non-determinism in IEEE floats unless you use hardware atomics. Non-determinism is exposed and introduced in explicit concurrent data structures [0]. [0]: If you count a datastructure with a simple spinlock guarding it as a concurrent datastructure. PS: I'll add references / links later on the laptop.