3 ms·
GPU scheduling only affects the result due to buggy optimizations. It's the exact same mechanism as fp rounding error on the CPU or updates to globally shared P
by fc417fc802 10d ago
GPU scheduling only affects the result due to buggy optimizations. It's the exact same mechanism as fp rounding error on the CPU or updates to globally shared PRNG state. We use lots of buggy optimizations because they don't matter in practice in most situations (see ex -ffast-math).
- qarl 10d agoI don't know the details but apparently it has something to do with multiprocess contention for the GPU and batch sizing.
- fc417fc802 9d agoRight but regardless, it's a buggy optimization. The calculations are all fully deterministic when done "properly" and fully consistently but we almost never bother with that because it slows things down and the errors don't matter in practice 99% of the time.
- qarl 9d agoAGREED. Although whoever designed that architecture might take issue with your description as "buggy".