3 ms·

Multi-core math on GPU and CPU is non-deterministic for performance and scheduling reasons. The errors are small rounding errors that maybe don't have any seri

Multi-core math on GPU and CPU is non-deterministic for performance and scheduling reasons. The errors are small rounding errors that maybe don't have any serious implications right now. But the larger models get and the more operations and cores it takes to train them the more the rounding errors creep up.

fl7305 4y ago
> Multi-core math on GPU and CPU is non-deterministic for performance and scheduling reasons. Ok, I see what you mean. I can see how that could be the case. It depends on how the software is designed. Now that I looked up it, I was surprised to see that PyTorch may generate non-reproducible results: https://pytorch.org/docs/stable/notes/randomness.html https://pytorch.org/docs/stable/notes/randomness.html But it looks like the sources of non-determinism in PyTorch are known, and can be avoided with a lot of work and loss of performance? And for the general case, I don't think it's impossible to write deterministic code for multi-core processors? > The errors are small rounding errors But rounding errors don't imply non-deterministic answers, right? Just that the answer is different from the true answer? Calculating the square root of 2 will have a rounding error with 32-bit floating point, but are you saying that you'll get different bit patterns in your FP32 due to rounding errors?