3 ms·

> "They are the output of non-deterministic matrix multiplication operations" Just a nit-pick: Aren't neural networks and LLMs perfectly deterministic? I thin

by fl7305 4y ago

> "They are the output of non-deterministic matrix multiplication operations" Just a nit-pick: Aren't neural networks and LLMs perfectly deterministic? I think you can reproduce GPT-4 perfectly if you have access to the same source code, training data, and the seeds for the random number generators that they used? As a side note, I think it'd be theoretically possible to do this on a small 8-bit microcontroller given enough time and external storage. That's the beauty of Turing machines. This would not be practical in the least. But it sure was cool seeing a guy boot Linux in just 3.5 hours on a small 8-bit AVR microcontroller. https://dmitry.gr/?r=05.Projects&proj=07.%20Linux%20on%208bit https://dmitry.gr/?r=05.Projects&proj=07.%20Linux%20on%208bi...

MacsHeadroom 4y ago
Multi-core math on GPU and CPU is non-deterministic for performance and scheduling reasons. The errors are small rounding errors that maybe don't have any serious implications right now. But the larger models get and the more operations and cores it takes to train them the more the rounding errors creep up.
- fl7305 4y ago
  > Multi-core math on GPU and CPU is non-deterministic for performance and scheduling reasons. Ok, I see what you mean. I can see how that could be the case. It depends on how the software is designed. Now that I looked up it, I was surprised to see that PyTorch may generate non-reproducible results: https://pytorch.org/docs/stable/notes/randomness.html https://pytorch.org/docs/stable/notes/randomness.html But it looks like the sources of non-determinism in PyTorch are known, and can be avoided with a lot of work and loss of performance? And for the general case, I don't think it's impossible to write deterministic code for multi-core processors? > The errors are small rounding errors But rounding errors don't imply non-deterministic answers, right? Just that the answer is different from the true answer? Calculating the square root of 2 will have a rounding error with 32-bit floating point, but are you saying that you'll get different bit patterns in your FP32 due to rounding errors?

by fl7305 4y ago

MacsHeadroom 4y ago

Multi-core math on GPU and CPU is non-deterministic for performance and scheduling reasons. The errors are small rounding errors that maybe don't have any serious implications right now. But the larger models get and the more operations and cores it takes to train them the more the rounding errors creep up.

fl7305 4y ago
> Multi-core math on GPU and CPU is non-deterministic for performance and scheduling reasons. Ok, I see what you mean. I can see how that could be the case. It depends on how the software is designed. Now that I looked up it, I was surprised to see that PyTorch may generate non-reproducible results: https://pytorch.org/docs/stable/notes/randomness.html https://pytorch.org/docs/stable/notes/randomness.html But it looks like the sources of non-determinism in PyTorch are known, and can be avoided with a lot of work and loss of performance? And for the general case, I don't think it's impossible to write deterministic code for multi-core processors? > The errors are small rounding errors But rounding errors don't imply non-deterministic answers, right? Just that the answer is different from the true answer? Calculating the square root of 2 will have a rounding error with 32-bit floating point, but are you saying that you'll get different bit patterns in your FP32 due to rounding errors?