2 ms·
Is model training reproducible with identical software? I was under the impression that a number of CUDA expressions were non-deterministic, especially in the l
by fwip 7y ago
Is model training reproducible with identical software? I was under the impression that a number of CUDA expressions were non-deterministic, especially in the least-significant bits. And that's if you're running on the same hardware models, there's more variation between models.
- tbenst 7y agoIt’s largely but not fully deterministic eg backward pass is not deterministic: https://stackoverflow.com/questions/50744565/how-to-handle-non-determinism-when-training-on-a-gpu https://stackoverflow.com/questions/50744565/how-to-handle-n... That said, it’s hard enough to even run an old model 2 years later let alone reproduce to significant digits. I’d be happy enough to solve the former and get “close enough” training results