2 ms·
ML training runs are not reproducible, GPUs are non-deterministic when doing large sums, the order operands are added changes the result, thread execution times
by visarga 3y ago
ML training runs are not reproducible, GPUs are non-deterministic when doing large sums, the order operands are added changes the result, thread execution times also depend on caching, which is hard to predict. If you want to force deterministic mode be prepared for a huge slowdown.
- monocasa 3y agoA lot of work at larger orgs is put into reproducible training runs. It's about the only way to debug 'did the small parameter tweak tank performance because of a hardware failure or is there something special about that parameter?'