4 ms·
Fully reproducible model training might simply not be possible if information from the training environment is not captured. In addition to data and code you mi
by rapatel0 3y ago
Fully reproducible model training might simply not be possible if information from the training environment is not captured. In addition to data and code you might have additional uncertainty from:
- pseudo/true random number generator and initialization
- certain speculative optimizations associated with training environments (distributed)
- Speculative optimizations associated with model compression
- Image decompression algorithm mismatch (basically this is library versioning)
- ....things I'm forgetting...
It's just a lot of things to remember to capture, communicate, and reproduce.
- martincmartin 3y agopseudo/true random number generator and initialization It's not just the generator and initialization. If you do anything multithreaded, like a producer/consumer queue, then you need to know which pieces of work went to which thread in which order. It's a lot like reproducing subtle and rare race conditions.
- monocasa 3y agoMost of the mature ML environments are pretty focused on reproducible training though. It's pretty necessary for debugging and iteration.