4 ms·
> The dataset and pipeline to rebuild these weights are not included. This is a good point. But its probably not deterministic and reproducible. I bet the res
by brucethemoose2 3y ago
> The dataset and pipeline to rebuild these weights are not included.
This is a good point.
But its probably not deterministic and reproducible. I bet the researchers themselves couldn't remake the exact weights (or even similar weights) from scratch.
- monocasa 3y agoI mean most binaries aren't reproducible. That takes a lot of work and isn't worth it most of the time. However, I've heard from ML engineers at the big shops that hermetic, reproducible builds are key to making any progress at scale with ML. Apparently that goes all the way back to when Jeff Dean took over Google Brain.
- Scene_Cast2 3y agoYep. Even if the initial seed for parameter init, the example shuffling seed, etc were constant, the distributed nature of training (and syncing the model across machines) would kill reproducibility. Not to mention resuming from checkpoints after gradient explosions, etc.
- monocasa 3y agoI've heard from ML engineers at larger shops that reproducibility is key to working at scale. That's how you track down "this training regime went to shit because of something we changed" versus "this training regime went to shit because on of the GPUs training it is starting to fail".