I mean most binaries aren't reproducible. That takes a lot of work and isn't worth it most of the time. However, I've heard from ML engineers at the big shops

I mean most binaries aren't reproducible. That takes a lot of work and isn't worth it most of the time. However, I've heard from ML engineers at the big shops that hermetic, reproducible builds are key to making any progress at scale with ML. Apparently that goes all the way back to when Jeff Dean took over Google Brain.