6 ms·
Byte for byte reproducibility is an extremely high bar. We don't even have that for most open source software. The reproducibility standard that is practice is
by jononor 2y ago
Byte for byte reproducibility is an extremely high bar. We don't even have that for most open source software. The reproducibility standard that is practice is perform within -+ epsilon on the test sets. That would be sufficient, and more practical, in my opinion.
- stavros 2y agoWe don't have that for software because compiler introduce a lot of randomness. Maybe training introduces randomness as well, I don't know (floating point errors accumulating?). If it does, your metric sounds good, if not, why not go for byte-for-byte...
- pabs3 2y agoReproducible Builds are quite common now, there is even a movement about them: https://reproducible-builds.org/ https://reproducible-builds.org/
- Zambyte 2y agoWhen does a compiler produce randomness? I think non reproducible builds are usually due to environmental inputs (for example: build a date macro) or an undeespecified build process (such as: what compiler version, what build optimizations are used, etc). All of that can be constrained / defined, it's just work that people often don't do.