4 ms·
The weights aren't needed to make it reproducable. The code and training data are needed. Hopefully if you used those, you'd ultimately reach the same result.
by _ea1k 3y ago
The weights aren't needed to make it reproducable. The code and training data are needed. Hopefully if you used those, you'd ultimately reach the same result.
- zcw100 3y agoRandom initialization would keep you from producing the exact same results.
- _ea1k 3y agoYes, but there's a difference between exact results and reproducible results. I should get similar performance, otherwise there is an issue.
- tbalsam 3y agoEven in the days where this was standard, that is not the case entirely. There is a whole other world between "released code" and "getting the results as seen in the paper". Unfortunately. The reproducibility crisis is very much well and alive! :'( Much more to go into but it is a deep rabbit hole, indeedy. :'((((
- _ea1k 3y agoI guess I'm saying that if there are reproducibility problems without the weights, then there's still a reproducibility problem with them. A paper with weights that magically work, when training on the same data and algorithm doesn't work is a paper that isn't reproducible. IMO, having the weights available sometimes just papers over a deeper issue.
- abdullin 3y agoTraining, especially on large GPU clusters, is inherently non-deterministic. Even, if all seeds are fixed. This boils down to framework implementations, timing issues and extra cost of trying to ensure determinism (without guarantees).