4 ms·

It's easy to do this with anything in computer science as others can implement or recreate whatever is being discussed easily. Not so with a 6 month experiment

It's easy to do this with anything in computer science as others can implement or recreate whatever is being discussed easily. Not so with a 6 month experiment with several groups of humans, or even a 2 week one with rats.

godelski 3y ago
This is not always true. ML has reproducibility issues despite the status quo being open models. Even if we don't include proprietary datasets there are still issues. Compute is one issue, but we can even ignore that[0]. The Lottery Ticket[1,2] plays a big role, in that you can just get lucky. This really should have resulted in treating benchmarks as weaker indicators but see other comments about reviewing incentives. Another issue is that it is status quo to optimize hyperparameters on test data and this results in information leakage. While this won't affect results of running a checkpoint there are issues in reproducing the work. It also adds noise. Generative papers also have a large issue in not showing random uncurated samples, which introduces huge biases and we can argue this is a reproducibility issue as you can't reproduce the results show in the works. Anyone that's played with things like Stable Diffusion (or any generative model) will be familiar with this. There are more nuanced issues and things that require intimate domain knowledge to fully understand, but I wanted to push back at this comment because I see a lot of people just brush off reproducibility concerns by pointing to GitHub. While it helps, it definitely doesn't make reproducibility "easy" or concerns nonexistent. [0] Sometimes we can't though as scaling has more effects than obvious. e.g. GANs can't scale batch on a per GPU level but scaling by multiple GPUs/nodes does give an advantage in quality (not just training times). This is non-obvious and many things can play a role. [1] https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635 [2] https://arxiv.org/abs/2109.08203 https://arxiv.org/abs/2109.08203

godelski 3y ago

This is not always true. ML has reproducibility issues despite the status quo being open models. Even if we don't include proprietary datasets there are still issues. Compute is one issue, but we can even ignore that[0]. The Lottery Ticket[1,2] plays a big role, in that you can just get lucky. This really should have resulted in treating benchmarks as weaker indicators but see other comments about reviewing incentives. Another issue is that it is status quo to optimize hyperparameters on test data and this results in information leakage. While this won't affect results of running a checkpoint there are issues in reproducing the work. It also adds noise. Generative papers also have a large issue in not showing random uncurated samples, which introduces huge biases and we can argue this is a reproducibility issue as you can't reproduce the results show in the works. Anyone that's played with things like Stable Diffusion (or any generative model) will be familiar with this. There are more nuanced issues and things that require intimate domain knowledge to fully understand, but I wanted to push back at this comment because I see a lot of people just brush off reproducibility concerns by pointing to GitHub. While it helps, it definitely doesn't make reproducibility "easy" or concerns nonexistent. [0] Sometimes we can't though as scaling has more effects than obvious. e.g. GANs can't scale batch on a per GPU level but scaling by multiple GPUs/nodes does give an advantage in quality (not just training times). This is non-obvious and many things can play a role. [1] https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635 [2] https://arxiv.org/abs/2109.08203 https://arxiv.org/abs/2109.08203