3 ms·

Its good to see some discussion about this. I'm currently maintaining an open source automated speech transcription library (https://github.com/persephone-tools

by JanisL 8y ago

Its good to see some discussion about this. I'm currently maintaining an open source automated speech transcription library (https://github.com/persephone-tools/persephone https://github.com/persephone-tools/persephone) that's being used in some research and we are extremely concerned about enabling reproducible research. Because the system is neural net based (Tensorflow) we have gone to effort beyond just open sourcing the code to make sure that people can rerun experiments. Reading this and the comments makes me aware that there might be sources of nondeterminism or other issues I may not be aware of. What issues should we be aware of for making sure that people can replicate results that were trained earlier? Are there any resources that discuss how people have gone about doing this? Any advice would be much appreciated

nlowell 8y ago
I think distributing the weights will give you guaranteed reproducibility. If your users intend on retraining the network, another idea would be to set performance expectations on a validation set within ballpark. So you could tell them, "we got 90% accuracy on this dataset, if you re-train and get below 80 you've probably made a mistake somewhere." The scary thing to me would be the very small test cases where maybe different trained neural nets end up having a lot of variance because there is barely more than noise to learn.
- JanisL 8y ago
  Distributing the weights seems to be the safest method, something I'll have to look into a bit more. It's a bit disturbing that training with the same data and same parameters could lead to different weights and therefore different accuracies but it's better to know that this is the case than not.
sdenton4 8y ago
My understanding (though it may be out of date) is that reproducibility is tough, at least in part because of closed gpu drivers, and in part because the big frameworks aren't really designed for it (possibly because it's pointless given the driver situation). On the framework side, you'd want good ways to ensure that random seeds are well distributed across a cluster of workers (and be able to restore state of a machine goes down). Also need to ensure that workers consume training data in a fixed order. This will slow you down, since you've got to wait for the slow poke every time, before updating weights.

by JanisL 8y ago

nlowell 8y ago

I think distributing the weights will give you guaranteed reproducibility. If your users intend on retraining the network, another idea would be to set performance expectations on a validation set within ballpark. So you could tell them, "we got 90% accuracy on this dataset, if you re-train and get below 80 you've probably made a mistake somewhere." The scary thing to me would be the very small test cases where maybe different trained neural nets end up having a lot of variance because there is barely more than noise to learn.

JanisL 8y ago
Distributing the weights seems to be the safest method, something I'll have to look into a bit more. It's a bit disturbing that training with the same data and same parameters could lead to different weights and therefore different accuracies but it's better to know that this is the case than not.

sdenton4 8y ago

My understanding (though it may be out of date) is that reproducibility is tough, at least in part because of closed gpu drivers, and in part because the big frameworks aren't really designed for it (possibly because it's pointless given the driver situation). On the framework side, you'd want good ways to ensure that random seeds are well distributed across a cluster of workers (and be able to restore state of a machine goes down). Also need to ensure that workers consume training data in a fixed order. This will slow you down, since you've got to wait for the slow poke every time, before updating weights.