My understanding (though it may be out of date) is that reproducibility is tough, at least in part because of closed gpu drivers, and in part because the big fr

My understanding (though it may be out of date) is that reproducibility is tough, at least in part because of closed gpu drivers, and in part because the big frameworks aren't really designed for it (possibly because it's pointless given the driver situation). On the framework side, you'd want good ways to ensure that random seeds are well distributed across a cluster of workers (and be able to restore state of a machine goes down). Also need to ensure that workers consume training data in a fixed order. This will slow you down, since you've got to wait for the slow poke every time, before updating weights.