7 ms·

Controlling randomness can be extremely difficult to get right, especially when there's anything asynchronous about the code (e.g. multiple worker threads popul

Controlling randomness can be extremely difficult to get right, especially when there's anything asynchronous about the code (e.g. multiple worker threads populating a queue to load data). In machine learning, some of the most popular frameworks (e.g. TensorFlow [0]) don't offer this as a feature, and in other frameworks that do (PyTorch [1]) it will cripple the speed you get as a result as GPU accelerators rely on non-deterministic accumulation for reasonable speed. Scientific reproducibility does not mean, and has never meant, you rerun the code and the output perfectly matches bit-for-bit every time. If you can achieve that, great -- it's certainly a useful property to have for debugging. But a much stronger and more relevant form of reproducibility for actually advancing science is running the same study e.g. on different groups of participants (or in computer science / applied math/stats / etc., with different codebases, with different model variants/hyperparameters, on different datasets) and the overall conclusions hold. To paraphrase a comment I saw from another thread on HN: "Plenty of good science got done before modern devops came to be." [0] https://github.com/tensorflow/tensorflow/issues/12871 https://github.com/tensorflow/tensorflow/issues/12871 https://github.com/tensorflow/tensorflow/issues/18096 https://github.com/tensorflow/tensorflow/issues/18096 [1] https://pytorch.org/docs/stable/notes/randomness.html https://pytorch.org/docs/stable/notes/randomness.html ========== EDIT to reply to solatic's replies below (I'm being rate-limited): The social science arguments are probably fair (or at least I'll leave it to someone more knowledgeable to defend them if they want) -- perhaps I shouldn't have led with the example of "different groups of participants". > If you can achieve that, for the area of study in which you conduct your experiment, it should be required. Deciding to forego formal reproducibility should be justified with a clear explanation as to why reproducibility is infeasible for your experiment, and peer-review should reject studies that could have be reproducible but weren't in practice. This might be a reasonable thing to enforce if everyone in the field were using the same computing platform. Given that they're not (and that telling everyone that all published results have to be done using AWS with this particular machine configuration is not a tenable solution) I don't see how this could ever be a realistic requirement. Or if you don't want to enforce that the results remain identical across different platforms, what's the point of the requirement in the first place? How would it be enforced if nobody else has the exact combination of hardware/software to do so? And then even if someone does, almost inevitably there'll be some detail of the setup that the researcher didn't think to report and results will differ slightly anyway. Besides, if you're allowing for exemptions, just about every paper in machine learning studying datasets larger than MNIST (where asynchronous prefetching of data is pretty much required to achieve decent speeds) would have a good reason to be exempt. It's possible that there are other fields where this sort of requirement would be both useful and feasible for a large amount of the research in that field, but I don't know what they are. > Also, reading through the issues you linked points to: https://github.com/NVIDIA/framework-determinism https://github.com/NVIDIA/framework-determinism which is a relatively recent attempt by nVidia to support deterministic computation for TensorFlow. Not perfect yet, but the effort is going there. (From your other comment.) Yes, there exists a $300B company with an ongoing-but-incomplete funded effort of so far >6 months' work (and that's just the part they've done in public) to make one of its own APIs optionally deterministic when it's being used through a single downstream client framework. If this isn't a perfect illustration that it's not realistic to expect exact determinism from software written by individual grad students studying chemistry, I'm not sure what to say.

throwawaygh 6y ago
> or in computer science / applied math/stats / etc., with different codebases, with different model variants, on different datasets) and the overall conclusions hold A lot of open sourced CS research is not reproducible. "the code still runs and gives the same output" is not the same as reproducibility.
- throwaway287391 6y ago
  > A lot of open sourced CS research is not reproducible. I'm not sure if this was meant to be a counter-argument to me, but I completely agree! > "the code still runs and gives the same output" is not the same as reproducibility. Yes, bit-for-bit identical results are neither necessary nor sufficient for reproducibility in the usual scientific sense.
  - throwawaygh 6y ago
    > I'm not sure if this was meant to be a counter-argument to me It wasn't :)
dnautics 6y ago
the correct way to control randomness in scientific code is to have the RNG be seeded with a flag and have the result check out with a snapshot value. Almost no one does this, but that doesn't mean it shouldn't be done.
- throwaway287391 6y ago
  Did you read my post? I know what a seed is. Setting one is typically not enough to ensure bit-for-bit identical results in high-performance code. I gave two examples of this: CUDA GPUs (which do non-deterministic accumulation) and asynchronous threads (which won't always run operations in the same order).
  - dnautics 6y ago
    Most scientific runs are scaled where you run multiple replicates. And not all scientific runs are high-performance in the HPC sense. Even if your code is HPC in the HPC sense, and requires CUDA, and 40,000 cores, you should consider creating a release flag where an end user can do at least single "slow" run on a CPU on a reduced dataset, in single threaded mode, to sanity check the results and at least verify that the computational and algorithmic pipeline is sound at the most basic level. I used to be a scientist. I get it, getting scientists to do this is like pulling teeth, but it's the least you could do to give other people confidence in your results.

throwawaygh 6y ago

> or in computer science / applied math/stats / etc., with different codebases, with different model variants, on different datasets) and the overall conclusions hold A lot of open sourced CS research is not reproducible. "the code still runs and gives the same output" is not the same as reproducibility.

throwaway287391 6y ago
> A lot of open sourced CS research is not reproducible. I'm not sure if this was meant to be a counter-argument to me, but I completely agree! > "the code still runs and gives the same output" is not the same as reproducibility. Yes, bit-for-bit identical results are neither necessary nor sufficient for reproducibility in the usual scientific sense.
- throwawaygh 6y ago
  > I'm not sure if this was meant to be a counter-argument to me It wasn't :)

dnautics 6y ago

the correct way to control randomness in scientific code is to have the RNG be seeded with a flag and have the result check out with a snapshot value. Almost no one does this, but that doesn't mean it shouldn't be done.

throwaway287391 6y ago
Did you read my post? I know what a seed is. Setting one is typically not enough to ensure bit-for-bit identical results in high-performance code. I gave two examples of this: CUDA GPUs (which do non-deterministic accumulation) and asynchronous threads (which won't always run operations in the same order).
- dnautics 6y ago
  Most scientific runs are scaled where you run multiple replicates. And not all scientific runs are high-performance in the HPC sense. Even if your code is HPC in the HPC sense, and requires CUDA, and 40,000 cores, you should consider creating a release flag where an end user can do at least single "slow" run on a CPU on a reduced dataset, in single threaded mode, to sanity check the results and at least verify that the computational and algorithmic pipeline is sound at the most basic level. I used to be a scientist. I get it, getting scientists to do this is like pulling teeth, but it's the least you could do to give other people confidence in your results.