7 ms·

Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard. Does scientific-grade code need to handle a

Let's be clear - scientific-grade code is a substandard of production-grade code. But it is still a real standard. Does scientific-grade code need to handle a large number of users running it at the same time? Probably not a genuine concern, since those users will run their own copies of the code on their own hardware, and it's not necessary or relevant for users to see the same networked results from the same instance of the program running on a central machine. Does scientific-grade code need to publish telemetry? Eh, usually no. Set up alerting so that on-call engineers can be paged when (not if) it falls over? Nope. Does scientific-grade code need to handle the authorization and authentication of users? Nope. Does scientific-grade code need to be reproducible? Yes. Fundamentally yes. The reproducibility of results is core to the scientific method. Yes, that includes Monte Carlo code, when there is no such thing as truly random number generation on contemporary computers, only pseudorandom number generation, and what matters for cryptographic purposes is that the seed numbers for the pseudorandom generation are sufficiently hidden / unknown. For scientific purposes, the seed numbers should be published on purpose, so that a) the exact results you found, sufficiently random as they are for the purpose of your experiment, can still be independently verified by a peer reviewer, b) a peer reviewer can intentionally decide to pick a different seed value, which will lead to different results but should still lead to the same conclusion if your decision to reject / refuse to reject the null hypothesis was correct.

throwaway287391 6y ago
Controlling randomness can be extremely difficult to get right, especially when there's anything asynchronous about the code (e.g. multiple worker threads populating a queue to load data). In machine learning, some of the most popular frameworks (e.g. TensorFlow [0]) don't offer this as a feature, and in other frameworks that do (PyTorch [1]) it will cripple the speed you get as a result as GPU accelerators rely on non-deterministic accumulation for reasonable speed. Scientific reproducibility does not mean, and has never meant, you rerun the code and the output perfectly matches bit-for-bit every time. If you can achieve that, great -- it's certainly a useful property to have for debugging. But a much stronger and more relevant form of reproducibility for actually advancing science is running the same study e.g. on different groups of participants (or in computer science / applied math/stats / etc., with different codebases, with different model variants/hyperparameters, on different datasets) and the overall conclusions hold. To paraphrase a comment I saw from another thread on HN: "Plenty of good science got done before modern devops came to be." [0] https://github.com/tensorflow/tensorflow/issues/12871 https://github.com/tensorflow/tensorflow/issues/12871 https://github.com/tensorflow/tensorflow/issues/18096 https://github.com/tensorflow/tensorflow/issues/18096 [1] https://pytorch.org/docs/stable/notes/randomness.html https://pytorch.org/docs/stable/notes/randomness.html ========== EDIT to reply to solatic's replies below (I'm being rate-limited): The social science arguments are probably fair (or at least I'll leave it to someone more knowledgeable to defend them if they want) -- perhaps I shouldn't have led with the example of "different groups of participants". > If you can achieve that, for the area of study in which you conduct your experiment, it should be required. Deciding to forego formal reproducibility should be justified with a clear explanation as to why reproducibility is infeasible for your experiment, and peer-review should reject studies that could have be reproducible but weren't in practice. This might be a reasonable thing to enforce if everyone in the field were using the same computing platform. Given that they're not (and that telling everyone that all published results have to be done using AWS with this particular machine configuration is not a tenable solution) I don't see how this could ever be a realistic requirement. Or if you don't want to enforce that the results remain identical across different platforms, what's the point of the requirement in the first place? How would it be enforced if nobody else has the exact combination of hardware/software to do so? And then even if someone does, almost inevitably there'll be some detail of the setup that the researcher didn't think to report and results will differ slightly anyway. Besides, if you're allowing for exemptions, just about every paper in machine learning studying datasets larger than MNIST (where asynchronous prefetching of data is pretty much required to achieve decent speeds) would have a good reason to be exempt. It's possible that there are other fields where this sort of requirement would be both useful and feasible for a large amount of the research in that field, but I don't know what they are. > Also, reading through the issues you linked points to: https://github.com/NVIDIA/framework-determinism https://github.com/NVIDIA/framework-determinism which is a relatively recent attempt by nVidia to support deterministic computation for TensorFlow. Not perfect yet, but the effort is going there. (From your other comment.) Yes, there exists a $300B company with an ongoing-but-incomplete funded effort of so far >6 months' work (and that's just the part they've done in public) to make one of its own APIs optionally deterministic when it's being used through a single downstream client framework. If this isn't a perfect illustration that it's not realistic to expect exact determinism from software written by individual grad students studying chemistry, I'm not sure what to say.
- throwawaygh 6y ago
  > or in computer science / applied math/stats / etc., with different codebases, with different model variants, on different datasets) and the overall conclusions hold A lot of open sourced CS research is not reproducible. "the code still runs and gives the same output" is not the same as reproducibility.
  - throwaway287391 6y ago
    > A lot of open sourced CS research is not reproducible. I'm not sure if this was meant to be a counter-argument to me, but I completely agree! > "the code still runs and gives the same output" is not the same as reproducibility. Yes, bit-for-bit identical results are neither necessary nor sufficient for reproducibility in the usual scientific sense.
    - throwawaygh 6y ago
      > I'm not sure if this was meant to be a counter-argument to me It wasn't :)
- dnautics 6y ago
  the correct way to control randomness in scientific code is to have the RNG be seeded with a flag and have the result check out with a snapshot value. Almost no one does this, but that doesn't mean it shouldn't be done.
  - throwaway287391 6y ago
    Did you read my post? I know what a seed is. Setting one is typically not enough to ensure bit-for-bit identical results in high-performance code. I gave two examples of this: CUDA GPUs (which do non-deterministic accumulation) and asynchronous threads (which won't always run operations in the same order).
    - dnautics 6y ago
dllthomas 6y ago
> Does scientific-grade code need to be reproducible? Yes. Fundamentally yes. I agree that this is a good property for scientific code to have, but I think we need to be careful not to treat re-running of existing code the same way we treat genuinely independent replication. Traditionally, people freshly constructed any necessary apparatus, and people walked through the steps of the procedures. This is an interaction between experiment and human brain meats that's missing when code is simply reused (whether we consider it apparatus or procedure). Once we have multiple implementations, if there is a meaningful difference between them, at that point replayability is of tremendous value in identifying why they differ. But it is not reproducibility, as we want that term to be used in science.
- BadInformatics 6y ago
  Conversely though, it is often impossible to obtain the original code to replay and identify differences once that step is reached without some sort of strong incentive or mandate for researchers to publish it. When the only copy is lost in the now-inaccessible home folder of some former grad student's old lab machine, there is a strong disincentive to try replicating at all because one has little to consult on whether/how close the replicated methods are to the original ones.
  - dllthomas 6y ago
    And so we find ourselves in the same situation as the rest of the scientific process, throughout history. When I try to replicate your published paper and I fail, it's completely unclear whether it's "your fault" or "my fault" or pure happenstance, and there's a lot of picking apart that needs to be done with usually no access to the original experimental apparatus and sometimes no access to the original experimenters. The fact that we can have that option is an amazing opportunity that a confluence of attributes of software (specificity, replayability, easy of copying) afford us. Where we are not exploiting this like we could be, it is a failure of our institutions! But it is different-in-kind from traditional reproducibility.
    - BadInformatics 6y ago
dekhn 6y ago
As an ex-scientist who used to run lots of simulations, I really fail to see a truly compelling reason why most numerical results (for publication purposes) truly need to publish (and support) deterministic seeding. We've certainly done a lot, scientifically speaking (in terms of post-validated studies), without that level of reproducibility.
- jnxx 6y ago
  If nothing else, it helps debugging code which tries to reproduce your findings.
  - dekhn 6y ago
    The code I work with is not debuggable in that way under most circumstances. It's a complex distributed system. You don't attempt to debug it by being deterministic- you debug it by sampling its properties.
kag0 6y ago
> there is no such thing as truly random number generation on contemporary computers well that's just not true. there's no shortage of noise we can sample to get true random numbers. we just often stretch the random numbers for performance purposes.
- James_Henry 6y ago
  There is a shortage of truly random easily sampled noise though.
a_zaydak 6y ago
I do agree with you on publishing seeds for Monte Carlo simulations however the argument against it is also very strong. Usually when you run a monte carlo simulation you are quoting the results in terms of statistics. I think it would be sufficient to say that you can 'reproduce' the results as long as your statistics (over many simulations with different seeds) is consistent with the published results. If you run a single simulation with are particular seed you should get the same results however this might be cherry picking a particular simulation result. This is good for code testing but probably not for scientific results. I think by running the code with new seeds is a better way to test the science.
andrewprock 6y ago
> Does scientific-grade code need to be reproducible? Yes. Fundamentally yes This is definitely not correct. The experiment as a whole needs to be reproducible independently. This is very different, and more robust, from requiring that a particular portion of a previous version of the experiment to be reproducible in isolation.
konjin 6y ago
You can run the script multiple times and get a statistical representation of what the results should be. That's the point of science. This reminds me of being in gradschool and the comp-sci people complaining that we don't get bit-wise equal floats when we solve DEs. Having to re-implement a library from scratch for a project is much more valuable than running the same code in two places. The same way that getting the same results from two different machines is a lot more significant than getting the same result from two cloned machines. In short: code does not need to be reproducible because scientists know how to average.
jnxx 6y ago
> Does scientific-grade code need to be reproducible? Yes. Fundamentally yes. The reproducibility of results is core to the scientific method. Yes, that includes Monte Carlo code, [...] Reproducibility in the scientific sense is different from running the same program with the same input, and getting exactly the same result. Repreducibility means that if you repeat the measurements in another environment, getting somewhat different data, and apply the same theory and methods, you get to the same conclusion. The property of a computer program that when you run it again with the same input, you get the same output, is nice and very helpful for debugging. But the fact that you can run the same program does not mean that it is bug-free, as much as the fact that you can copy a paper with a mathematical proof does not mean that the proof is correct. Also, multi-threaded and parallel code is inherently non-deterministic. > when there is no such thing as truly random number generation on contemporary computers, only pseudorandom number generation, That is wrong. Linux for example uses latency measurements from drivers such as HDD drive seek latencies or keyboards to generate entropy. While it might not the best thing to rely on for purposes of cryptography, it is surely not deterministic. If it would matter, you could download real-time astronomical noise measurements and use them to seed your Mersenne Twister generator.
K0balt 6y ago
As a software developer, I tgi k maybe you misunderstand scientific reproducibility. Other scientists should be building their own apparatus, writing and running their own code. That the experiment is actually different is what validates the hypothesis, which specifies the salient conditions leading to the outcome. That an identical experiment leads to an identical outcome fails to validate the hypothesis, because the casual factors may have been misidentified. Precise reproducibility still matters if generalized reproduction fails, however, because the differences in the experimental implementation may lead to new and more accurate hypothesis about causality.
- logifail 6y ago
  > That an identical experiment leads to an identical outcome [..] So what happens if/when an identical experiment fails to lead to an identical outcome?
- Izkata 6y ago
  > Other scientists should be building their own apparatus, writing and running their own code. That the experiment is actually different is what validates the hypothesis, which specifies the salient conditions leading to the outcome. The problem here is a step before this: The results of these "identical" experiments are so wildly different there is nothing valid to propose, let alone for a recreation to compare against.

throwaway287391 6y ago

Controlling randomness can be extremely difficult to get right, especially when there's anything asynchronous about the code (e.g. multiple worker threads populating a queue to load data). In machine learning, some of the most popular frameworks (e.g. TensorFlow [0]) don't offer this as a feature, and in other frameworks that do (PyTorch [1]) it will cripple the speed you get as a result as GPU accelerators rely on non-deterministic accumulation for reasonable speed. Scientific reproducibility does not mean, and has never meant, you rerun the code and the output perfectly matches bit-for-bit every time. If you can achieve that, great -- it's certainly a useful property to have for debugging. But a much stronger and more relevant form of reproducibility for actually advancing science is running the same study e.g. on different groups of participants (or in computer science / applied math/stats / etc., with different codebases, with different model variants/hyperparameters, on different datasets) and the overall conclusions hold. To paraphrase a comment I saw from another thread on HN: "Plenty of good science got done before modern devops came to be." [0] https://github.com/tensorflow/tensorflow/issues/12871 https://github.com/tensorflow/tensorflow/issues/12871 https://github.com/tensorflow/tensorflow/issues/18096 https://github.com/tensorflow/tensorflow/issues/18096 [1] https://pytorch.org/docs/stable/notes/randomness.html https://pytorch.org/docs/stable/notes/randomness.html ========== EDIT to reply to solatic's replies below (I'm being rate-limited): The social science arguments are probably fair (or at least I'll leave it to someone more knowledgeable to defend them if they want) -- perhaps I shouldn't have led with the example of "different groups of participants". > If you can achieve that, for the area of study in which you conduct your experiment, it should be required. Deciding to forego formal reproducibility should be justified with a clear explanation as to why reproducibility is infeasible for your experiment, and peer-review should reject studies that could have be reproducible but weren't in practice. This might be a reasonable thing to enforce if everyone in the field were using the same computing platform. Given that they're not (and that telling everyone that all published results have to be done using AWS with this particular machine configuration is not a tenable solution) I don't see how this could ever be a realistic requirement. Or if you don't want to enforce that the results remain identical across different platforms, what's the point of the requirement in the first place? How would it be enforced if nobody else has the exact combination of hardware/software to do so? And then even if someone does, almost inevitably there'll be some detail of the setup that the researcher didn't think to report and results will differ slightly anyway. Besides, if you're allowing for exemptions, just about every paper in machine learning studying datasets larger than MNIST (where asynchronous prefetching of data is pretty much required to achieve decent speeds) would have a good reason to be exempt. It's possible that there are other fields where this sort of requirement would be both useful and feasible for a large amount of the research in that field, but I don't know what they are. > Also, reading through the issues you linked points to: https://github.com/NVIDIA/framework-determinism https://github.com/NVIDIA/framework-determinism which is a relatively recent attempt by nVidia to support deterministic computation for TensorFlow. Not perfect yet, but the effort is going there. (From your other comment.) Yes, there exists a $300B company with an ongoing-but-incomplete funded effort of so far >6 months' work (and that's just the part they've done in public) to make one of its own APIs optionally deterministic when it's being used through a single downstream client framework. If this isn't a perfect illustration that it's not realistic to expect exact determinism from software written by individual grad students studying chemistry, I'm not sure what to say.

throwawaygh 6y ago
> or in computer science / applied math/stats / etc., with different codebases, with different model variants, on different datasets) and the overall conclusions hold A lot of open sourced CS research is not reproducible. "the code still runs and gives the same output" is not the same as reproducibility.
- throwaway287391 6y ago
  > A lot of open sourced CS research is not reproducible. I'm not sure if this was meant to be a counter-argument to me, but I completely agree! > "the code still runs and gives the same output" is not the same as reproducibility. Yes, bit-for-bit identical results are neither necessary nor sufficient for reproducibility in the usual scientific sense.
  - throwawaygh 6y ago
    > I'm not sure if this was meant to be a counter-argument to me It wasn't :)
dnautics 6y ago
the correct way to control randomness in scientific code is to have the RNG be seeded with a flag and have the result check out with a snapshot value. Almost no one does this, but that doesn't mean it shouldn't be done.
- throwaway287391 6y ago
  Did you read my post? I know what a seed is. Setting one is typically not enough to ensure bit-for-bit identical results in high-performance code. I gave two examples of this: CUDA GPUs (which do non-deterministic accumulation) and asynchronous threads (which won't always run operations in the same order).
  - dnautics 6y ago

dllthomas 6y ago

> Does scientific-grade code need to be reproducible? Yes. Fundamentally yes. I agree that this is a good property for scientific code to have, but I think we need to be careful not to treat re-running of existing code the same way we treat genuinely independent replication. Traditionally, people freshly constructed any necessary apparatus, and people walked through the steps of the procedures. This is an interaction between experiment and human brain meats that's missing when code is simply reused (whether we consider it apparatus or procedure). Once we have multiple implementations, if there is a meaningful difference between them, at that point replayability is of tremendous value in identifying why they differ. But it is not reproducibility, as we want that term to be used in science.

BadInformatics 6y ago
Conversely though, it is often impossible to obtain the original code to replay and identify differences once that step is reached without some sort of strong incentive or mandate for researchers to publish it. When the only copy is lost in the now-inaccessible home folder of some former grad student's old lab machine, there is a strong disincentive to try replicating at all because one has little to consult on whether/how close the replicated methods are to the original ones.
- dllthomas 6y ago
  And so we find ourselves in the same situation as the rest of the scientific process, throughout history. When I try to replicate your published paper and I fail, it's completely unclear whether it's "your fault" or "my fault" or pure happenstance, and there's a lot of picking apart that needs to be done with usually no access to the original experimental apparatus and sometimes no access to the original experimenters. The fact that we can have that option is an amazing opportunity that a confluence of attributes of software (specificity, replayability, easy of copying) afford us. Where we are not exploiting this like we could be, it is a failure of our institutions! But it is different-in-kind from traditional reproducibility.
  - BadInformatics 6y ago

dekhn 6y ago

As an ex-scientist who used to run lots of simulations, I really fail to see a truly compelling reason why most numerical results (for publication purposes) truly need to publish (and support) deterministic seeding. We've certainly done a lot, scientifically speaking (in terms of post-validated studies), without that level of reproducibility.

jnxx 6y ago
If nothing else, it helps debugging code which tries to reproduce your findings.
- dekhn 6y ago
  The code I work with is not debuggable in that way under most circumstances. It's a complex distributed system. You don't attempt to debug it by being deterministic- you debug it by sampling its properties.

kag0 6y ago

> there is no such thing as truly random number generation on contemporary computers well that's just not true. there's no shortage of noise we can sample to get true random numbers. we just often stretch the random numbers for performance purposes.

James_Henry 6y ago
There is a shortage of truly random easily sampled noise though.

a_zaydak 6y ago

I do agree with you on publishing seeds for Monte Carlo simulations however the argument against it is also very strong. Usually when you run a monte carlo simulation you are quoting the results in terms of statistics. I think it would be sufficient to say that you can 'reproduce' the results as long as your statistics (over many simulations with different seeds) is consistent with the published results. If you run a single simulation with are particular seed you should get the same results however this might be cherry picking a particular simulation result. This is good for code testing but probably not for scientific results. I think by running the code with new seeds is a better way to test the science.

andrewprock 6y ago

> Does scientific-grade code need to be reproducible? Yes. Fundamentally yes This is definitely not correct. The experiment as a whole needs to be reproducible independently. This is very different, and more robust, from requiring that a particular portion of a previous version of the experiment to be reproducible in isolation.

konjin 6y ago

You can run the script multiple times and get a statistical representation of what the results should be. That's the point of science. This reminds me of being in gradschool and the comp-sci people complaining that we don't get bit-wise equal floats when we solve DEs. Having to re-implement a library from scratch for a project is much more valuable than running the same code in two places. The same way that getting the same results from two different machines is a lot more significant than getting the same result from two cloned machines. In short: code does not need to be reproducible because scientists know how to average.

jnxx 6y ago

> Does scientific-grade code need to be reproducible? Yes. Fundamentally yes. The reproducibility of results is core to the scientific method. Yes, that includes Monte Carlo code, [...] Reproducibility in the scientific sense is different from running the same program with the same input, and getting exactly the same result. Repreducibility means that if you repeat the measurements in another environment, getting somewhat different data, and apply the same theory and methods, you get to the same conclusion. The property of a computer program that when you run it again with the same input, you get the same output, is nice and very helpful for debugging. But the fact that you can run the same program does not mean that it is bug-free, as much as the fact that you can copy a paper with a mathematical proof does not mean that the proof is correct. Also, multi-threaded and parallel code is inherently non-deterministic. > when there is no such thing as truly random number generation on contemporary computers, only pseudorandom number generation, That is wrong. Linux for example uses latency measurements from drivers such as HDD drive seek latencies or keyboards to generate entropy. While it might not the best thing to rely on for purposes of cryptography, it is surely not deterministic. If it would matter, you could download real-time astronomical noise measurements and use them to seed your Mersenne Twister generator.

K0balt 6y ago

As a software developer, I tgi k maybe you misunderstand scientific reproducibility. Other scientists should be building their own apparatus, writing and running their own code. That the experiment is actually different is what validates the hypothesis, which specifies the salient conditions leading to the outcome. That an identical experiment leads to an identical outcome fails to validate the hypothesis, because the casual factors may have been misidentified. Precise reproducibility still matters if generalized reproduction fails, however, because the differences in the experimental implementation may lead to new and more accurate hypothesis about causality.

logifail 6y ago
> That an identical experiment leads to an identical outcome [..] So what happens if/when an identical experiment fails to lead to an identical outcome?
Izkata 6y ago
> Other scientists should be building their own apparatus, writing and running their own code. That the experiment is actually different is what validates the hypothesis, which specifies the salient conditions leading to the outcome. The problem here is a step before this: The results of these "identical" experiments are so wildly different there is nothing valid to propose, let alone for a recreation to compare against.