3 ms·

edit: please read the grandchild comment before going off on the idea that some random programmer on the Internet dares to criticize scientific code he does not

by jnxx 6y ago

edit: please read the grandchild comment before going off on the idea that some random programmer on the Internet dares to criticize scientific code he does not understand. What is crucial in the argument here is indeed the distinction between methods employing pseudo-randomness, like Monte Carlo simulation, and non-determinism caused by undefined behavior. > I'm an accelerator physicist and I wouldn't want my code to end up on acceleratorskeptics.com with people that don't understand the material making low effort critiques of minor technical points. The person which wrote the linked blog post writes that it was a software engineer at google. Unfortunately, that claim is not falsifiable as the person decided to remain anonymous. > As an example, you seem to be complaining that their Monte Carlo code has non-deterministic output when that is the entire point of Monte Carlo methods and doesn't change their result. The claim is that even with the same random seed for the random generator, the program produces different results, and this is explained by the allegation that it runs non-deterministic (in the sense of undefined behavior) in multiple threads. It claims also that it produces significantly different results depending on which output file format is chosen. If this is true, the code would have race conditions, and as being impacted by race conditions is a form of undefined behavior, this would make any result of the program questionable, as the program would not be well-defined. Personally, I am very doubtful whether this is true, this would be incredibly sloppy by the imperial college scientists. Some more careful analysis by a recognized programmer might be warranted. However it underlines well the importance of the main topic that scientific code should be open to analysis. > What I'm saying is that scientific code doesn't need to handle every special case or be easily usable by non-experts. Fully agree with this. But it should try to document its limitations.

aspaceman 6y ago
> If this is true, the code would have race conditions, and as being impacted by race conditions is a form of undefined behavior, this would make any result of the program questionable, as the program would not be well-defined. That’s not at all what that means. What are you talking about? As long as a Monte Carlo process works towards the same result it’s equivalent. You’re speaking genuine nonsense as far as I’m concerned. Randomness doesn’t imply non deterministic. Non-determinitism in no way implies race conditions or undefined behavior. We care that the random process reaches the same result, not that the exact sequence of steps is the same. This is what scientists are talking about. A bunch of (pretty stupid) nonexperts want to criticize your code, so they feel smart on the internet.
- jnxx 6y ago
  I am referring to this blog post: https://lockdownsceptics.org/code-review-of-fergusons-model/ https://lockdownsceptics.org/code-review-of-fergusons-model/ It says, word-by-word: > Clearly, the documentation wants us to think that, given a starting seed, the model will always produce the same results. > >Investigation reveals the truth: the code produces critically different results, even for identical starting seeds and parameters. > I’ll illustrate with a few bugs. In issue 116 a UK “red team” at Edinburgh University reports that they tried to use a mode that stores data tables in a more efficient format for faster loading, and discovered – to their surprise – that the resulting predictions varied by around 80,000 deaths after 80 days: ... The bugs which the blog post implies here are such ones as described by Jens Regehr: https://blog.regehr.org/archives/213 https://blog.regehr.org/archives/213 Not that I do not endorse these statements in the blog - I am rather skeptical whether they are true at all. What the authors of the blob post mean is clearly "undefined behaviour" in the sense of non-deterministic program execution of a program that is not well-formed. It is clear that many non-experts could confuse that with the pseudo-randomness implicit in Monte-Carlo simulations, but this is a very different thing. The first is basically a broken, invalid, and untrustworthy program. The second is the established method to produce a computational result by introducing stochastic behavior, which is for example how modern weather models work. These are wildly different things. I do not understand why your comment just adds to the confusion between these two things?? > A bunch of (pretty stupid) nonexperts want to criticize your code, so they feel smart on the internet. As said, I don't endorse the critique in the blog. However, critique in a software implementation, as well as in scientific matters, should never carry a call on authority - it should logically explain what is the problem, with concrete points. Unfortunately, the cited blog post remains very vague about this, while claiming: > My background. I have been writing software for 30 years. I worked at Google between 2006 and 2014, where I was a senior software engineer working on Maps, Gmail and account security. I spent the last five years at a US/UK firm where I designed the company’s database product, amongst other jobs and projects. I was also an independent consultant for a couple of years. It would be much better if, instead claiming that there could be race conditions, it could point to lines in the code with actual race conditions, and show how the results of the simulation are different when the race conditions are fixed. Otherwise, it just looks like he claims that the program is buggy, because he is in no position to question the science, and does not like the result.
UncleMeat 6y ago
Race conditions aren't undefined behavior in C/C++. Data races are. Lots and lots of real systems contain race conditions without catastrophe.
- jnxx 6y ago
  > Race conditions aren't undefined behavior in C/C++. Data races are. You are right with the distinction, I had data race conditions in mind. Race conditions can well happen in a correct C/C++ multi-threaded program in the sense that the order of specific computation steps is sometimes random. And for operations such as floating-point addition, where order of operations does matter, the exact result can be random as a consequence. But the end result should not depend dramatically on it (which is what the poster at lockdownskeptics.org claims).

by jnxx 6y ago

aspaceman 6y ago

> If this is true, the code would have race conditions, and as being impacted by race conditions is a form of undefined behavior, this would make any result of the program questionable, as the program would not be well-defined. That’s not at all what that means. What are you talking about? As long as a Monte Carlo process works towards the same result it’s equivalent. You’re speaking genuine nonsense as far as I’m concerned. Randomness doesn’t imply non deterministic. Non-determinitism in no way implies race conditions or undefined behavior. We care that the random process reaches the same result, not that the exact sequence of steps is the same. This is what scientists are talking about. A bunch of (pretty stupid) nonexperts want to criticize your code, so they feel smart on the internet.

jnxx 6y ago
I am referring to this blog post: https://lockdownsceptics.org/code-review-of-fergusons-model/ https://lockdownsceptics.org/code-review-of-fergusons-model/ It says, word-by-word: > Clearly, the documentation wants us to think that, given a starting seed, the model will always produce the same results. > >Investigation reveals the truth: the code produces critically different results, even for identical starting seeds and parameters. > I’ll illustrate with a few bugs. In issue 116 a UK “red team” at Edinburgh University reports that they tried to use a mode that stores data tables in a more efficient format for faster loading, and discovered – to their surprise – that the resulting predictions varied by around 80,000 deaths after 80 days: ... The bugs which the blog post implies here are such ones as described by Jens Regehr: https://blog.regehr.org/archives/213 https://blog.regehr.org/archives/213 Not that I do not endorse these statements in the blog - I am rather skeptical whether they are true at all. What the authors of the blob post mean is clearly "undefined behaviour" in the sense of non-deterministic program execution of a program that is not well-formed. It is clear that many non-experts could confuse that with the pseudo-randomness implicit in Monte-Carlo simulations, but this is a very different thing. The first is basically a broken, invalid, and untrustworthy program. The second is the established method to produce a computational result by introducing stochastic behavior, which is for example how modern weather models work. These are wildly different things. I do not understand why your comment just adds to the confusion between these two things?? > A bunch of (pretty stupid) nonexperts want to criticize your code, so they feel smart on the internet. As said, I don't endorse the critique in the blog. However, critique in a software implementation, as well as in scientific matters, should never carry a call on authority - it should logically explain what is the problem, with concrete points. Unfortunately, the cited blog post remains very vague about this, while claiming: > My background. I have been writing software for 30 years. I worked at Google between 2006 and 2014, where I was a senior software engineer working on Maps, Gmail and account security. I spent the last five years at a US/UK firm where I designed the company’s database product, amongst other jobs and projects. I was also an independent consultant for a couple of years. It would be much better if, instead claiming that there could be race conditions, it could point to lines in the code with actual race conditions, and show how the results of the simulation are different when the race conditions are fixed. Otherwise, it just looks like he claims that the program is buggy, because he is in no position to question the science, and does not like the result.

UncleMeat 6y ago

Race conditions aren't undefined behavior in C/C++. Data races are. Lots and lots of real systems contain race conditions without catastrophe.

jnxx 6y ago
> Race conditions aren't undefined behavior in C/C++. Data races are. You are right with the distinction, I had data race conditions in mind. Race conditions can well happen in a correct C/C++ multi-threaded program in the sense that the order of specific computation steps is sometimes random. And for operations such as floating-point addition, where order of operations does matter, the exact result can be random as a consequence. But the end result should not depend dramatically on it (which is what the poster at lockdownskeptics.org claims).