5 ms·

"The model generated totally different output every time you ran it, even with fixed seeds." - I remember seeing code takedowns of the model from anti-lockdown

"The model generated totally different output every time you ran it, even with fixed seeds." - I remember seeing code takedowns of the model from anti-lockdown people who repeatedly cite this issue. But there is a valid reason for this to happen, and it doesn't mean bugs in the code. If the code is run in a distributed way (multiple threads, processes or machines), which it was, the order of execution is never guaranteed. So even setting the seed will produce a different set of results if the outcomes of each separate instance depend on each other further in the computation. There are ways to mitigate this, depending on the situation and the amount of slowdown that's acceptable. Since this model was collecting outcomes to create a statistical distribution, rather than a single deterministic number, it didn't need to. The fact the model was also drawing distributions also will be why different runs produced possibly vastly different results. Those results would be at different ends of a distribution. Only distributions are sampled and used, not single numbers. Regarding the GCSE level comment, my concern was the opposite, that the model was trying to model too much, and that inaccuracies would build up. No model is perfect (including this one) and the more assumptions made the larger the room for error. But they validated the model with some simpler models as a sanity check. My view on criticisms of the model were that they were more politically motivated, and the code takedowns were done by people who may have been good coders, but didn't know enough about statistical modeling. Here's John Cormack's take on the model, he saw and cleaned up the original version before it was released. https://mobile.twitter.com/id_aa_carmack/status/1254872368763277313?lang=en https://mobile.twitter.com/id_aa_carmack/status/125487236876...

mike_hearn 5y ago
"There are ways to mitigate this, but since the model was collecting outcomes to create a statistical distribution, rather than a single deterministic number, it didn't need to." This is the justification the academics used - because we're modelling probability distributions, bugs don't matter. Sorry, but no, this is 100% wrong. Doing statistics is not a get-out-of-jail-free card for arbitrary levels of bugginess. Firstly, the program wasn't generating a probability distribution as you claim. It produced a single set of numbers on each run. To the extent the team generated confidence intervals at all (which for Report 9 I don't think they did), it was by running the app several times and then claiming the variance in the results represented the underlying uncertainty of the data, when in reality it was representing their inability to write code properly. Secondly, remember that this model was being used to drive policy. How many hospitals shall we build? If you run the model and say 10, and then someone makes the graph formatting more helpful, reruns it and it now it says 4, that's a massive real-world difference. Nobody outside of academia thinks it's acceptable to just shrug and say, well it's just probability so it's OK for the answers to just wildly thrash around like that. Thirdly, such bugs make unit testing of your code impossible. You can't prove the correctness of a sub-calculation because it's incorporating kernel scheduling decisions into the output. Sure enough Ferguson's model had no functioning tests. If it did, they might have been able to detect all the non-threading related bugs. Finally, this "justification" breeds a culture of irresponsibility and it's exactly that endemic culture that's destroying people's confidence in science. You can easily write mathematical software that's correctly reproducible. They weren't able to do it due to a lack of care and competence. Once someone gave them this wafer thin intellectual sounding argument for why scientific reproducibility doesn't matter they started blowing off all types of bugs with your argument, including bugs like out of bounds array reads. This culture is widespread - I've talked to other programmers who worked in epidemiology and they told me about things like pointers being accidentally used in place of dereferenced values in calculations. That model had been used to support hundreds of papers. When the bugs were pointed out, the researchers lied and claimed that in only 20 minutes they'd checked all the results and the bugs had no impact on any of them. Once a team goes down the route of "our bugs are just CIs on a probability distribution" they have lost the plot and their work deserves to be classed as dangerous misinformation. "My view on criticisms of the model were that it was more politically motivated" Are you an academic? Because that's exactly the take they like to always use - any criticism of academia is "political" or "ideological". But expecting academics to produce work that isn't filled with bugs isn't politically motivated. It's basic stuff. For as long as people defend this obvious incompetence, people's trust in science will correctly continue to plummet. "Here's John Cormack's take on the model, he saw and cleaned up the original version before it was released. https://mobile.twitter.com/id_aa_carmack/status/125487236876 https://mobile.twitter.com/id_aa_carmack/status/125487236876..." If you check the commit history, you'll see that he quite obviously didn't work with the code much at all. Regardless, if he thinks the model is not worthless, he's wrong. Anyone who reviews their bug tracker can see that immediately.
- wizzwizz4 5y ago
  > To the extent the team generated confidence intervals at all (which for Report 9 I don't think they did), it was by running the app several times and then claiming the variance in the results represented the underlying uncertainty of the data, when in reality it was representing their inability to write code properly. Functionally, what's the difference? The output of their model varied based on environmental factors (how the OS chose to schedule things). The lower-order bits of some of the values got corrupted, due to floating-point errors. In essence, their model had noise, bias, and lower precision than a floating point number – all things that scientists are used to.
  - mike_hearn 5y ago
    Argh. Scientists are used to some level of unavoidable noise from experiments done on the natural world because the natural world is not fully controllable. Thus they are expected to work hard to minimize the uncertainty in their measurements, then characterize what's left and take that into account in their calculations. They are not expected to make beginner level mistakes when solving simple mathematical equations. Avoidable errors introduced by doing their maths wrong is fundamentally different to unavoidable measurement uncertainty. The whole point of doing simulations in silico is to avoid the problems of the natural world and give you a fully controllable and precisely measurable environment, in which you can re-run the simulation whilst altering only a single variable. That's the justification for creating these sorts of models in the first place! Perhaps you think the errors were small. The errors in their model due to their bugs were of the same order of magnitude as the predictions themselves. They knew this but presented the outputs to the government as "science" anyway, then systematically attacked the character and motives of anyone who pointed out they were making mistakes. Every single member of that team should have been fired years ago, yet instead what happened is the attitude you're displaying here: a widespread argument that scientists shouldn't be or can't be held to the quality standards we expect of a $10 video game. How can anyone trust the output of "science" when this attitude is so widespread? We wouldn't accept this kind of argument from people in any other field.
- mrtedbear 5y ago
  At the time, critics of the model were claiming the model was buggy because multiple runs would produce different results. My comment above explains why that is not evidence for the model being buggy. Report 9 talks about parameters being modeled as probability distributions, i.e. its a stochastic model. I doubt they would draw conclusions from a single run, as the code is drawing a single sample from a probability distribution. And, if you look at the paper describing the original model (cited in report 9), they do test the model with multiple runs. On top of that they perform sensitivity analyses to check erroneous assumptions aren't driving the model. I have spent time in academia, but I'm not an academic, and don't feel any obligation to fly the flag for academia. Regarding the politics, contrast how the people who forensically examined Ferguson's papers were so ready to accept the competing (and clearly incorrect https://www.youtube.com/watch?v=DKh6kJ-RSMI https://www.youtube.com/watch?v=DKh6kJ-RSMI) results from Sunetra Gupta's group. Fair point about academic code being messy. It's a big issue, but the incentives are not there at the moment to write quality code. I assume you're a programmer - if you wanted to be the change you want to see, you could join an academic group, reduce your salary by 3x-4x, and be in a place where what you do is not a priority.
kortex 5y ago
> "The model generated totally different output every time you ran it, even with fixed seeds." - I remember seeing code takedowns of the model from anti-lockdown people who repeatedly cite this issue. But there is a valid reason for this to happen, and it doesn't mean bugs in the code. If the code is run in a distributed way (multiple threads, processes or machines), which it was, the order of execution is never guaranteed. Then there's literally no point to using PRNG seeding. The whole point of PRNG seeding is so you can define some model in terms of "def model(inputs, state) ->" output, and get the "same" output for the same input. I put "same" in quotes because defining sameness on FP hardware is challenging. But usually 0.001% relative tolerance is sufficiently "same" to account for FP implementation weirdness. If you can't do that, then your model is not a pure function, in which case setting the seed is pointless at best, and biasing/false sense of security in the worst case. As you mention, non-pure models have their place, but reproducing their results is very challenging, and requires generating distributions with error bars - you essentially "zoom out" until you are a pure function again, with respect to aggregate statistics. It does not sound like this model was "zoomed out" enough to provide adequate confidence intervals such that you could run the simulation, and statistically guarantee you'd get a result in-bounds.
- mrtedbear 5y ago
  I reckon the PRNG seeding in such a case might be used during development/testing. So, run the code with a seed in a non distributed way (e.g. in R turn off all parallelism), and then the results should be the same in every run. Then once this test output is validated, depending on the nature of the model, it can be run in parallel, and guarantees of deterministic behaviour will go, but that's ok. I didn't develop the model, so can't really say anything in depth beyond the published materials. I just found it odd at the time how this specific detail was incorrectly used by some to claim the model was broken/irredeemably buggy. Edit: Actually, in general, perhaps there's one other situation where the seed might be useful, assuming you have used a seed in the first place. Depending on the distributed environment, there's no guarantee that the processes or random number draws will be run in the same order. But it might be that in most cases they're in the same order. This might bias the distribution of the samples you take. So you might want to change the seed on every run to protect yourself from such nasty phantom effects.

mike_hearn 5y ago

"There are ways to mitigate this, but since the model was collecting outcomes to create a statistical distribution, rather than a single deterministic number, it didn't need to." This is the justification the academics used - because we're modelling probability distributions, bugs don't matter. Sorry, but no, this is 100% wrong. Doing statistics is not a get-out-of-jail-free card for arbitrary levels of bugginess. Firstly, the program wasn't generating a probability distribution as you claim. It produced a single set of numbers on each run. To the extent the team generated confidence intervals at all (which for Report 9 I don't think they did), it was by running the app several times and then claiming the variance in the results represented the underlying uncertainty of the data, when in reality it was representing their inability to write code properly. Secondly, remember that this model was being used to drive policy. How many hospitals shall we build? If you run the model and say 10, and then someone makes the graph formatting more helpful, reruns it and it now it says 4, that's a massive real-world difference. Nobody outside of academia thinks it's acceptable to just shrug and say, well it's just probability so it's OK for the answers to just wildly thrash around like that. Thirdly, such bugs make unit testing of your code impossible. You can't prove the correctness of a sub-calculation because it's incorporating kernel scheduling decisions into the output. Sure enough Ferguson's model had no functioning tests. If it did, they might have been able to detect all the non-threading related bugs. Finally, this "justification" breeds a culture of irresponsibility and it's exactly that endemic culture that's destroying people's confidence in science. You can easily write mathematical software that's correctly reproducible. They weren't able to do it due to a lack of care and competence. Once someone gave them this wafer thin intellectual sounding argument for why scientific reproducibility doesn't matter they started blowing off all types of bugs with your argument, including bugs like out of bounds array reads. This culture is widespread - I've talked to other programmers who worked in epidemiology and they told me about things like pointers being accidentally used in place of dereferenced values in calculations. That model had been used to support hundreds of papers. When the bugs were pointed out, the researchers lied and claimed that in only 20 minutes they'd checked all the results and the bugs had no impact on any of them. Once a team goes down the route of "our bugs are just CIs on a probability distribution" they have lost the plot and their work deserves to be classed as dangerous misinformation. "My view on criticisms of the model were that it was more politically motivated" Are you an academic? Because that's exactly the take they like to always use - any criticism of academia is "political" or "ideological". But expecting academics to produce work that isn't filled with bugs isn't politically motivated. It's basic stuff. For as long as people defend this obvious incompetence, people's trust in science will correctly continue to plummet. "Here's John Cormack's take on the model, he saw and cleaned up the original version before it was released. https://mobile.twitter.com/id_aa_carmack/status/125487236876 https://mobile.twitter.com/id_aa_carmack/status/125487236876..." If you check the commit history, you'll see that he quite obviously didn't work with the code much at all. Regardless, if he thinks the model is not worthless, he's wrong. Anyone who reviews their bug tracker can see that immediately.

wizzwizz4 5y ago
> To the extent the team generated confidence intervals at all (which for Report 9 I don't think they did), it was by running the app several times and then claiming the variance in the results represented the underlying uncertainty of the data, when in reality it was representing their inability to write code properly. Functionally, what's the difference? The output of their model varied based on environmental factors (how the OS chose to schedule things). The lower-order bits of some of the values got corrupted, due to floating-point errors. In essence, their model had noise, bias, and lower precision than a floating point number – all things that scientists are used to.
- mike_hearn 5y ago
  Argh. Scientists are used to some level of unavoidable noise from experiments done on the natural world because the natural world is not fully controllable. Thus they are expected to work hard to minimize the uncertainty in their measurements, then characterize what's left and take that into account in their calculations. They are not expected to make beginner level mistakes when solving simple mathematical equations. Avoidable errors introduced by doing their maths wrong is fundamentally different to unavoidable measurement uncertainty. The whole point of doing simulations in silico is to avoid the problems of the natural world and give you a fully controllable and precisely measurable environment, in which you can re-run the simulation whilst altering only a single variable. That's the justification for creating these sorts of models in the first place! Perhaps you think the errors were small. The errors in their model due to their bugs were of the same order of magnitude as the predictions themselves. They knew this but presented the outputs to the government as "science" anyway, then systematically attacked the character and motives of anyone who pointed out they were making mistakes. Every single member of that team should have been fired years ago, yet instead what happened is the attitude you're displaying here: a widespread argument that scientists shouldn't be or can't be held to the quality standards we expect of a $10 video game. How can anyone trust the output of "science" when this attitude is so widespread? We wouldn't accept this kind of argument from people in any other field.
mrtedbear 5y ago
At the time, critics of the model were claiming the model was buggy because multiple runs would produce different results. My comment above explains why that is not evidence for the model being buggy. Report 9 talks about parameters being modeled as probability distributions, i.e. its a stochastic model. I doubt they would draw conclusions from a single run, as the code is drawing a single sample from a probability distribution. And, if you look at the paper describing the original model (cited in report 9), they do test the model with multiple runs. On top of that they perform sensitivity analyses to check erroneous assumptions aren't driving the model. I have spent time in academia, but I'm not an academic, and don't feel any obligation to fly the flag for academia. Regarding the politics, contrast how the people who forensically examined Ferguson's papers were so ready to accept the competing (and clearly incorrect https://www.youtube.com/watch?v=DKh6kJ-RSMI https://www.youtube.com/watch?v=DKh6kJ-RSMI) results from Sunetra Gupta's group. Fair point about academic code being messy. It's a big issue, but the incentives are not there at the moment to write quality code. I assume you're a programmer - if you wanted to be the change you want to see, you could join an academic group, reduce your salary by 3x-4x, and be in a place where what you do is not a priority.

kortex 5y ago

> "The model generated totally different output every time you ran it, even with fixed seeds." - I remember seeing code takedowns of the model from anti-lockdown people who repeatedly cite this issue. But there is a valid reason for this to happen, and it doesn't mean bugs in the code. If the code is run in a distributed way (multiple threads, processes or machines), which it was, the order of execution is never guaranteed. Then there's literally no point to using PRNG seeding. The whole point of PRNG seeding is so you can define some model in terms of "def model(inputs, state) ->" output, and get the "same" output for the same input. I put "same" in quotes because defining sameness on FP hardware is challenging. But usually 0.001% relative tolerance is sufficiently "same" to account for FP implementation weirdness. If you can't do that, then your model is not a pure function, in which case setting the seed is pointless at best, and biasing/false sense of security in the worst case. As you mention, non-pure models have their place, but reproducing their results is very challenging, and requires generating distributions with error bars - you essentially "zoom out" until you are a pure function again, with respect to aggregate statistics. It does not sound like this model was "zoomed out" enough to provide adequate confidence intervals such that you could run the simulation, and statistically guarantee you'd get a result in-bounds.

mrtedbear 5y ago
I reckon the PRNG seeding in such a case might be used during development/testing. So, run the code with a seed in a non distributed way (e.g. in R turn off all parallelism), and then the results should be the same in every run. Then once this test output is validated, depending on the nature of the model, it can be run in parallel, and guarantees of deterministic behaviour will go, but that's ok. I didn't develop the model, so can't really say anything in depth beyond the published materials. I just found it odd at the time how this specific detail was incorrectly used by some to claim the model was broken/irredeemably buggy. Edit: Actually, in general, perhaps there's one other situation where the seed might be useful, assuming you have used a seed in the first place. Depending on the distributed environment, there's no guarantee that the processes or random number draws will be run in the same order. But it might be that in most cases they're in the same order. This might bias the distribution of the samples you take. So you might want to change the seed on every run to protect yourself from such nasty phantom effects.