4 ms·

I read the critique in depth. Let me summarize the claims: 1. The developers had a couple of bugs that cause non-deterministic outputs in weird settings. It's

I read the critique in depth. Let me summarize the claims: 1. The developers had a couple of bugs that cause non-deterministic outputs in weird settings. It's a randomized model so this isn't a disaster, but obviously it would be better to have all non-determinism derived from a single seed. However, none of the bugs point to scientific issues. Still, these are real bugs that should be fixed. And indeed: another team pointed out some issues, they triaged the report, and it appears that they've been fixing these issues. 2. The code doesn't run well multi-threaded, and so the scientists run it on a single core. This seems like a feature the developers have de-prioritized. The critics make a HUGE deal about how using a single core is inefficient, but at the end of the day: if scientists can get simulations done in a reasonable time using a single core, this seems like a valid prioritization of limited development effort. The critics know better, obviously. 3. There aren't a bunch of unit tests. But as the authors point out code is in development and being altered continuously to reflect new developments in society and new knowledge about the virus, so it seems hard to develop unit tests that won't become broken as the underlying components change. 4. The critics seem to have a strong grounding in epidemiology. I'm just kidding, they claim to be pure software engineers, but still insist on spending a significant fraction of their critique disputing the calculation of R0 and lecturing the epidemiologists on how they think it should be calculated. This is a giant red flag given that the critics' sole stated expertise is software engineering. 5. The developers are continuing to add features to simulate new aspects of the ongoing pandemic (e.g., new code to simulate the impact of contact tracing apps) instead of fixing every minor issue in the code. Given that we're in the midst of a deadly epidemic that is putting human beings into the ground, this seems like a completely unavoidable resource allocation decision to me. 6. The critics can't even be bothered to pretend their criticism is objective. E.g., the piece concludes with "on a personal level, I’d go further and suggest that all academic epidemiology be defunded." Yeah, sure, you're a totally disinterested software expert here. I guarantee you if you look at any piece of software -- commercial or scientific, no matter the budget -- with the politically-motivated goal of tearing apart the author, you will succeed. I can say this having spent years doing security and expert witness code reviews for some of the biggest and smallest companies in the software industry. This piece isn't a critique, it's just a hatchet job. And not even a particularly creative one.

Silhouette 6y ago
That may all be true, but it doesn't change the fact that the code is written in a style that has proven to be almost totally immune to proper verification of any kind. That being the case, and regardless of the soundness or utility of the underlying science or mathematical modelling, you don't know if it works. You can't. Neither can I. Neither can the people who wrote it, or the people relying on their advice. Any arguments that the developers of this code are somehow above needing to mechanically test, or formally verify, or independently peer review the code to ensure quality are trivially debunked by the history of bug fixes and changes in behaviour that we do know about, even before making any judgements based on the apparent attempts to hide that history and to conceal the original code from before they brought in programming experts to help tidy it up. If you have indeed been an expert witness doing code reviews, are you seriously contending that you would not annihilate any case made in a court that hinged on the results of code like this?
- matthewdgreen 6y ago
  As an expert witness, I would be expected to present evidence that the code is wrong in a way that produces scientifically-invalid output. If instead I produced a bunch of nonsense about how the code would be in my opinion more efficient if it ran on multiple cores, that would be strong evidence that I failed in my task and was blowing smoke to cover it up.
  - VMG 6y ago
    Is there any evidence that the code produces scientifically-valid output?
    - matthewdgreen 6y ago
      You're asking a scientific question. To answer it you'd need to do difficult things like read scientific papers, understand epidemiology well enough to determine whether their models made sense, check their past outputs against real epidemic outcomes (accounting for the fact that national policy changes really matter). In short, you'd have to do exactly what the scientific community does all day, not just glance at some software and make loud claims. This is much more difficult than reading a Git issue tracker, so obviously the critics have nothing to offer us here.

Silhouette 6y ago

That may all be true, but it doesn't change the fact that the code is written in a style that has proven to be almost totally immune to proper verification of any kind. That being the case, and regardless of the soundness or utility of the underlying science or mathematical modelling, you don't know if it works. You can't. Neither can I. Neither can the people who wrote it, or the people relying on their advice. Any arguments that the developers of this code are somehow above needing to mechanically test, or formally verify, or independently peer review the code to ensure quality are trivially debunked by the history of bug fixes and changes in behaviour that we do know about, even before making any judgements based on the apparent attempts to hide that history and to conceal the original code from before they brought in programming experts to help tidy it up. If you have indeed been an expert witness doing code reviews, are you seriously contending that you would not annihilate any case made in a court that hinged on the results of code like this?

matthewdgreen 6y ago
As an expert witness, I would be expected to present evidence that the code is wrong in a way that produces scientifically-invalid output. If instead I produced a bunch of nonsense about how the code would be in my opinion more efficient if it ran on multiple cores, that would be strong evidence that I failed in my task and was blowing smoke to cover it up.
- VMG 6y ago
  Is there any evidence that the code produces scientifically-valid output?
  - matthewdgreen 6y ago
    You're asking a scientific question. To answer it you'd need to do difficult things like read scientific papers, understand epidemiology well enough to determine whether their models made sense, check their past outputs against real epidemic outcomes (accounting for the fact that national policy changes really matter). In short, you'd have to do exactly what the scientific community does all day, not just glance at some software and make loud claims. This is much more difficult than reading a Git issue tracker, so obviously the critics have nothing to offer us here.