4 ms·

Is there any evidence that the code produces scientifically-valid output?

by VMG 6y ago

matthewdgreen 6y ago
You're asking a scientific question. To answer it you'd need to do difficult things like read scientific papers, understand epidemiology well enough to determine whether their models made sense, check their past outputs against real epidemic outcomes (accounting for the fact that national policy changes really matter). In short, you'd have to do exactly what the scientific community does all day, not just glance at some software and make loud claims. This is much more difficult than reading a Git issue tracker, so obviously the critics have nothing to offer us here.
- rpdillon 6y ago
  I don't claim any expertise at all, but without documentation and tests (both of which are called out as largely missing for this codebase), how can the expert epidemiologists tell if the software is behaving correctly?
  - matthewdgreen 6y ago
    I presume they run it many times and compare the final and intermediate outputs to reality, then adjust parameters and see how it performs under those parameters. And other models do similar things and compare to you. There are hundreds of papers on Google Scholar about epidemic simulation. This isn't an area where your performance is going to be determined by a few unit tests.
    - rpdillon 6y ago
      So the argument is that determining the correctness of the simulation is so complex that any kind of automated tests won't matter. The rebuttal makes a couple of statements about how scientists spend their time to verify their work: > Scientific software tends to be validated in a more ad hoc way. This is a domain-specific norm, not a matter of competence. > Practically every scientist I know spends most of their time checking their results rather than coding, and I sincerely doubt that the Imperial group is any different. I wonder if the team could iterate more quickly if some of these ad hoc validations were automated.
- Silhouette 6y ago
  You're asking a scientific question. But it's not just a science question. That is the point of the criticism. Assuming you are the Matthew D Green known for his cryptography work, surely you of all people understand that a poor implementation can have profound consequences even if the original mathematics is sound?
  - matthewdgreen 6y ago
    I am, and I've written both production and non-production software. I know that every piece of software has limited development resources, and has different demands on those resources. We're in the midst of a pandemic right now and demands are particularly acute for fresh simulation data, with simulation parameters changing constantly based on new information and policy proposals. So what do you do with your limited resources? Do you tell the government you can't get them any preliminary data because your multithreading system doesn't work, and you need four months to refactor the code (even though you could just run single-core)? Do you tell them that because you have to write a complete unit testing framework, you can't give them any modeling data or update your model parameters, sorry, good luck with everything. Do you tell them that another 30K people may have to die based on wrong policy choices because you have decided to prioritize "shipping-production-quality" software standards, even though you are not shipping your software? You can pretend that this is not a trolley problem. But in this case it literally is. You are standing at the switch in the tracks, and on one side is a detailed set of unit tests and multi-core support, and on the other hand might be tens of thousands of living people. And you know your code better than anyone else in the world. What do you choose? I don't have an answer. I'm saying the answer is hard, and we're not equipped to lay claims on it based on glancing at a Git repo.
    - Silhouette 6y ago
      I'm not arguing that there is much you can usefully change now. It's probably far too late to make much material difference to public policy now, for the reasons you have set out yourself. But this could surely have been much better handled during a usefully long window back towards the start of the year when attention was starting to be paid to the virus but it hadn't taken hold in the UK on any significant scale yet. Also, just to be clear, all of the specific issues with the Imperial code that you've mentioned here and in other parts of our discussion today, such as whether to rewrite for better multi-core behaviour or just use single-core, are coming from you not me. My argument is a much simpler one: when it started to become clear that this virus was a significant risk, the government should have provided the necessary resources and the expert advisors should have switched to a model where scientists did the science and programmers did the programming. I don't think anyone can credibly claim that there wasn't enough awareness that this could potentially become a serious issue and so that bringing in broader expertise was appropriate by the time we had things like COBRA meetings being held and secret expert advisory groups being assembled.
- c0sin 6y ago
  Let's put it this way: were there any of your peers - who read all the papers, understood epidemiology and all other things you have just suggested - able to reproduce the results and confirm them as a solid ground for the policy recommendations? Perhaps, you can also share with us the _original_ code of the model, based on which the "Report 9" has been made? So far, as I know, those calls went answered.

Is there any evidence that the code produces scientifically-valid output?

by VMG 6y ago

Is there any evidence that the code produces scientifically-valid output?

You're asking a scientific question. To answer it you'd need to do difficult things like read scientific papers, understand epidemiology well enough to determine whether their models made sense, check their past outputs against real epidemic outcomes (accounting for the fact that national policy changes really matter). In short, you'd have to do exactly what the scientific community does all day, not just glance at some software and make loud claims. This is much more difficult than reading a Git issue tracker, so obviously the critics have nothing to offer us here.

rpdillon 6y ago
I don't claim any expertise at all, but without documentation and tests (both of which are called out as largely missing for this codebase), how can the expert epidemiologists tell if the software is behaving correctly?
- matthewdgreen 6y ago
  I presume they run it many times and compare the final and intermediate outputs to reality, then adjust parameters and see how it performs under those parameters. And other models do similar things and compare to you. There are hundreds of papers on Google Scholar about epidemic simulation. This isn't an area where your performance is going to be determined by a few unit tests.
  - rpdillon 6y ago
    So the argument is that determining the correctness of the simulation is so complex that any kind of automated tests won't matter. The rebuttal makes a couple of statements about how scientists spend their time to verify their work: > Scientific software tends to be validated in a more ad hoc way. This is a domain-specific norm, not a matter of competence. > Practically every scientist I know spends most of their time checking their results rather than coding, and I sincerely doubt that the Imperial group is any different. I wonder if the team could iterate more quickly if some of these ad hoc validations were automated.
Silhouette 6y ago
You're asking a scientific question. But it's not just a science question. That is the point of the criticism. Assuming you are the Matthew D Green known for his cryptography work, surely you of all people understand that a poor implementation can have profound consequences even if the original mathematics is sound?
- matthewdgreen 6y ago
  I am, and I've written both production and non-production software. I know that every piece of software has limited development resources, and has different demands on those resources. We're in the midst of a pandemic right now and demands are particularly acute for fresh simulation data, with simulation parameters changing constantly based on new information and policy proposals. So what do you do with your limited resources? Do you tell the government you can't get them any preliminary data because your multithreading system doesn't work, and you need four months to refactor the code (even though you could just run single-core)? Do you tell them that because you have to write a complete unit testing framework, you can't give them any modeling data or update your model parameters, sorry, good luck with everything. Do you tell them that another 30K people may have to die based on wrong policy choices because you have decided to prioritize "shipping-production-quality" software standards, even though you are not shipping your software? You can pretend that this is not a trolley problem. But in this case it literally is. You are standing at the switch in the tracks, and on one side is a detailed set of unit tests and multi-core support, and on the other hand might be tens of thousands of living people. And you know your code better than anyone else in the world. What do you choose? I don't have an answer. I'm saying the answer is hard, and we're not equipped to lay claims on it based on glancing at a Git repo.
  - Silhouette 6y ago
    I'm not arguing that there is much you can usefully change now. It's probably far too late to make much material difference to public policy now, for the reasons you have set out yourself. But this could surely have been much better handled during a usefully long window back towards the start of the year when attention was starting to be paid to the virus but it hadn't taken hold in the UK on any significant scale yet. Also, just to be clear, all of the specific issues with the Imperial code that you've mentioned here and in other parts of our discussion today, such as whether to rewrite for better multi-core behaviour or just use single-core, are coming from you not me. My argument is a much simpler one: when it started to become clear that this virus was a significant risk, the government should have provided the necessary resources and the expert advisors should have switched to a model where scientists did the science and programmers did the programming. I don't think anyone can credibly claim that there wasn't enough awareness that this could potentially become a serious issue and so that bringing in broader expertise was appropriate by the time we had things like COBRA meetings being held and secret expert advisory groups being assembled.
c0sin 6y ago
Let's put it this way: were there any of your peers - who read all the papers, understood epidemiology and all other things you have just suggested - able to reproduce the results and confirm them as a solid ground for the policy recommendations? Perhaps, you can also share with us the _original_ code of the model, based on which the "Report 9" has been made? So far, as I know, those calls went answered.