4 ms·

This is an incredibly poor take. It's well known that academic software doesn't follow great software engineering practices. The model is an incredibly complex

This is an incredibly poor take. It's well known that academic software doesn't follow great software engineering practices. The model is an incredibly complex piece of software that attempts to model stochastic behaviour. That it isn't fully deterministic (with a fixed random seed) doesn't make thee research invalid and shouldn't discredit research that builds on the model. >On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best done by the insurance sector. This is another level of crazy.

hartator 6y ago
> It's well known that academic software doesn't follow great software engineering practices. Well, know that literally trillions of dollars have been lost, it's not a surprise that the computer model at the origin of the panic is being scrutinized.
s9w 6y ago
Why do you think that's a crazy conclusion? That code is worthless - such quality of work would not be accepted in the private sector. And it's doubtful it can be of any value at all. So when the work they do is this bad, I think it's reasonable to question whether they should be continued to be paid for no value to the public.
- lbeltrame 6y ago
  > such quality of work would not be accepted in the private sector. To be fair, if such code would be used for a paper and the flaws came out, I think it would warrant a retraction. I think I saw not too long ago a story right here where a software error caused completely wrong results.
vikinghckr 6y ago
>This is an incredibly poor take. No it isn't. > It's well known that academic software doesn't follow great software engineering practices. That doesn't make it acceptable. Also, it's also "well known" that vast majority of academic work has zero tangible effect on society. This isn't one of those works. It's possibly the most important piece of academic work that has happened in recent memory. So the bar for this is MUCH higher than typical academic research work. >. That it isn't fully deterministic (with a fixed random seed) doesn't make thee research invalid and shouldn't discredit research that builds on the model. It does make it invalid, when the difference between runs is as big as 80,000 estimated deaths which can lead to dramatically different government policies. > This is another level of crazy. No it's not. Academia is way behind the industry when it comes to modeling the economy and the real world.
- spamizbad 6y ago
  > No it's not. Academia is way behind the industry when it comes to modeling the economy and the real world. The insurance industry is expected to ask the government for bailouts because none of their models can account for the fallout from this, just like AIG did during the '08 crisis.
  - smsm42 6y ago
    Well, even if they did they'd ask for it because they know they'd likely get it. And then what exactly is the incentive to account for it and pay for it with your own money if you can not account for it and get money from the government if anything bad happens?
  - twelve40 6y ago
    Not to protect the insurance industry, which I know nothing about, but how does their hypothetical failure, misery and bailout relate to, or validate the study in question?
    - arkades 6y ago
      Criticism: Academia fails to model the real world, in contrast to industry ("is way behind industry"). Response: An entire industry of competing and well-funded actors that specialize in modeling and predicting expensive failure states also failed to model the world accurately. Implication of Response: Where is the evidence that this is inferior to industry?
      - no-s 6y ago
        >>Implication of Response: Where is the evidence that this is inferior to industry? A meta implication is that expert models were insufficient to inform decisions.
- Nimitz14 6y ago
  > It does make it invalid, when the difference between runs is as big as 80,000 estimated deaths which can lead to dramatically different government policies. Wrong. Nobody decides policy based on whether there will be 320k or 400k deaths. What matters is the order of magnitude.
below43 6y ago
Suggesting all academic epidemiology be defunded really highlights the true intentions and bias of the writer. It might be better, perhaps, for an academic to perform a code review instead.
- lbeltrame 6y ago
  Regardless of the bias of the piece, AFAICT the GH issues linked to weren't made by the author (to be honest, my own bias suggested, before reading the text, that the author was the only one that pointed out the mistakes) but by others trying to use the model. This makes the criticism (although partisan) at least worthy of attention.
smsm42 6y ago
There's a big difference between randomized model and code giving random results because you have race conditions in the code. If you do an experiment, you can expect some randomness because no experiment gives same value ever. But behind it is the common reality that you can measure. Now, if during an experiment somebody would come and kick the equipment, spit on it, spill water on it, subject it to random electric shocks, shoot at it, etc. and then when results come out all over the place say "well, it's supposed to be random!", completely ignoring what was done to it - you'd say that person is insane, it's supposed to be random but not like that. If you code has race conditions that make your simulation roll out and produce different results, then you aren't measuring the result of the simulation, you are measuring race conditions. Of course, you could be lucky and race condition could be harmless or contribute almost nothing to the result. But this is unusual, needs proof and indications are it's not the case. Responding to that "well, it's stochastic so what do you want!" is not the way it works. Poor code quality is a sad reality in many scientific projects, yes. But it's not just a code that is unreadable. It's the code where race conditions knowingly mess with the model results! You can't just dismiss that.
ilyaeck 6y ago
Big decisions impacting many livelihoods were made based on this model's projections. Its scope is way beyond academic.

hartator 6y ago

> It's well known that academic software doesn't follow great software engineering practices. Well, know that literally trillions of dollars have been lost, it's not a surprise that the computer model at the origin of the panic is being scrutinized.

s9w 6y ago

Why do you think that's a crazy conclusion? That code is worthless - such quality of work would not be accepted in the private sector. And it's doubtful it can be of any value at all. So when the work they do is this bad, I think it's reasonable to question whether they should be continued to be paid for no value to the public.

lbeltrame 6y ago
> such quality of work would not be accepted in the private sector. To be fair, if such code would be used for a paper and the flaws came out, I think it would warrant a retraction. I think I saw not too long ago a story right here where a software error caused completely wrong results.

vikinghckr 6y ago

>This is an incredibly poor take. No it isn't. > It's well known that academic software doesn't follow great software engineering practices. That doesn't make it acceptable. Also, it's also "well known" that vast majority of academic work has zero tangible effect on society. This isn't one of those works. It's possibly the most important piece of academic work that has happened in recent memory. So the bar for this is MUCH higher than typical academic research work. >. That it isn't fully deterministic (with a fixed random seed) doesn't make thee research invalid and shouldn't discredit research that builds on the model. It does make it invalid, when the difference between runs is as big as 80,000 estimated deaths which can lead to dramatically different government policies. > This is another level of crazy. No it's not. Academia is way behind the industry when it comes to modeling the economy and the real world.

spamizbad 6y ago
> No it's not. Academia is way behind the industry when it comes to modeling the economy and the real world. The insurance industry is expected to ask the government for bailouts because none of their models can account for the fallout from this, just like AIG did during the '08 crisis.
- smsm42 6y ago
  Well, even if they did they'd ask for it because they know they'd likely get it. And then what exactly is the incentive to account for it and pay for it with your own money if you can not account for it and get money from the government if anything bad happens?
- twelve40 6y ago
  Not to protect the insurance industry, which I know nothing about, but how does their hypothetical failure, misery and bailout relate to, or validate the study in question?
  - arkades 6y ago
    Criticism: Academia fails to model the real world, in contrast to industry ("is way behind industry"). Response: An entire industry of competing and well-funded actors that specialize in modeling and predicting expensive failure states also failed to model the world accurately. Implication of Response: Where is the evidence that this is inferior to industry?
    - no-s 6y ago
      >>Implication of Response: Where is the evidence that this is inferior to industry? A meta implication is that expert models were insufficient to inform decisions.
Nimitz14 6y ago
> It does make it invalid, when the difference between runs is as big as 80,000 estimated deaths which can lead to dramatically different government policies. Wrong. Nobody decides policy based on whether there will be 320k or 400k deaths. What matters is the order of magnitude.

below43 6y ago

Suggesting all academic epidemiology be defunded really highlights the true intentions and bias of the writer. It might be better, perhaps, for an academic to perform a code review instead.

lbeltrame 6y ago
Regardless of the bias of the piece, AFAICT the GH issues linked to weren't made by the author (to be honest, my own bias suggested, before reading the text, that the author was the only one that pointed out the mistakes) but by others trying to use the model. This makes the criticism (although partisan) at least worthy of attention.

smsm42 6y ago

There's a big difference between randomized model and code giving random results because you have race conditions in the code. If you do an experiment, you can expect some randomness because no experiment gives same value ever. But behind it is the common reality that you can measure. Now, if during an experiment somebody would come and kick the equipment, spit on it, spill water on it, subject it to random electric shocks, shoot at it, etc. and then when results come out all over the place say "well, it's supposed to be random!", completely ignoring what was done to it - you'd say that person is insane, it's supposed to be random but not like that. If you code has race conditions that make your simulation roll out and produce different results, then you aren't measuring the result of the simulation, you are measuring race conditions. Of course, you could be lucky and race condition could be harmless or contribute almost nothing to the result. But this is unusual, needs proof and indications are it's not the case. Responding to that "well, it's stochastic so what do you want!" is not the way it works. Poor code quality is a sad reality in many scientific projects, yes. But it's not just a code that is unreadable. It's the code where race conditions knowingly mess with the model results! You can't just dismiss that.

ilyaeck 6y ago

Big decisions impacting many livelihoods were made based on this model's projections. Its scope is way beyond academic.