5 ms·

This letter feels as though it is overlooking a large point of contention. >The scientists who wrote this horrible code most probably had no training in softwa

This letter feels as though it is overlooking a large point of contention. >The scientists who wrote this horrible code most probably had no training in software engineering, and no funding to hire software engineers Shouldn't the argument be, that for research that is reliant on coding models, funding be allocated to experts that can assist in creating said models (software engineers)?

deleted 6y ago
[deleted]
klenwell 6y ago
The conclusions from the first critical code review cited: All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one. On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best done by the insurance sector. Insurers employ modellers and data scientists, but also employ managers whose job is to decide whether a model is accurate enough for real world usage and professional software engineers to ensure model software is properly tested, understandable and so on. Academic efforts don’t have these people, and the results speak for themselves. https://lockdownsceptics.org/code-review-of-fergusons-model/ https://lockdownsceptics.org/code-review-of-fergusons-model/
- rscho 6y ago
  Yeah really, why don't we leave out all modelling duties to companies who optimize for more money instead of leaving it with the only actor trying to optimize towards actual public health? What could go wrong?
  - SpicyLemonZest 6y ago
    If you're willing to dismiss all companies as optimizing for more money, it seems only fair to say that academics optimize for prestige and publication in good journals.
    - rscho 6y ago
      That's also true. Now the million $ question: Whom would you like our society to rely on to generate quality work? I don't think there is a satisfactory answer to this question. Public research becomes more and more of an industry every year with the publish-or-perish game, while a solely private solution is obviously open to very biased conclusions. There is no smart solution to a stupid problem. But the truth is that _as an institution_ the NHS is the only actor whose mission is to optimize towards public health.
      - SpicyLemonZest 6y ago
        In most fields, our society relies on private industry to generate quality work, even when the work is very important and doing it wrong might kill people. I'm not an anarchist, I do recognize there are reasons that the government should provide some things. But the idea that private industry uniformly produces bad results because they don't care about anything but profit just seems silly to me. Producing good results is profitable!
        mturmon 6y ago
        > In most fields, our society relies on private industry to generate quality work ... This is just not true. Military, police, courts? What does "most fields" even mean?
        SpicyLemonZest 6y ago
        My job, food, apartment, utilities, entertainment, all come from private companies.
    - klyrs 6y ago
      > it seems only fair to say that academics optimize for prestige and publication in good journals. On one hand you have lots of people arguing that the legal duty of a company is only to make money for its shareholders. When large companies fail at that goal, it's bailout time. On the other hand you have peer-reviewed journals where authors are incentivized to find accurate results, and researchers will cite articles on the basis of their veracity (or, if incorrect, as punching bags). Of course that's a fallible process and just as vulnerable to cronyism, but when researchers are caught cooking the books they're discredited, not rewarded.
  - toolz 6y ago
    Everyone working for a living optimizes their work towards making more money. That doesn't change for people funded with public money. It's pretty widely accepted that the people in charge of public funding (politicians) sometimes act outside of the publics interest for self-gain. I make no claim as to which achieves better results for the public because it's such a complicated problem, but I think it's rather naive to just assume publicly funded incentives are more aligned with social health than private incentives.
- lern_too_spel 6y ago
  That was a shitty code review. Seeding issues like the ones cited don't affect the results of a Monte Carlo simulation, and there are tests in the repo, just not automated ones. The section you quoted shows the reason for the review's sloppiness. The reviewer set out to find a way to justify their own beliefs instead of to actually read the code.
  - Joeboy 6y ago
    > That was a shitty code review Damn straight. The only sense in which it was not a shitty code review, is that it didn't actually review any code. Looking into the linked tickets is a time consuming faff, but people should actually do it before taking the blog post at face value.
  - klenwell 6y ago
    I agree. Moreover, the cited bug ("predictions varied by around 80,000 deaths after 80 days") doesn't really seem to impact the over all policy implications: no lockdown means exceptional number (400 to 480k) of deaths. Unless the entire simulation is bogus, it comes off as nitpicking.
  - thu2111 6y ago
    Seriously? Doing a Monte Carlo simulation means you adjust the seeds to get different runs. It doesn't mean your program can read uninitialised memory or reuse variables that weren't reset to zero and still be correct. Where are people getting this idea that you can just average away the results of out-of-bounds reads and race conditions?
    - lern_too_spel 6y ago
      What does reading uninitialized memory or reusing variables that weren't set to zero have to do with seeding issues? Read my comment again and reply to its content instead of making up a comment that you would like to reply to.
- Nursie 6y ago
  Yeah that's a weird set of conclusions there. I would think the papers based on it want another look though.
- pjc50 6y ago
  That's the most biased opinion imaginable by an astroturf group. There's no information on who they are, but the article currently on their front page is by Toby Young, who is a Spectator/Quillete guy and to be found backing almost all stupid ideas within British politics. https://www.spectator.co.uk/article/This-lockdown-may-kill-me https://www.spectator.co.uk/article/This-lockdown-may-kill-m...
- spamizbad 6y ago
  That last part is mindblowing given the fact that the health insurance industry in the US is trying to argue it shouldn’t have to pay for COVID-19 treatments because it’s part of a pandemic and not part of normal medical treatments. I guess if your model is: if pandemic and COVID: is_covered = False You can have a very clean pandemic model.
lbeltrame 6y ago
Back in 2008-2009 my then PI tried this: it worked for a while, but then it proved to be untenable. Not because there wasn't enough money (there was) but because the university did not like the idea to get someone on board just to help researchers develop software, and thus there were so many roadblocks at one point that it was impossible to go on.
- btrettel 6y ago
  I'm interested in hearing more details on this, if you're able to discuss them.
  - lbeltrame 6y ago
    I don't honestly know how much I can disclose (FTR, I was not the one doing that job - I was working as a postdoc there at the time), but I'll try to summarize it briefly: - PI got a couple of EU funded grants - PI wanted to build a program / programs out of some ideas he had before moving to the institution he was currently employed at - PI hired a software developer (actually a software engineer) to do this job - Statistician drafted algorithms, developer created the software - I used the software for my Ph.D. thesis, got hired at the lab of the PI as a postdoc, new requirements arose due to the way I used it - Developer made a preliminary version of a new version of the software following discussions with me - University made it very hard to keep developer on the team, due to bureaucracy and kind of hostility towards this kind of employment - At some point (I can't recall the details exactly, but it was something that spanned almost one year), the form of keeping the developer on board was no longer possible - PI offers an alternative contract, but it is financially wasteful to the developer (not the fault of the PI, but the way certain things work in my country) - Developer leaves the project Also the university, to my knowledge, complained that the developer cost a lot (IIRC, the project was paid at market rate, so in line with other, non academic software projects). I can't comment on the quality of the software we used (sadly it was never open sourced) as it was in Java and I only have a passing understanding of the language, but the approach of having a dedicated developer IMO worked (and also net quite a number of publications over that period).
malandrew 6y ago
> experts Being an expert means you know your tools well. If you can't code well and coding is a critical part of your toolset, then you're not an expert yet. You don't get a free pass because you allegedly are strong in other parts of your craft, especially if the part of your craft you're weak in can be this problematic.
rch 6y ago
For this to happen there would need to be a nationally accepted PE certification for 'software engineers' on par with other engineering disciplines. It's unreasonable to expect non-experts, and experts from other fields, to perform credentialing on a case by case basis.
- vonmoltke 6y ago
  There is no "nationally accepted" PE license for any discipline. It's a state-level credential and reciprocity is not universal or automatic.
  - rch 6y ago
    I included the "nationally accepted" bit since I'd prefer to see some incremental improvement over current models, but I don't have confidence that an international standard would be accepted in all relevant contexts. It could still be administered at the state level. I'm also ignoring the requirement that candidates must start with a degree from an ABET-accredited institution, which seems to feature prominently in proposals from IEEE and others. Ideally I think there should be some way around that, but alternatives I'm familiar with aren't great either (e.g. FINRA).

deleted 6y ago

[deleted]

klenwell 6y ago

The conclusions from the first critical code review cited: All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one. On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best done by the insurance sector. Insurers employ modellers and data scientists, but also employ managers whose job is to decide whether a model is accurate enough for real world usage and professional software engineers to ensure model software is properly tested, understandable and so on. Academic efforts don’t have these people, and the results speak for themselves. https://lockdownsceptics.org/code-review-of-fergusons-model/ https://lockdownsceptics.org/code-review-of-fergusons-model/

rscho 6y ago
Yeah really, why don't we leave out all modelling duties to companies who optimize for more money instead of leaving it with the only actor trying to optimize towards actual public health? What could go wrong?
- SpicyLemonZest 6y ago
  If you're willing to dismiss all companies as optimizing for more money, it seems only fair to say that academics optimize for prestige and publication in good journals.
  - rscho 6y ago
    That's also true. Now the million $ question: Whom would you like our society to rely on to generate quality work? I don't think there is a satisfactory answer to this question. Public research becomes more and more of an industry every year with the publish-or-perish game, while a solely private solution is obviously open to very biased conclusions. There is no smart solution to a stupid problem. But the truth is that _as an institution_ the NHS is the only actor whose mission is to optimize towards public health.
    - SpicyLemonZest 6y ago
      In most fields, our society relies on private industry to generate quality work, even when the work is very important and doing it wrong might kill people. I'm not an anarchist, I do recognize there are reasons that the government should provide some things. But the idea that private industry uniformly produces bad results because they don't care about anything but profit just seems silly to me. Producing good results is profitable!
      - mturmon 6y ago
        > In most fields, our society relies on private industry to generate quality work ... This is just not true. Military, police, courts? What does "most fields" even mean?
        SpicyLemonZest 6y ago
        My job, food, apartment, utilities, entertainment, all come from private companies.
  - klyrs 6y ago
    > it seems only fair to say that academics optimize for prestige and publication in good journals. On one hand you have lots of people arguing that the legal duty of a company is only to make money for its shareholders. When large companies fail at that goal, it's bailout time. On the other hand you have peer-reviewed journals where authors are incentivized to find accurate results, and researchers will cite articles on the basis of their veracity (or, if incorrect, as punching bags). Of course that's a fallible process and just as vulnerable to cronyism, but when researchers are caught cooking the books they're discredited, not rewarded.
- toolz 6y ago
  Everyone working for a living optimizes their work towards making more money. That doesn't change for people funded with public money. It's pretty widely accepted that the people in charge of public funding (politicians) sometimes act outside of the publics interest for self-gain. I make no claim as to which achieves better results for the public because it's such a complicated problem, but I think it's rather naive to just assume publicly funded incentives are more aligned with social health than private incentives.
lern_too_spel 6y ago
That was a shitty code review. Seeding issues like the ones cited don't affect the results of a Monte Carlo simulation, and there are tests in the repo, just not automated ones. The section you quoted shows the reason for the review's sloppiness. The reviewer set out to find a way to justify their own beliefs instead of to actually read the code.
- Joeboy 6y ago
  > That was a shitty code review Damn straight. The only sense in which it was not a shitty code review, is that it didn't actually review any code. Looking into the linked tickets is a time consuming faff, but people should actually do it before taking the blog post at face value.
- klenwell 6y ago
  I agree. Moreover, the cited bug ("predictions varied by around 80,000 deaths after 80 days") doesn't really seem to impact the over all policy implications: no lockdown means exceptional number (400 to 480k) of deaths. Unless the entire simulation is bogus, it comes off as nitpicking.
- thu2111 6y ago
  Seriously? Doing a Monte Carlo simulation means you adjust the seeds to get different runs. It doesn't mean your program can read uninitialised memory or reuse variables that weren't reset to zero and still be correct. Where are people getting this idea that you can just average away the results of out-of-bounds reads and race conditions?
  - lern_too_spel 6y ago
    What does reading uninitialized memory or reusing variables that weren't set to zero have to do with seeding issues? Read my comment again and reply to its content instead of making up a comment that you would like to reply to.
Nursie 6y ago
Yeah that's a weird set of conclusions there. I would think the papers based on it want another look though.
pjc50 6y ago
That's the most biased opinion imaginable by an astroturf group. There's no information on who they are, but the article currently on their front page is by Toby Young, who is a Spectator/Quillete guy and to be found backing almost all stupid ideas within British politics. https://www.spectator.co.uk/article/This-lockdown-may-kill-me https://www.spectator.co.uk/article/This-lockdown-may-kill-m...
spamizbad 6y ago
That last part is mindblowing given the fact that the health insurance industry in the US is trying to argue it shouldn’t have to pay for COVID-19 treatments because it’s part of a pandemic and not part of normal medical treatments. I guess if your model is: if pandemic and COVID: is_covered = False You can have a very clean pandemic model.

lbeltrame 6y ago

Back in 2008-2009 my then PI tried this: it worked for a while, but then it proved to be untenable. Not because there wasn't enough money (there was) but because the university did not like the idea to get someone on board just to help researchers develop software, and thus there were so many roadblocks at one point that it was impossible to go on.

btrettel 6y ago
I'm interested in hearing more details on this, if you're able to discuss them.
- lbeltrame 6y ago
  I don't honestly know how much I can disclose (FTR, I was not the one doing that job - I was working as a postdoc there at the time), but I'll try to summarize it briefly: - PI got a couple of EU funded grants - PI wanted to build a program / programs out of some ideas he had before moving to the institution he was currently employed at - PI hired a software developer (actually a software engineer) to do this job - Statistician drafted algorithms, developer created the software - I used the software for my Ph.D. thesis, got hired at the lab of the PI as a postdoc, new requirements arose due to the way I used it - Developer made a preliminary version of a new version of the software following discussions with me - University made it very hard to keep developer on the team, due to bureaucracy and kind of hostility towards this kind of employment - At some point (I can't recall the details exactly, but it was something that spanned almost one year), the form of keeping the developer on board was no longer possible - PI offers an alternative contract, but it is financially wasteful to the developer (not the fault of the PI, but the way certain things work in my country) - Developer leaves the project Also the university, to my knowledge, complained that the developer cost a lot (IIRC, the project was paid at market rate, so in line with other, non academic software projects). I can't comment on the quality of the software we used (sadly it was never open sourced) as it was in Java and I only have a passing understanding of the language, but the approach of having a dedicated developer IMO worked (and also net quite a number of publications over that period).

malandrew 6y ago

> experts Being an expert means you know your tools well. If you can't code well and coding is a critical part of your toolset, then you're not an expert yet. You don't get a free pass because you allegedly are strong in other parts of your craft, especially if the part of your craft you're weak in can be this problematic.

rch 6y ago

For this to happen there would need to be a nationally accepted PE certification for 'software engineers' on par with other engineering disciplines. It's unreasonable to expect non-experts, and experts from other fields, to perform credentialing on a case by case basis.

vonmoltke 6y ago
There is no "nationally accepted" PE license for any discipline. It's a state-level credential and reciprocity is not universal or automatic.
- rch 6y ago
  I included the "nationally accepted" bit since I'd prefer to see some incremental improvement over current models, but I don't have confidence that an international standard would be accepted in all relevant contexts. It could still be administered at the state level. I'm also ignoring the requirement that candidates must start with a degree from an ABET-accredited institution, which seems to feature prominently in proposals from IEEE and others. Ideally I think there should be some way around that, but alternatives I'm familiar with aren't great either (e.g. FINRA).