4 ms·

If anyone is in any doubt about this, I would like to point them towards this github issue: https://github.com/mrc-ide/covid-sim/issues/165 https://github.com/m

by ljm 6y ago

If anyone is in any doubt about this, I would like to point them towards this github issue: https://github.com/mrc-ide/covid-sim/issues/165 https://github.com/mrc-ide/covid-sim/issues/165

guygurari 6y ago
I don’t doubt it, but covid-sim is not the best example of this effect. In this case, people are worried that poorly written simulation code is informing policy decisions that affect billions of people. They are not demanding that it should be fixed, they are demanding that it should not be relied on in its current state.
- ohgodplsno 6y ago
  Dunning-Kruger powered software developers with beliefs that they are in any way qualified to say that mathematical simulations checked by dozens of actual scientists with PhDs are not valid ask that results are ignored because there's no unit tests.
  - 1000units 6y ago
    It's funny that none of these people has produced a failing test.
- wizzwizz4 6y ago
  They're worried that there isn't testing code to prove that it's correct. If it's proven correct in other ways, it doesn't need unit tests. Yeah, the code's bad – so what? It wasn't written by programmers. Most simulation code is bad, but if it's been proven correct it doesn't need to be good. From what I can tell, this is a matter of conflicting conventions in different fields meeting head-on.
  - guygurari 6y ago
    To be clear, I am not arguing for or against the validity of the claims regarding covid-sim. I am only saying that this is not a good example of the effect where people feel entitled to get open source code that fits their needs.
maest 6y ago
Some of the commenters in that issue have a very regressive and myopic stance. Raising such a public ruckus will heavily disincentivise scientists from publishing their code in the future. Decisions will still be made, but in the dark. There's a certain amount of self-stroking going on there with people feeling better by putting down code produced by non-professional programmers. A much better response would be: 1. Scrutinise end users of the code: demand that governments only use this data if proper due dil has been carried out. 2. Submit code improvements. 3. (maybe) Demand that peer reviewers are more rigorous when checking the way results are generated. The issue with this is that this would also be a strong disincentive for scientist to publish their work.
- dirtydroog 6y ago
  Well, there was this situation a few years ago https://www.theverge.com/2013/4/17/4234136/excel-calculation-error-infamous-economic-study https://www.theverge.com/2013/4/17/4234136/excel-calculation... While the tone of the covid model scrutiny leaves a lot to be desired it is understandable seeing as how the results are affecting the entire world. The situation is unprecedented.
  - maest 6y ago
    I don't think anyone is saying the code can't be better or even that the results might be wrong (I don't know as I haven't spent much time going through the codebase). My meta point is that people are: 1. pressuring and blaming the wrong party. 2. doings something which will have an unintended strongly negative consequence (make scientists averse to publishing their code).
noelsusman 6y ago
I don't even know where to begin with this, it's so insane. It's extra funny because John Carmack was the one who did a lot of the refactoring of this code. It's also not remotely true that billions of lives have been disrupted on the basis of this code being correct. This is exactly why scientists don't release their code.
- tsbertalan 6y ago
  Another reason scientists often don't release code is that code is not considered the valuable artifact produced by research--in more methods-focused fields in particular, it's instead the mathematical model, as embodied in some latex in the published paper. "Reproducing" the results then doesn't consist of trivially rerunning the same code, but in reimplementing it, possibly in a different language. This could maybe be seen as valuable in that the reproduction will surface both implementation errors and logic/modeling errors (though, really, having the source facilitates both). In fact, more applications-focused researchers (those who combine real data with established models, for instance) tend to write higher-level scripts stringing more-engineered tools. In this case, open sourcing the scripts would be both easier and more pointless, since they will be almost the same as what is stated in English, tables, and plots in the paper. Epidemiology is usually this way, in my experience, though the linked repository seems to have some of both flavors. The underlying misunderstand in that issue thread seems to me to be a disagreement on what the main valuable byproducts of scientific programming are. Professionally programmers will naturally think it's the code, but, traditionally, the code has been seen in academia as only a temporary vehicle for the model and it's parameters, which is the real product. (Also, the "program" might really need to include the lab, its papers, and its institutional knowledge, which is harder to package and open-source.) Right or wrong, the assumption is that any competent grad student could reproduce the result in a week or so of (admittedly unnecessary) greenfield coding. But this is clearly not ideal, and newer work does strive for more professionalism in coding, open-source-by-default, and therefore faster replication. The project in question clearly predates this trend. (Of course, a third reason academics don't open source is that some secrecy is required before publication in competitive fields. On a months-long project, you might not want to be committing publicly before publication. But this isn't much of an excuse.)
ww520 6y ago
WOW. This is over the top toxic.
kirill5pol 6y ago
That was painful to read. :(
umvi 6y ago
The scientism is very strong in that thread. Isn't "faith in science" an oxymoron? If anything, people should default to skepticism for hastily thrown together scientific models.
UncleMeat 6y ago
Yup. This whole experience was immensely frustrating. First people complain that the code isn't open source and as soon as it is available suddenly there are piles of people swooping in to shit on everything. That's not going to make scientists write better code. It is going to make scientists refuse to open their code. The only thing more frustrating is when I see people swoop into open source code with "security vulns" that are based on nonsense threat models.

by ljm 6y ago

If anyone is in any doubt about this, I would like to point them towards this github issue: https://github.com/mrc-ide/covid-sim/issues/165 https://github.com/mrc-ide/covid-sim/issues/165

guygurari 6y ago

I don’t doubt it, but covid-sim is not the best example of this effect. In this case, people are worried that poorly written simulation code is informing policy decisions that affect billions of people. They are not demanding that it should be fixed, they are demanding that it should not be relied on in its current state.

ohgodplsno 6y ago
Dunning-Kruger powered software developers with beliefs that they are in any way qualified to say that mathematical simulations checked by dozens of actual scientists with PhDs are not valid ask that results are ignored because there's no unit tests.
- 1000units 6y ago
  It's funny that none of these people has produced a failing test.
wizzwizz4 6y ago
They're worried that there isn't testing code to prove that it's correct. If it's proven correct in other ways, it doesn't need unit tests. Yeah, the code's bad – so what? It wasn't written by programmers. Most simulation code is bad, but if it's been proven correct it doesn't need to be good. From what I can tell, this is a matter of conflicting conventions in different fields meeting head-on.
- guygurari 6y ago
  To be clear, I am not arguing for or against the validity of the claims regarding covid-sim. I am only saying that this is not a good example of the effect where people feel entitled to get open source code that fits their needs.

maest 6y ago

Some of the commenters in that issue have a very regressive and myopic stance. Raising such a public ruckus will heavily disincentivise scientists from publishing their code in the future. Decisions will still be made, but in the dark. There's a certain amount of self-stroking going on there with people feeling better by putting down code produced by non-professional programmers. A much better response would be: 1. Scrutinise end users of the code: demand that governments only use this data if proper due dil has been carried out. 2. Submit code improvements. 3. (maybe) Demand that peer reviewers are more rigorous when checking the way results are generated. The issue with this is that this would also be a strong disincentive for scientist to publish their work.

dirtydroog 6y ago
Well, there was this situation a few years ago https://www.theverge.com/2013/4/17/4234136/excel-calculation-error-infamous-economic-study https://www.theverge.com/2013/4/17/4234136/excel-calculation... While the tone of the covid model scrutiny leaves a lot to be desired it is understandable seeing as how the results are affecting the entire world. The situation is unprecedented.
- maest 6y ago
  I don't think anyone is saying the code can't be better or even that the results might be wrong (I don't know as I haven't spent much time going through the codebase). My meta point is that people are: 1. pressuring and blaming the wrong party. 2. doings something which will have an unintended strongly negative consequence (make scientists averse to publishing their code).

noelsusman 6y ago

I don't even know where to begin with this, it's so insane. It's extra funny because John Carmack was the one who did a lot of the refactoring of this code. It's also not remotely true that billions of lives have been disrupted on the basis of this code being correct. This is exactly why scientists don't release their code.

tsbertalan 6y ago
Another reason scientists often don't release code is that code is not considered the valuable artifact produced by research--in more methods-focused fields in particular, it's instead the mathematical model, as embodied in some latex in the published paper. "Reproducing" the results then doesn't consist of trivially rerunning the same code, but in reimplementing it, possibly in a different language. This could maybe be seen as valuable in that the reproduction will surface both implementation errors and logic/modeling errors (though, really, having the source facilitates both). In fact, more applications-focused researchers (those who combine real data with established models, for instance) tend to write higher-level scripts stringing more-engineered tools. In this case, open sourcing the scripts would be both easier and more pointless, since they will be almost the same as what is stated in English, tables, and plots in the paper. Epidemiology is usually this way, in my experience, though the linked repository seems to have some of both flavors. The underlying misunderstand in that issue thread seems to me to be a disagreement on what the main valuable byproducts of scientific programming are. Professionally programmers will naturally think it's the code, but, traditionally, the code has been seen in academia as only a temporary vehicle for the model and it's parameters, which is the real product. (Also, the "program" might really need to include the lab, its papers, and its institutional knowledge, which is harder to package and open-source.) Right or wrong, the assumption is that any competent grad student could reproduce the result in a week or so of (admittedly unnecessary) greenfield coding. But this is clearly not ideal, and newer work does strive for more professionalism in coding, open-source-by-default, and therefore faster replication. The project in question clearly predates this trend. (Of course, a third reason academics don't open source is that some secrecy is required before publication in competitive fields. On a months-long project, you might not want to be committing publicly before publication. But this isn't much of an excuse.)

ww520 6y ago

WOW. This is over the top toxic.

kirill5pol 6y ago

That was painful to read. :(

umvi 6y ago

The scientism is very strong in that thread. Isn't "faith in science" an oxymoron? If anything, people should default to skepticism for hastily thrown together scientific models.

UncleMeat 6y ago

Yup. This whole experience was immensely frustrating. First people complain that the code isn't open source and as soon as it is available suddenly there are piles of people swooping in to shit on everything. That's not going to make scientists write better code. It is going to make scientists refuse to open their code. The only thing more frustrating is when I see people swoop into open source code with "security vulns" that are based on nonsense threat models.