6 ms·

Biology postdoc here, seconded. But what really gets me is the disconnect between "most scientists agree there is a reproducibility crisis" and "most scientist

by cgiles 7y ago

Biology postdoc here, seconded. But what really gets me is the disconnect between "most scientists agree there is a reproducibility crisis" and "most scientists believe most of the papers they read are basically true and reproducible". This was mentioned in the survey and it conforms to my informal experience of attitudes. I do not know how you square that circle. Maybe, because of the pressures you mention, we are all supposed to engage in an informal convention of pretending to believe most previously published work is true, was done correctly, and is reproducible even if we know damn well how unlikely that is. I find it hard to do. One day the public is going to cotton on to all of this. I cringe every time I hear extremely authoritative lectures on "what scientists say" about highly politicized public policy matters. These are not my fields, but if they are as prone to error, bias, irreproducibility, etc, as my own, I'd exercise some humility. It is one thing for we scientists to lie -- errr, project unwarranted confidence -- to each other for the sake of getting grants, but it is quite another to do it directly to the public. But when the public does figure it out, what do you think will happen to funding? It will get tighter, and make the problem even worse. We need reform from within before the hammer falls, and quickly.

itcrowd 7y ago
I agree with the last two paragraphs of your post viz. humility and reform. > the disconnect between "most scientists agree there is a reproducibility crisis" and "most scientists believe most of the papers they read are basically true and reproducible" Could it be that many scientists think: "there is a crisis, but not in my field!"? To be honest, I personally think this quite often (and then realize I am being naive).
keldaris 7y ago
I don't know that we can reform - if you spend too much time trying, the system will automatically cull you. Personally, I'm lucky enough to be at least slightly insulated from the true scale of the problem by working in theoretical/computational soft matter physics, where the costs/grants/impact factors are comparatively on the small end of things. Medicine and biology seem to be affected the most, for good reason - this is where the messiest problems intersect with the most interest from society at large, so you get the most acutely misaligned incentives. That being said, communicating limits of applicability and degrees of certainty to a popular audience is hellishly difficult even if you're trying to be perfectly honest. Even in a hypothetical world where we've somehow fixed academia, this will always be a hard problem most scientists are ill equipped to tackle.
- staticautomatic 7y ago
  I bet a lot would change if the largest funding institutions publicly prioritized funding scientists who have a history of publishing reproducible work.
  - keldaris 7y ago
    Sure - for some definition of "reproducible". Any metric you come up with that's easily measurable will be gamed into oblivion. Anything meaningful - like looking at dedicated reproduction studies, etc. - requires a complete change in how academia functions, because right now publishing a paper that just aims to reproduce (or fail to) a previous work is effectively impossible in most fields. Not to mention that given the incentive structure you'd be pretty crazy to even try.
    - staticautomatic 7y ago
      It may well be some kind of law that every positive incentive creates at least one perverse incentive, but that doesn't mean positive incentives don't work or aren't worth trying.
  - Fomite 7y ago
    First they'd have to fund the reproducibility studies.
    - taneq 7y ago
      Well they shouldn’t have any trouble funding the first one...
      - Fomite 7y ago
        They have plenty of trouble funding much of anything.
  - lettergram 7y ago
    I tend to argue that 10%-20% of a grant should go to reproducibility. This does two things: 1. It would help fund / ensure funding to several labs in the space. 2. It would help ensure reproducibility of results Yes, it can be gamed, but generally it should be easier to reproduce results so 10%-20% of the funding to follow directions should be okay. Of course, this could lead to one group just constantly doctoring results to show something is not reproducible. In which case, a third lab would need to get some funding to check it out.
TeMPOraL 7y ago
> But what really gets me is the disconnect between "most scientists agree there is a reproducibility crisis" and "most scientists believe most of the papers they read are basically true and reproducible". It sounds very much like an academic equivalent of Gell-Mann Amnesia - the phenomenon in which you notice that every news article on a topic in your field is complete garbage, people you know in other professions report the same situation with articles on topics in their fields, and yet when you turn the page and see an article on something outside your field, you forget about the whole thing and treat it as a gospel.
- inimino 7y ago
  Isn't it the exact opposite? Presumably the papers scientists are reading are primarily the ones in their own field.
  - TeMPOraL 7y ago
    I'm arguing that the phenomena seem identical in their underlying structures. In both cases, we're dealing with a situation in which a person faces mountains of evidence that a source (a newspaper, or scientific papers in a given field) keeps pumping out inaccurate or wrong publications, but despite all that evidence, they assume that whatever isn't explicitly pointed out as wrong must be 100% true, accurate and honest.
    - inimino 7y ago
      Ah, yes. Wishful thinking.
  - Frost1x 7y ago
    That depends on how critical you are. Our current incentive structure is out of control across the board and everything is skewing towards maximizing ROI in about every aspect of life money is involved in. I apply this perspective to everything I see ("how are they getting money from this?") then work backwards and it seems to lead to accurate predictions, at least anecdotally. Perhaps I'm just jaded and cynical but it works well, unfortunately.
mehrdadn 7y ago
> "most scientists agree there is a reproducibility crisis" and "most scientists believe most of the papers they read are basically true and reproducible". [...] I do not know how you square that circle. I don't understand why these seem contradictory? To me the obvious (maybe naive?) reading is that scientist believe reproducibility is essential to the advancement of their fields, but also believe that being unable to reproduce a result can often be explained by their lack of sufficient information (or resources) to reproduce said results, rather than always assuming the other person must be lying (or just pretty darn lucky). I'm guessing this must've happened because they have actually come across such situations and observed many results that are indeed correct, but just difficult to reproduce. Why can't this explanation square that circle?
- cgiles 7y ago
  > but also believe that being unable to reproduce a result can often be explained by their lack of sufficient information (or resources) to reproduce said results, rather than always assuming the other person must be lying (or just pretty darn lucky). Most certainly, when you go to replicate something, and fail, your first assumption is that you did something wrong. Even the second or third time. But after enough tries, you start to look for another explanation. "Irreproducibility", as Nature and the common scientist would use the term, doesn't mean "I tried once and failed", it means "I tried everything I could possibly think of and it still doesn't work". Also, for many types of experiment it is questionable whether they are reproducible even in principle. One of the common analyses I do is called RNA-seq. It quantifies, for each of ~25K genes, whether and how much it went up or down with a given perturbation. So the result of such an experiment is a 25K long list with numbers attached to each gene. What would it mean to replicate such a result? Surely the list of significant genes and pathways will not be identical even if a robot were to perform exactly the same experiment, due to biological variability. Some people find this property of nonfalsifiability to be convenient, however...
  - sjy 7y ago
    > "Irreproducibility", as Nature and the common scientist would use the term, doesn't mean "I tried once and failed", it means "I tried everything I could possibly think of and it still doesn't work". I’m not a scientist, but it is surprising to me that the term would not be regarded as at least ambiguous. The article quotes a microbiologist who says “there is no consensus on what reproducibility is or should be.”
irq11 7y ago
”One day the public is going to cotton on to all of this. I cringe every time I hear extremely authoritative lectures on "what scientists say" about highly politicized public policy matters....I'd exercise some humility.” You shouldn’t be cringing. You should be educating people about how science actually works, and how it simply doesn’t matter very much whether any particular paper is reproducible. It’s a straw-man argument, because most papers aren’t worth reproducing. I don’t know many good scientists who take what they read (in any journal) at face value. If you do, you’ve been mislead somewhere during your training (fwiw, I also have a PhD). At best, papers are sources of ideas. Interesting ideas get tested again. Most get ignored. Even if a few hokum theories become popular for a while, eventually they’re revealed for what they are. The tiny percentage of subjects that rise to the level of public policy discussion end up being so extensively investigated that reproduction of results is essentially guaranteed. And yeah, you hear lots of silly noise from university PR departments, but that stuff is a flash in the pan. For example, nobody legitimate is in doubt of the broader facts of global climate change or evolution or vaccination, even if 95% of (say) social science reaults turn out to be complete bunk. Yet climate deniers, anti-vaxxers and “intelligent design” trolls absolutely love it when this distinction is ignored, because it allows them to confuse the public on the legitimacy of science as a process.
- inimino 7y ago
  It's true that science doesn't value every published paper equally, but it's also true that publish or perish is creating ever-growing mountains of worthless papers. This is a real problem, and drags the quality of everything down. Besides, the fact that there isn't one reputable journal in most fields that remains untarnished by the replication crisis is both a practical problem and a problem of public trust. A lot of this BS science is paid for directly by the public's tax dollars, or else by their student loans. I wouldn't expect the public to be so forgiving if 95% of it is bunk.
  - irq11 7y ago
    “it's also true that publish or perish is creating ever-growing mountains of worthless papers.” Is this true? Prove your claim. ”This is a real problem, and drags the quality of everything down.“ Let’s assume your first assertion is true. Is it automatically true that your second claim follows? Why? I see no evidence that the individual productivity of scientists has changed much in the last 30 years, nor do I notice much of a change in the aggregate quality of science. Crappy science existed hundreds of years ago, and it continues to exist today. The main difference, as far as I can tell, is that we have a lot more scientists now. In any case, these are just assertions, not arguments.
sjy 7y ago
I don’t think the public is going to cotton on. The incentives that created the reproducibility crisis (the application of demand-driven managerialism to education and research generally) have also compromised the public’s ability to observe and care about the problem. Changes to the funding model for basic research will continue to be driven by bureaucrats, not politicians responding to public pressure.
myire 7y ago
Full Disclaimer: I am CEO and founder of a software company that works on reproducibility. The disconnect between "most scientists agree there is a reproducibility crisis" and "most scientists believe most of the papers they read are basically true and reproducible" has extended to every domain in my experience. With the way the funding model currently works, my hypothesis is that doing more, completely reproducible studies will drown out the non-reproducible studies (and funding model) with reproducible studies (funders ultimately vote with their checkbooks). That seems like a second logical disconnect and completely impossible if the cost model remained the same. However, the question is do studies operate in a way that is maximally efficient for the scientists or for the economy of science publication and knowledge dissemination. We, at MyIRE, believe that technology can tip the scales in the favor of scientists. Read more about our platform and business plan if you're interested: https://docs.myire.com https://docs.myire.com
- sombremesa 7y ago
  NB: your website looks like "My Ire" is that intentional? https://www.merriam-webster.com/dictionary/ire https://www.merriam-webster.com/dictionary/ire
- electric_sheep 7y ago
  I'll also plug https://www.protocols.io/ https://www.protocols.io/ and https://codeocean.com/ https://codeocean.com/ here --- might be interesting platforms for folks concerned about reproducibility to have on their radar.
michaelt 7y ago
> But what really gets me is the disconnect between "most scientists agree there is a reproducibility crisis" and "most scientists believe most of the papers they read are basically true and reproducible". [...] I do not know how you square that circle. If I was an academic, I imagine I would think: 1. Well, most of the papers I read are widely cited, reputable papers in widely read, reputable journals. A paper with 50 citations couldn't possibly be unreproducable! 2. Everyone knows there are scam journals and conferences that will accept everything. I'm sure it'd be easy to get a bad work accepted there, but I don't read anything like that. 3. And anyway, aren't the problems mostly in other fields? Everyone knows there are problems in comparative international underwater sports broadcasting studies, not serious subjects like mine.
Balgair 7y ago
> One day the public is going to cotton on to all of this. I mean, the flat-earth, anti-vax, climate-denier folks are well and cottoned. In the US, one of the major political parties what controls a fair amount of federal and state legislature seats is pretty sure 'scientists' are just liars and should be treated as such. These chickens aren't going to come home to roost, they are roosting and their chicks are hatching already.
- SlowRobotAhead 7y ago
  In a thread where the topic is “lots of science might be wrong” your take away is “doesn’t matter, everything is republican’s fault” and not “we should slow on using complex non-reproducible science to drive public policy because we know lots can’t be confirmed and there is no great way to tell what is definite“ ... well I suppose you demonstrated why it’s a “crisis”.

by cgiles 7y ago

itcrowd 7y ago

I agree with the last two paragraphs of your post viz. humility and reform. > the disconnect between "most scientists agree there is a reproducibility crisis" and "most scientists believe most of the papers they read are basically true and reproducible" Could it be that many scientists think: "there is a crisis, but not in my field!"? To be honest, I personally think this quite often (and then realize I am being naive).

keldaris 7y ago

I don't know that we can reform - if you spend too much time trying, the system will automatically cull you. Personally, I'm lucky enough to be at least slightly insulated from the true scale of the problem by working in theoretical/computational soft matter physics, where the costs/grants/impact factors are comparatively on the small end of things. Medicine and biology seem to be affected the most, for good reason - this is where the messiest problems intersect with the most interest from society at large, so you get the most acutely misaligned incentives. That being said, communicating limits of applicability and degrees of certainty to a popular audience is hellishly difficult even if you're trying to be perfectly honest. Even in a hypothetical world where we've somehow fixed academia, this will always be a hard problem most scientists are ill equipped to tackle.

staticautomatic 7y ago
I bet a lot would change if the largest funding institutions publicly prioritized funding scientists who have a history of publishing reproducible work.
- keldaris 7y ago
  Sure - for some definition of "reproducible". Any metric you come up with that's easily measurable will be gamed into oblivion. Anything meaningful - like looking at dedicated reproduction studies, etc. - requires a complete change in how academia functions, because right now publishing a paper that just aims to reproduce (or fail to) a previous work is effectively impossible in most fields. Not to mention that given the incentive structure you'd be pretty crazy to even try.
  - staticautomatic 7y ago
    It may well be some kind of law that every positive incentive creates at least one perverse incentive, but that doesn't mean positive incentives don't work or aren't worth trying.
- Fomite 7y ago
  First they'd have to fund the reproducibility studies.
  - taneq 7y ago
    Well they shouldn’t have any trouble funding the first one...
    - Fomite 7y ago
      They have plenty of trouble funding much of anything.
- lettergram 7y ago
  I tend to argue that 10%-20% of a grant should go to reproducibility. This does two things: 1. It would help fund / ensure funding to several labs in the space. 2. It would help ensure reproducibility of results Yes, it can be gamed, but generally it should be easier to reproduce results so 10%-20% of the funding to follow directions should be okay. Of course, this could lead to one group just constantly doctoring results to show something is not reproducible. In which case, a third lab would need to get some funding to check it out.

TeMPOraL 7y ago

> But what really gets me is the disconnect between "most scientists agree there is a reproducibility crisis" and "most scientists believe most of the papers they read are basically true and reproducible". It sounds very much like an academic equivalent of Gell-Mann Amnesia - the phenomenon in which you notice that every news article on a topic in your field is complete garbage, people you know in other professions report the same situation with articles on topics in their fields, and yet when you turn the page and see an article on something outside your field, you forget about the whole thing and treat it as a gospel.

inimino 7y ago
Isn't it the exact opposite? Presumably the papers scientists are reading are primarily the ones in their own field.
- TeMPOraL 7y ago
  I'm arguing that the phenomena seem identical in their underlying structures. In both cases, we're dealing with a situation in which a person faces mountains of evidence that a source (a newspaper, or scientific papers in a given field) keeps pumping out inaccurate or wrong publications, but despite all that evidence, they assume that whatever isn't explicitly pointed out as wrong must be 100% true, accurate and honest.
  - inimino 7y ago
    Ah, yes. Wishful thinking.
- Frost1x 7y ago
  That depends on how critical you are. Our current incentive structure is out of control across the board and everything is skewing towards maximizing ROI in about every aspect of life money is involved in. I apply this perspective to everything I see ("how are they getting money from this?") then work backwards and it seems to lead to accurate predictions, at least anecdotally. Perhaps I'm just jaded and cynical but it works well, unfortunately.

mehrdadn 7y ago

> "most scientists agree there is a reproducibility crisis" and "most scientists believe most of the papers they read are basically true and reproducible". [...] I do not know how you square that circle. I don't understand why these seem contradictory? To me the obvious (maybe naive?) reading is that scientist believe reproducibility is essential to the advancement of their fields, but also believe that being unable to reproduce a result can often be explained by their lack of sufficient information (or resources) to reproduce said results, rather than always assuming the other person must be lying (or just pretty darn lucky). I'm guessing this must've happened because they have actually come across such situations and observed many results that are indeed correct, but just difficult to reproduce. Why can't this explanation square that circle?

cgiles 7y ago
> but also believe that being unable to reproduce a result can often be explained by their lack of sufficient information (or resources) to reproduce said results, rather than always assuming the other person must be lying (or just pretty darn lucky). Most certainly, when you go to replicate something, and fail, your first assumption is that you did something wrong. Even the second or third time. But after enough tries, you start to look for another explanation. "Irreproducibility", as Nature and the common scientist would use the term, doesn't mean "I tried once and failed", it means "I tried everything I could possibly think of and it still doesn't work". Also, for many types of experiment it is questionable whether they are reproducible even in principle. One of the common analyses I do is called RNA-seq. It quantifies, for each of ~25K genes, whether and how much it went up or down with a given perturbation. So the result of such an experiment is a 25K long list with numbers attached to each gene. What would it mean to replicate such a result? Surely the list of significant genes and pathways will not be identical even if a robot were to perform exactly the same experiment, due to biological variability. Some people find this property of nonfalsifiability to be convenient, however...
- sjy 7y ago
  > "Irreproducibility", as Nature and the common scientist would use the term, doesn't mean "I tried once and failed", it means "I tried everything I could possibly think of and it still doesn't work". I’m not a scientist, but it is surprising to me that the term would not be regarded as at least ambiguous. The article quotes a microbiologist who says “there is no consensus on what reproducibility is or should be.”

irq11 7y ago

”One day the public is going to cotton on to all of this. I cringe every time I hear extremely authoritative lectures on "what scientists say" about highly politicized public policy matters....I'd exercise some humility.” You shouldn’t be cringing. You should be educating people about how science actually works, and how it simply doesn’t matter very much whether any particular paper is reproducible. It’s a straw-man argument, because most papers aren’t worth reproducing. I don’t know many good scientists who take what they read (in any journal) at face value. If you do, you’ve been mislead somewhere during your training (fwiw, I also have a PhD). At best, papers are sources of ideas. Interesting ideas get tested again. Most get ignored. Even if a few hokum theories become popular for a while, eventually they’re revealed for what they are. The tiny percentage of subjects that rise to the level of public policy discussion end up being so extensively investigated that reproduction of results is essentially guaranteed. And yeah, you hear lots of silly noise from university PR departments, but that stuff is a flash in the pan. For example, nobody legitimate is in doubt of the broader facts of global climate change or evolution or vaccination, even if 95% of (say) social science reaults turn out to be complete bunk. Yet climate deniers, anti-vaxxers and “intelligent design” trolls absolutely love it when this distinction is ignored, because it allows them to confuse the public on the legitimacy of science as a process.

inimino 7y ago
It's true that science doesn't value every published paper equally, but it's also true that publish or perish is creating ever-growing mountains of worthless papers. This is a real problem, and drags the quality of everything down. Besides, the fact that there isn't one reputable journal in most fields that remains untarnished by the replication crisis is both a practical problem and a problem of public trust. A lot of this BS science is paid for directly by the public's tax dollars, or else by their student loans. I wouldn't expect the public to be so forgiving if 95% of it is bunk.
- irq11 7y ago
  “it's also true that publish or perish is creating ever-growing mountains of worthless papers.” Is this true? Prove your claim. ”This is a real problem, and drags the quality of everything down.“ Let’s assume your first assertion is true. Is it automatically true that your second claim follows? Why? I see no evidence that the individual productivity of scientists has changed much in the last 30 years, nor do I notice much of a change in the aggregate quality of science. Crappy science existed hundreds of years ago, and it continues to exist today. The main difference, as far as I can tell, is that we have a lot more scientists now. In any case, these are just assertions, not arguments.

sjy 7y ago

I don’t think the public is going to cotton on. The incentives that created the reproducibility crisis (the application of demand-driven managerialism to education and research generally) have also compromised the public’s ability to observe and care about the problem. Changes to the funding model for basic research will continue to be driven by bureaucrats, not politicians responding to public pressure.

myire 7y ago

Full Disclaimer: I am CEO and founder of a software company that works on reproducibility. The disconnect between "most scientists agree there is a reproducibility crisis" and "most scientists believe most of the papers they read are basically true and reproducible" has extended to every domain in my experience. With the way the funding model currently works, my hypothesis is that doing more, completely reproducible studies will drown out the non-reproducible studies (and funding model) with reproducible studies (funders ultimately vote with their checkbooks). That seems like a second logical disconnect and completely impossible if the cost model remained the same. However, the question is do studies operate in a way that is maximally efficient for the scientists or for the economy of science publication and knowledge dissemination. We, at MyIRE, believe that technology can tip the scales in the favor of scientists. Read more about our platform and business plan if you're interested: https://docs.myire.com https://docs.myire.com

sombremesa 7y ago
NB: your website looks like "My Ire" is that intentional? https://www.merriam-webster.com/dictionary/ire https://www.merriam-webster.com/dictionary/ire
electric_sheep 7y ago
I'll also plug https://www.protocols.io/ https://www.protocols.io/ and https://codeocean.com/ https://codeocean.com/ here --- might be interesting platforms for folks concerned about reproducibility to have on their radar.

michaelt 7y ago

> But what really gets me is the disconnect between "most scientists agree there is a reproducibility crisis" and "most scientists believe most of the papers they read are basically true and reproducible". [...] I do not know how you square that circle. If I was an academic, I imagine I would think: 1. Well, most of the papers I read are widely cited, reputable papers in widely read, reputable journals. A paper with 50 citations couldn't possibly be unreproducable! 2. Everyone knows there are scam journals and conferences that will accept everything. I'm sure it'd be easy to get a bad work accepted there, but I don't read anything like that. 3. And anyway, aren't the problems mostly in other fields? Everyone knows there are problems in comparative international underwater sports broadcasting studies, not serious subjects like mine.

Balgair 7y ago

> One day the public is going to cotton on to all of this. I mean, the flat-earth, anti-vax, climate-denier folks are well and cottoned. In the US, one of the major political parties what controls a fair amount of federal and state legislature seats is pretty sure 'scientists' are just liars and should be treated as such. These chickens aren't going to come home to roost, they are roosting and their chicks are hatching already.

SlowRobotAhead 7y ago
In a thread where the topic is “lots of science might be wrong” your take away is “doesn’t matter, everything is republican’s fault” and not “we should slow on using complex non-reproducible science to drive public policy because we know lots can’t be confirmed and there is no great way to tell what is definite“ ... well I suppose you demonstrated why it’s a “crisis”.