4 ms·

Yes! This is the point that most people miss. No scientist treats published studies as gospel. Our focus shouldn't be on exact replication, and should be on how

by skosuri 10y ago

Yes! This is the point that most people miss. No scientist treats published studies as gospel. Our focus shouldn't be on exact replication, and should be on how generalizable such results are. If the results don't hold in slightly altered systems, it falls into the wastebin of ideas.

sndean 10y ago
> Our focus shouldn't be on exact replication, and should be on how generalizable such results are. This is how I normally look at things. If I can't easily replicate an experiment, then it's very likely to be wrong. Sadly, it's pretty rare (and exciting) when you can easily do something based on the methods in someone's paper
- skosuri 10y ago
  Well that shouldn't necessarily be the standard. There are lots of people I know that can't do a miniprep or run a gel. It doesn't mean that those things don't work or aren't replicable.
skosuri 10y ago
Also, you should read the Peter Walter piece that I think is the best discussion of the topic: https://www.ascb.org/newsletter/2016-marchapril-newsletter/on-reproducibility-and-clocks/ https://www.ascb.org/newsletter/2016-marchapril-newsletter/o...
return0 10y ago
> No scientist treats published studies as gospel. Unfortunately that is a problem. Imagine yourself trying to create a simulation of a biological system, which has to rely on experiments. You may come up with a plan, but every little line on the plan will be either very doubtful or outright false. The problem is that many of these doubts could be dispelled if the experiment was a lot more stringent (much larger sample size, much more controlled conditions etc). That would cost a hell lot more, but it would give you one answer you can rely on.
- skosuri 10y ago
  I don't think making experiments more stringent is the answer here (btw, I've spent a lot of time trying to build simulations of biological systems). Usually, we aren't doing the right experiment in the first place; this is hard to figure out ahead of time. Again, read the ACSB that I linked to below... it's probably the most nuanced and interesting discussion on the subject
  - return0 10y ago
    Yes, thanks that was a good read. It seems we need a "Map" of biological sciences, in which every study could be a 'pin' in a particular location , signifying that 'this study studies this very particular problem here'. Maybe that would help figure out where are the biggest gaps. Unfortunately, most studies broaden the impact of their results too much to the point that reading the paper abstract can be misleading. Maybe people should just publish their results, but not be allowed to make any claims about them; let others and the community make those claims.
lr4444lr 10y ago
Maybe not, but these studies get swept up into meta-studies, books, think tanks and special interest groups write policy papers based on that secondary layer, which then become position papers that inform policy makers about Big Decisions. There's a lot at stake when only scientists are aware of the fallacy.
- archgoon 10y ago
  The reason why meta studies get done is precisely because individual studies aren't reliable.
Svezik 10y ago
> No scientist treats published studies as gospel. Tell that to the climatologists burning their peers at the stake for heresy.
CodeWriter23 10y ago
> No scientist treats published studies as gospel. Why then do so many scientists in so many different fields insist "the science is settled"?
- j7ake 10y ago
  Which scientists? About what issues?

by skosuri 10y ago

sndean 10y ago

> Our focus shouldn't be on exact replication, and should be on how generalizable such results are. This is how I normally look at things. If I can't easily replicate an experiment, then it's very likely to be wrong. Sadly, it's pretty rare (and exciting) when you can easily do something based on the methods in someone's paper

skosuri 10y ago
Well that shouldn't necessarily be the standard. There are lots of people I know that can't do a miniprep or run a gel. It doesn't mean that those things don't work or aren't replicable.

skosuri 10y ago

Also, you should read the Peter Walter piece that I think is the best discussion of the topic: https://www.ascb.org/newsletter/2016-marchapril-newsletter/on-reproducibility-and-clocks/ https://www.ascb.org/newsletter/2016-marchapril-newsletter/o...

return0 10y ago

> No scientist treats published studies as gospel. Unfortunately that is a problem. Imagine yourself trying to create a simulation of a biological system, which has to rely on experiments. You may come up with a plan, but every little line on the plan will be either very doubtful or outright false. The problem is that many of these doubts could be dispelled if the experiment was a lot more stringent (much larger sample size, much more controlled conditions etc). That would cost a hell lot more, but it would give you one answer you can rely on.

skosuri 10y ago
I don't think making experiments more stringent is the answer here (btw, I've spent a lot of time trying to build simulations of biological systems). Usually, we aren't doing the right experiment in the first place; this is hard to figure out ahead of time. Again, read the ACSB that I linked to below... it's probably the most nuanced and interesting discussion on the subject
- return0 10y ago
  Yes, thanks that was a good read. It seems we need a "Map" of biological sciences, in which every study could be a 'pin' in a particular location , signifying that 'this study studies this very particular problem here'. Maybe that would help figure out where are the biggest gaps. Unfortunately, most studies broaden the impact of their results too much to the point that reading the paper abstract can be misleading. Maybe people should just publish their results, but not be allowed to make any claims about them; let others and the community make those claims.

lr4444lr 10y ago

Maybe not, but these studies get swept up into meta-studies, books, think tanks and special interest groups write policy papers based on that secondary layer, which then become position papers that inform policy makers about Big Decisions. There's a lot at stake when only scientists are aware of the fallacy.

archgoon 10y ago
The reason why meta studies get done is precisely because individual studies aren't reliable.

Svezik 10y ago

> No scientist treats published studies as gospel. Tell that to the climatologists burning their peers at the stake for heresy.

CodeWriter23 10y ago

> No scientist treats published studies as gospel. Why then do so many scientists in so many different fields insist "the science is settled"?

j7ake 10y ago
Which scientists? About what issues?