6 ms·

We don't need to 'define repetition', we need to foster a culture that a) accepts repetition and b) does not accept something for a fact just because it's in a

by return0 10y ago

We don't need to 'define repetition', we need to foster a culture that a) accepts repetition and b) does not accept something for a fact just because it's in a journal. Right now, (a) is not even acceptable; nobody will publish a replication study. Ofc, LHC is impossible to replicate, but the vast majority of life science studies are. I should think this is mostly needed in life sciences. Other, more 'exact' sciences seem to not have this problem.

throwaway729 10y ago
> we need to foster a culture that a) accepts repetition Do we? I don't think we do. I think we need to foster a culture of honesty and rigor. Of good science. Which is decidedly different from fostering a culture of "repetition" for its own sake. Paying for the cost of mountains upon mountains of lab techs and materials that it would require to replicate every study published in a major journal just isn't a good use of ever-dwindling science dollars. Replicate where it's not far off the critical path. Replicate where the study is going to have a profound effect on the direction of research in several labs. But don't just replicate because "science!" In fact, one could argue that the increased strain on funding sources introduced by the huge cost of reproducing a bunch of stuff would increase the cut-throat culture of science and thereby decrease the scientist's natural proclivity toward honesty. > and b) does not accept something for a fact just because it's in a journal Again, it's entirely unclear what you mean here. It's impossible to re-verify every single paper you read (I've read three since breakfast). That would be like re-writing every single line of code of every dependency you pull into a project. And I'm pretty sure literally no scientist takes a paper's own description of its results at face value without reading through methods and looking at (at least) a summary of the data. Taking papers at face value is really only a problem in science reporting and at (very) sub-par institutions/venues. I don't care about the latter, and neither should you. WRT the former, science reporters often grossly misunderstand the paper anyways. All the good reproducible science in the world is of zero help if science reporters are going to bastardize the results beyond recognition anyways...
- flubert 10y ago
  >ever-dwindling science dollars Do we have a good source of information on that? https://en.wikipedia.org/wiki/File:U.S._research_funding.png https://en.wikipedia.org/wiki/File:U.S._research_funding.png http://www.aaas.org/sites/default/files/DefNon%3B.jpg http://www.aaas.org/sites/default/files/DefNon%3B.jpg http://www.aaas.org/sites/default/files/Agencies%3B.jpg http://www.aaas.org/sites/default/files/Agencies%3B.jpg
  - throwaway729 10y ago
    I know I made the original "dwindling funding" claim, but it's actually a red herring. I should have said something like "increasingly competitive nature of grants". The argument relevant to this article is NOT about whether there's enough funding for science. Rather, the argument is about how difficult it is to get funding. I think it's fairly uncontroversial (and correct) to say that getting good science funded has become considerably harder over the years. I'm not sure anyone has done the work of trying to quantify that; maybe someone else can help find data for that. But to address your question anyways: 1. USFG scientific funding instutitions are only one source of science funding. There are many others. If you look across the federal government, there's a downward trend: http://www.aaas.org/sites/default/files/DefNon%3B.jpg http://www.aaas.org/sites/default/files/DefNon%3B.jpg One must also take into account non-federal-governemnt sources, which in many cases have substantially decreased their investments in R&D since 2008. 2. As pcnt of GDP there's been a steady decline: http://www.aaas.org/sites/default/files/RDGDP%3B.jpg http://www.aaas.org/sites/default/files/RDGDP%3B.jpg 3. From an impact-on-culture perspective -- which is the relevant one in my comment -- I think (2) is more intresting than your data and also more interesting than (1). The question should be "how difficult is it to fund good science", not "how much are spending in absolute or relative terms". This is, of course, very difficult to quantify. But looking at percentage of GDP is at least better than looking at absolute dollars.
- JustUhThought 10y ago
  Trust but confirm. The entire point of science, and this is not a hyperbole, is that results are reproducible. If the experiment is not reproducible one must take the results on faith. There is no such thing as faith based science. In order to build a shared body of knowledge based on scientific facts, then, results must be repeated. It is how different people can talk about the same thing, without fearing an asymmetry of knowledge and understanding about the axioms on which their discussion of the world rests. Otherwise it is faith or religion or narrative, something other than than science.
wrsh07 10y ago
I think there's also a certain lack of transparency in the scientific process that needs to be fixed. But that's just like my opinion.
refurb 10y ago
and b) does not accept something for a fact just because it's in a journal. I can't say I know any scientists who think any differently.
mcarlise 10y ago
>Other, more 'exact' sciences seem to not have this problem I'm not sure of this. By 'exact' disciplines I'm assuming you mean disciplines of science more dependent on mathematical proofs. A CACM paper a little bit ago discusses this, and found a large number of papers where not repeatable. If I recall, it mostly focused on that nobody shares code and/or the code didn't build. http://cacm.acm.org/magazines/2016/3/198873-repeatability-in-computer-systems-research/fulltext http://cacm.acm.org/magazines/2016/3/198873-repeatability-in...
bmmayer1 10y ago
The key here is we need a culture of skepticism. That's the fundamental core philosophical trait of the enlightenment: not belief in what anyone says (just 'cause they say it or have authority to say it) but doubt in claims until they are rationally proven, and proven again, and proven again. Skepticism builds on itself because a culture where everyone is skeptical is a culture where the burden of proof lies with the claimant. This applies to science as well: whether an experiment is repeated once or hundreds of times, the real measure is to what degree it convinces the winnowing number of skeptics aligned against the idea.
Tloewald 10y ago
Does doing 50 studies and only publishing the 3 that produce favorable results count as repetition? Because, BTW, that's exactly what Big Pharma does, and also intrinsically what the bias towards publishing only significant results does.
- unfamiliar 10y ago
  > Because, BTW, that's exactly what Big Pharma does That's exactly what almost everyone in academia does. Money is not the only corrupting factor.
- Retric 10y ago
  Pharma needs to send information about studies beforehand and regulators pay attention to this for new drugs which counters the effect your talking about. Further, they are happy to report inconclusive results the FDA mostly cares about harm to people not effectiveness. Now, after drugs are on the market then there is another wave of less reputable research. But, the FDA has already approved the drug so they don't care as much.

by return0 10y ago

throwaway729 10y ago

> we need to foster a culture that a) accepts repetition Do we? I don't think we do. I think we need to foster a culture of honesty and rigor. Of good science. Which is decidedly different from fostering a culture of "repetition" for its own sake. Paying for the cost of mountains upon mountains of lab techs and materials that it would require to replicate every study published in a major journal just isn't a good use of ever-dwindling science dollars. Replicate where it's not far off the critical path. Replicate where the study is going to have a profound effect on the direction of research in several labs. But don't just replicate because "science!" In fact, one could argue that the increased strain on funding sources introduced by the huge cost of reproducing a bunch of stuff would increase the cut-throat culture of science and thereby decrease the scientist's natural proclivity toward honesty. > and b) does not accept something for a fact just because it's in a journal Again, it's entirely unclear what you mean here. It's impossible to re-verify every single paper you read (I've read three since breakfast). That would be like re-writing every single line of code of every dependency you pull into a project. And I'm pretty sure literally no scientist takes a paper's own description of its results at face value without reading through methods and looking at (at least) a summary of the data. Taking papers at face value is really only a problem in science reporting and at (very) sub-par institutions/venues. I don't care about the latter, and neither should you. WRT the former, science reporters often grossly misunderstand the paper anyways. All the good reproducible science in the world is of zero help if science reporters are going to bastardize the results beyond recognition anyways...

flubert 10y ago
>ever-dwindling science dollars Do we have a good source of information on that? https://en.wikipedia.org/wiki/File:U.S._research_funding.png https://en.wikipedia.org/wiki/File:U.S._research_funding.png http://www.aaas.org/sites/default/files/DefNon%3B.jpg http://www.aaas.org/sites/default/files/DefNon%3B.jpg http://www.aaas.org/sites/default/files/Agencies%3B.jpg http://www.aaas.org/sites/default/files/Agencies%3B.jpg
- throwaway729 10y ago
  I know I made the original "dwindling funding" claim, but it's actually a red herring. I should have said something like "increasingly competitive nature of grants". The argument relevant to this article is NOT about whether there's enough funding for science. Rather, the argument is about how difficult it is to get funding. I think it's fairly uncontroversial (and correct) to say that getting good science funded has become considerably harder over the years. I'm not sure anyone has done the work of trying to quantify that; maybe someone else can help find data for that. But to address your question anyways: 1. USFG scientific funding instutitions are only one source of science funding. There are many others. If you look across the federal government, there's a downward trend: http://www.aaas.org/sites/default/files/DefNon%3B.jpg http://www.aaas.org/sites/default/files/DefNon%3B.jpg One must also take into account non-federal-governemnt sources, which in many cases have substantially decreased their investments in R&D since 2008. 2. As pcnt of GDP there's been a steady decline: http://www.aaas.org/sites/default/files/RDGDP%3B.jpg http://www.aaas.org/sites/default/files/RDGDP%3B.jpg 3. From an impact-on-culture perspective -- which is the relevant one in my comment -- I think (2) is more intresting than your data and also more interesting than (1). The question should be "how difficult is it to fund good science", not "how much are spending in absolute or relative terms". This is, of course, very difficult to quantify. But looking at percentage of GDP is at least better than looking at absolute dollars.
JustUhThought 10y ago
Trust but confirm. The entire point of science, and this is not a hyperbole, is that results are reproducible. If the experiment is not reproducible one must take the results on faith. There is no such thing as faith based science. In order to build a shared body of knowledge based on scientific facts, then, results must be repeated. It is how different people can talk about the same thing, without fearing an asymmetry of knowledge and understanding about the axioms on which their discussion of the world rests. Otherwise it is faith or religion or narrative, something other than than science.

wrsh07 10y ago

I think there's also a certain lack of transparency in the scientific process that needs to be fixed. But that's just like my opinion.

refurb 10y ago

and b) does not accept something for a fact just because it's in a journal. I can't say I know any scientists who think any differently.

mcarlise 10y ago

>Other, more 'exact' sciences seem to not have this problem I'm not sure of this. By 'exact' disciplines I'm assuming you mean disciplines of science more dependent on mathematical proofs. A CACM paper a little bit ago discusses this, and found a large number of papers where not repeatable. If I recall, it mostly focused on that nobody shares code and/or the code didn't build. http://cacm.acm.org/magazines/2016/3/198873-repeatability-in-computer-systems-research/fulltext http://cacm.acm.org/magazines/2016/3/198873-repeatability-in...

bmmayer1 10y ago

The key here is we need a culture of skepticism. That's the fundamental core philosophical trait of the enlightenment: not belief in what anyone says (just 'cause they say it or have authority to say it) but doubt in claims until they are rationally proven, and proven again, and proven again. Skepticism builds on itself because a culture where everyone is skeptical is a culture where the burden of proof lies with the claimant. This applies to science as well: whether an experiment is repeated once or hundreds of times, the real measure is to what degree it convinces the winnowing number of skeptics aligned against the idea.

Tloewald 10y ago

Does doing 50 studies and only publishing the 3 that produce favorable results count as repetition? Because, BTW, that's exactly what Big Pharma does, and also intrinsically what the bias towards publishing only significant results does.

unfamiliar 10y ago
> Because, BTW, that's exactly what Big Pharma does That's exactly what almost everyone in academia does. Money is not the only corrupting factor.
Retric 10y ago
Pharma needs to send information about studies beforehand and regulators pay attention to this for new drugs which counters the effect your talking about. Further, they are happy to report inconclusive results the FDA mostly cares about harm to people not effectiveness. Now, after drugs are on the market then there is another wave of less reputable research. But, the FDA has already approved the drug so they don't care as much.