4 ms·

Maybe conferences should have a "reproducibility" track for that purpose? Also, I don't know about other fields, but I'm pretty sure that in CS, if you just to

Maybe conferences should have a "reproducibility" track for that purpose? Also, I don't know about other fields, but I'm pretty sure that in CS, if you just took a paper and tried to reproduce the results, you'll get rejected on the ground that you offer no original contribution; no original contribution => no publication => no funding.

nxzero 10y ago
Real issue is that the way science is funded is broken.
- varjag 10y ago
  Broken assumes it was working well at some point. Nothing new here.
  - mathgeek 10y ago
    It worked pretty well back when scientific inquiry was funded by selling inventions or other work that the scientist did "as a day job." That was a long time ago, though.
    - varjag 10y ago
      This has worked and still works for technology, but is not how basic science research was done. There's never been "enough" money for sciences, nor they ever been allocated in a manner agreeable to all parties. For a long time, research was either a gentleman's hobby or a side project under auspices of unsuspecting donor. Lagrange has developed variational calculus while doing his artillery school tenure. Einstein's annus mirabilis happened at his stint in Swiss patent office.
studentrob 10y ago
For CS, reproducing should be easy given the code and input data right? Other sciences' input isn't so easily shared
- timwaagh 10y ago
  most cs papers i read (actually all) do -not- include code (or input data). which makes this an issue even in the case of cs.
- mannykannot 10y ago
  Rerunning code does not count as reproduction as it is not an independent test of what is being claimed. You do have the option of verifying the code against the goals of the study, though that is a review rather than a reproduction (which is still valuable.)
- maze-le 10y ago
  Not necessarily. There are fields like complexity and computability wich are pureley mathematical in nature and have little to do with actual code, but much with the abstract theoretical fundamentals of code in general. And on the other hand you have fields like machine learning, where you can have an algorithm, wich you can implement (or have as code), but you don't nessessarily know the values of certain parameters, specific for your problem space.
- dagw 10y ago
  Ideally you should re-implement the algorithm based on the description in the paper to verify that the description of the algorithm is correct. You should also test with your own data to make sure that the algorithm works on all reasonable data and not only on some provided cherry picked data. If you can't get the expected results with your own implementation and your own data then the results aren't reproduced.
  - studentrob 10y ago
    Great points, thanks!
  - eru 10y ago
    Yes. So being able to rerun with the same code and same inputs to get the same outputs is a lower bar. Many papers don't meet even that bar. (Mostly because they don't publish code nor data; and academic code is often a horrible mess, and the code was mucked around with between different stages of running.)
crispyambulance 10y ago
I don't think reproducibility should necessarily mean literally repeating the same study over again. Whether or not a paper is fully correct is less important that the further work it stimulates or informs-- either via more papers (impact factor) or via APPLICATIONS of the research. In the example you're giving, to get accepted, the author doesn't need to do much more than extend the research in some small way, compare it to other research, or explore some aspect more fully. If someone is going to take the time to repeat something, they might as well go a little further. If someone takes the time to apply research results they're going to, in a way, test the validity of the results. Maybe the author's experiences in the pharma world is very different, but I doubt that.
- jerf 10y ago
  "I don't think reproducibility should necessarily mean literally repeating the same study over again." Yes, it should. We have too many demonstrated instances of studies built on other studies that turned out to be flawed, but the second study, rather than showing the first one was flawed, was rationalized and massaged until it conformed to the first study because the first study already had the imprimatur of peer-reviewed correctness on it. Only someone replicating the original study directly (give or take sample size or duration or other simple such changes) will have the guts and moral authority to stand up and say "I can't replicate this. The original study may be wrong."
  - bbctol 10y ago
    But there should also be replication of the concepts of the study under a different paradigm, otherwise you might just be finding out effects of the particular procedure. An original study is rarely "wrong" (fraud is a serious problem, but not a typical one), it's that it's merely one data point, from one procedure, and a lot of the time the effect that can be replicated is "real," it's just a side effect of the process. Basically, there's way more replication that should be going on in science at many levels.
    - jerf 10y ago
      Yes. But replication is a starting point. It isn't just about fraud, it's also about error and failure to communicate the experiment. (It's also beyond time for paper journals to just contain the relevant summaries and conclusions for given studies and for "what fits in a journal" to cease being "what fits in publication". Authors ought to be able to publish as much stuff as they want with a paper, including full source code sets or data sets or whatever, with the paper limitations affecting only the top-level summary.)

nxzero 10y ago

Real issue is that the way science is funded is broken.

varjag 10y ago
Broken assumes it was working well at some point. Nothing new here.
- mathgeek 10y ago
  It worked pretty well back when scientific inquiry was funded by selling inventions or other work that the scientist did "as a day job." That was a long time ago, though.
  - varjag 10y ago
    This has worked and still works for technology, but is not how basic science research was done. There's never been "enough" money for sciences, nor they ever been allocated in a manner agreeable to all parties. For a long time, research was either a gentleman's hobby or a side project under auspices of unsuspecting donor. Lagrange has developed variational calculus while doing his artillery school tenure. Einstein's annus mirabilis happened at his stint in Swiss patent office.

studentrob 10y ago

For CS, reproducing should be easy given the code and input data right? Other sciences' input isn't so easily shared

timwaagh 10y ago
most cs papers i read (actually all) do -not- include code (or input data). which makes this an issue even in the case of cs.
mannykannot 10y ago
Rerunning code does not count as reproduction as it is not an independent test of what is being claimed. You do have the option of verifying the code against the goals of the study, though that is a review rather than a reproduction (which is still valuable.)
maze-le 10y ago
Not necessarily. There are fields like complexity and computability wich are pureley mathematical in nature and have little to do with actual code, but much with the abstract theoretical fundamentals of code in general. And on the other hand you have fields like machine learning, where you can have an algorithm, wich you can implement (or have as code), but you don't nessessarily know the values of certain parameters, specific for your problem space.
dagw 10y ago
Ideally you should re-implement the algorithm based on the description in the paper to verify that the description of the algorithm is correct. You should also test with your own data to make sure that the algorithm works on all reasonable data and not only on some provided cherry picked data. If you can't get the expected results with your own implementation and your own data then the results aren't reproduced.
- studentrob 10y ago
  Great points, thanks!
- eru 10y ago
  Yes. So being able to rerun with the same code and same inputs to get the same outputs is a lower bar. Many papers don't meet even that bar. (Mostly because they don't publish code nor data; and academic code is often a horrible mess, and the code was mucked around with between different stages of running.)

crispyambulance 10y ago

I don't think reproducibility should necessarily mean literally repeating the same study over again. Whether or not a paper is fully correct is less important that the further work it stimulates or informs-- either via more papers (impact factor) or via APPLICATIONS of the research. In the example you're giving, to get accepted, the author doesn't need to do much more than extend the research in some small way, compare it to other research, or explore some aspect more fully. If someone is going to take the time to repeat something, they might as well go a little further. If someone takes the time to apply research results they're going to, in a way, test the validity of the results. Maybe the author's experiences in the pharma world is very different, but I doubt that.

jerf 10y ago
"I don't think reproducibility should necessarily mean literally repeating the same study over again." Yes, it should. We have too many demonstrated instances of studies built on other studies that turned out to be flawed, but the second study, rather than showing the first one was flawed, was rationalized and massaged until it conformed to the first study because the first study already had the imprimatur of peer-reviewed correctness on it. Only someone replicating the original study directly (give or take sample size or duration or other simple such changes) will have the guts and moral authority to stand up and say "I can't replicate this. The original study may be wrong."
- bbctol 10y ago
  But there should also be replication of the concepts of the study under a different paradigm, otherwise you might just be finding out effects of the particular procedure. An original study is rarely "wrong" (fraud is a serious problem, but not a typical one), it's that it's merely one data point, from one procedure, and a lot of the time the effect that can be replicated is "real," it's just a side effect of the process. Basically, there's way more replication that should be going on in science at many levels.
  - jerf 10y ago
    Yes. But replication is a starting point. It isn't just about fraud, it's also about error and failure to communicate the experiment. (It's also beyond time for paper journals to just contain the relevant summaries and conclusions for given studies and for "what fits in a journal" to cease being "what fits in publication". Authors ought to be able to publish as much stuff as they want with a paper, including full source code sets or data sets or whatever, with the paper limitations affecting only the top-level summary.)