3 ms·

Their idea reminded of a similar concept I heard once: open source scientific papers. It works like this: you put up the .tex source, the data files, the R, Oc

by lkozma 17y ago

Their idea reminded of a similar concept I heard once: open source scientific papers. It works like this: you put up the .tex source, the data files, the R, Octave, etc. scripts, and if people download and run the scripts, they get the exact paper as a result, including all the graphs. This way one can check that the graphs really show what is claimed in the paper and that there has been no bug or "polishing" of the results.

TrevorBurnham 17y ago
That would be absolutely awesome. Of course, in principle, there's nothing preventing researchers from doing that now. I think that being able to actually manipulate things on the same website that the data exists on will be key to bringing "open source papers" into the academic mainstream.
- lkozma 17y ago
  Yeah, I like the outline of your product idea. Looking forward to trying it out.
marciovm123 17y ago
Perhaps this could work well with results based on publicly available data sets (like the Protein Data Bank at www.rcsb.org), but I'd be surprised if this isn't done already to a significant degree. Labs post programs (or their biological equivalents, protocols) on their websites all the time. Don't forget that data can be polished as well - for a hypothetical cheater, you'd simply be moving the "messaging" work from the finished project to the source files you mention above. There's a lot more to trusting a paper then just trusting the figures; you're trusting the scientific process that a lab is undertaking, and it's almost always impractical to require proof of every stage of that process. Another downside is that not every scientist is ok with giving access to all of their hard-earned data - that opens up analyses and potential scoops to other groups. This is especially true in biomedical/clinical studies, where data is very expensive to gather and can take years to analyze. Not saying raw data is never shared, but there is a balance there. Finally, scientists greatly prefers testing a published claim by pursuing a follow-up hypothesis rather then redoing an existing analyses. That way you kill two birds with one stone. It's inevitable that some irreproducible results slip in with this system, but if the claims are significant enough sooner or later someone who's trying to build on the claims (and failing) will go looking for more details and the fraud is exposed. The fear of being exposed in that way is what keeps the system honest, and to a large extent this system works.

by lkozma 17y ago

TrevorBurnham 17y ago

That would be absolutely awesome. Of course, in principle, there's nothing preventing researchers from doing that now. I think that being able to actually manipulate things on the same website that the data exists on will be key to bringing "open source papers" into the academic mainstream.

lkozma 17y ago
Yeah, I like the outline of your product idea. Looking forward to trying it out.

marciovm123 17y ago

Perhaps this could work well with results based on publicly available data sets (like the Protein Data Bank at www.rcsb.org), but I'd be surprised if this isn't done already to a significant degree. Labs post programs (or their biological equivalents, protocols) on their websites all the time. Don't forget that data can be polished as well - for a hypothetical cheater, you'd simply be moving the "messaging" work from the finished project to the source files you mention above. There's a lot more to trusting a paper then just trusting the figures; you're trusting the scientific process that a lab is undertaking, and it's almost always impractical to require proof of every stage of that process. Another downside is that not every scientist is ok with giving access to all of their hard-earned data - that opens up analyses and potential scoops to other groups. This is especially true in biomedical/clinical studies, where data is very expensive to gather and can take years to analyze. Not saying raw data is never shared, but there is a balance there. Finally, scientists greatly prefers testing a published claim by pursuing a follow-up hypothesis rather then redoing an existing analyses. That way you kill two birds with one stone. It's inevitable that some irreproducible results slip in with this system, but if the claims are significant enough sooner or later someone who's trying to build on the claims (and failing) will go looking for more details and the fraud is exposed. The fear of being exposed in that way is what keeps the system honest, and to a large extent this system works.