5 ms·

ReScience is a peer-reviewed journal that targets computational research

50 points by phreeza 11y ago

kjjw 11y ago
I'm not sure replicatable or reproducible are better or worse than each other. They are just different things. Replicatable seems to hint at an easy way to run quite literally the exact same experiment or process as someone else has done in the past, while reproducible suggests there is enough information made available for someone else to re-create from the ground up a previous experiment or process. It seems to me both have merits and one cannot assume that performing a 'replication' is a superior form of 'reproduction', nor the other way around. I suspect the merits and relative desirability of one or the other vary depending on discipline.
- phreeza 11y ago
  The way I understood it is that the authors treat replicable or reproducible as the same thing, the tag line just means that having actually replicated it is better than the possibility of replication existing. What is the difference between replication and reproduction?
  - jamessb 11y ago
    What is the difference between replication and reproduction? When people draw a distinction, it is usually that replication refers to doing exactly the same thing, and getting the same results, whereas reproduction refers to doing a different thing, but reaching the same conclusion. There is interesting discussion of different forms of verification n "Replication, Reproduction and Re-analysis: Three ways for verifying experimental findings": http://sequoia.cs.byu.edu/lab/files/reser2010/proceedings/Gomez%20-%20Replication%20Reproduction.pdf http://sequoia.cs.byu.edu/lab/files/reser2010/proceedings/Go...
- Nicolas_Rougier 11y ago
  ReScience makes a distinction between reproducible and replicatable. For ReScience, we explicitly ask for a new implementation from scratch. This means the model is reproducible because it has been replicated (if replicated).
x4m 11y ago
Will Scopus or WOS index it?
- fluidcruft 11y ago
  ...and of course the only question that actually matters has been downvoted (I'll add pubmed to the question).
- phreeza 11y ago
  I don't see why they shouldn't.
Fede_V 11y ago
The main advantage of replicable science is that it forces authors to be very explicit about their datasets and all the steps that make up their data analysis pipeline. A lot of computational analysis (at least in biology) is done by students with little background in CS and with no knowledge of version control or best practices. This leads to lots of results which are nearly impossible to reproduce because so many steps have just not been documented. Being able to show a detailed reproduction, with high quality code that shows all the steps in detail is immensely valuable, and is also a great pedagogic tool for teaching grad students. I hope it takes off and becomes widely used. Fingers crossed! Disclaimer: After Nicolas EuroSciPy talk I offered to help, and joined as a reviewer, so I'm obviously very biased.
streptomycin 11y ago
Like many other people, I have wasted a lot of time trying to reimplement models in the literature. Sometimes it succeeds, sometimes it fails, but the time it takes is always significant. I'm thinking of 3 examples off the top of my head... 1. An ODE model with ~10 equations. No code. I reimplemented it and exactly reproduced the plots from the paper. I assume this would be good for ReScience. I'll try to clean it up and submit it this weekend. 2. Another ODE model with ~10 equations. No code. I reimplemented it as best I could, but it doesn't quite reproduce the figures in the paper. Typo in my code? Typo in the paper? No way to know. Is there any place for something like that? 3. A stochastic model with ~4 equations. No code. I reimplemented it as best I could, but because it's stochastic, it's hard to say if it's exactly what the authors originally made. Now here's the interesting part... if I run the model 10 times, naturally I get 10 different outputs. Maybe 1 or 2 will be nice and pretty like the plots in the paper. The other ones look kind of shitty. Is that evidence that the authors ran their model a bunch of times and picked only the nicest looking results for the paper? Or is it evidence that my implementation isn't quite right? I don't know. Is there anything I can do with that code?
- mbrundle 11y ago
  I've also had issues replicating ODE models from papers in the past. I assume you've tried contacting the authors to ask for clarification and haven't had a reply? If so, I think you should submit all 3, because without any original code available, your effort is likely to be representative of anyone else's. (And it's easily vettable as it'll be on GitHub). The onus is then on the original author to point out how (if at all) your implementation differs from theirs.
- Fede_V 11y ago
  We would definitely be very interested. If the code is in Python, I can even sign up to review it.
  - streptomycin 11y ago
    Here's another problem: the code is in Matlab. What do you guys think about publishing reproductions in proprietary languages?
justifier 11y ago
this looks great this is a sort of research version of an effort i am making re: journalism.. shamefully, i call it 'death to the editorial' the nytimes has the slogan 'all the news that's fit to print' it frustrates me that this mentality has, for the most part, bled over into digital media as well where the potential strings of 0s and 1s seems permissibly infinite i think journalism should also embrace the notion: an article about computational result is advertising, not scholarship. The actual scholarship is the full software environment, code and data that produced the result state a fact? let me see the source reference a quote? let me see the source posit an opinion? let me see the method of discovery i want more information stead less if the journalist, or in this case the researcher, is unable or unwilling then i suppose crowdsourcing is a fine alternative
dalke 11y ago
> Buckheit & Donoho (1995) explained almost 20 years ago that, an article about computational result is advertising, not scholarship. It's been years since I read a PNAS paper but I still remember the disclaimer for each paper that since the author contributed money towards the publication, it must legally be described as "advertisement." Thus, advertising and scholarship are not incompatible. The original quote is slightly different: "An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instruction which generate the figures." My two, relatively minor but still important, qualms are: 1) the important thing is to be able to verify a claim, not reproduce the method to get the claim. In most cases those are the same thing. But consider the RSA Factoring Challenge. It's easy to verify that RSA640 was correctly factored, even without access to the underlying hardware or algorithms. At that point you know that only a fraction of the world's compute power is needed to factor that number. I've seen this in my own field of computational chemistry where one paper proposed a way to categorize molecules and a later paper published counter-examples which showed failure cases. The details of how to find the failure case aren't always important - they might even be done with only brain power. Thus there can be good scholarship without knowing the complete compute environment of the paper. 2) Not everyone has images which are machine reproducible from the raw data. For example, https://en.wikipedia.org/wiki/Ribbon_diagram#History https://en.wikipedia.org/wiki/Ribbon_diagram#History describes how Richardson converted 3D protein structures to hand-drawn diagrams: > These drawings were made in pen on tracing paper over a printout of a Cα trace of the atomic coordinates; they preserved positions, smoothed the backbone path, and incorporated small local shifts to disambiguate the visual appearance. While there are many computer programs which can do most of those steps, the final presentation may still be tweaked manually when disambiguation is more important than absolute fidelity. (The coordinates are usually available, so it's still possible to get the fidelity.)

50 points by phreeza 11y ago

kjjw 11y ago

I'm not sure replicatable or reproducible are better or worse than each other. They are just different things. Replicatable seems to hint at an easy way to run quite literally the exact same experiment or process as someone else has done in the past, while reproducible suggests there is enough information made available for someone else to re-create from the ground up a previous experiment or process. It seems to me both have merits and one cannot assume that performing a 'replication' is a superior form of 'reproduction', nor the other way around. I suspect the merits and relative desirability of one or the other vary depending on discipline.

phreeza 11y ago
The way I understood it is that the authors treat replicable or reproducible as the same thing, the tag line just means that having actually replicated it is better than the possibility of replication existing. What is the difference between replication and reproduction?
- jamessb 11y ago
  What is the difference between replication and reproduction? When people draw a distinction, it is usually that replication refers to doing exactly the same thing, and getting the same results, whereas reproduction refers to doing a different thing, but reaching the same conclusion. There is interesting discussion of different forms of verification n "Replication, Reproduction and Re-analysis: Three ways for verifying experimental findings": http://sequoia.cs.byu.edu/lab/files/reser2010/proceedings/Gomez%20-%20Replication%20Reproduction.pdf http://sequoia.cs.byu.edu/lab/files/reser2010/proceedings/Go...
Nicolas_Rougier 11y ago
ReScience makes a distinction between reproducible and replicatable. For ReScience, we explicitly ask for a new implementation from scratch. This means the model is reproducible because it has been replicated (if replicated).

x4m 11y ago

Will Scopus or WOS index it?

fluidcruft 11y ago
...and of course the only question that actually matters has been downvoted (I'll add pubmed to the question).
phreeza 11y ago
I don't see why they shouldn't.

Fede_V 11y ago

The main advantage of replicable science is that it forces authors to be very explicit about their datasets and all the steps that make up their data analysis pipeline. A lot of computational analysis (at least in biology) is done by students with little background in CS and with no knowledge of version control or best practices. This leads to lots of results which are nearly impossible to reproduce because so many steps have just not been documented. Being able to show a detailed reproduction, with high quality code that shows all the steps in detail is immensely valuable, and is also a great pedagogic tool for teaching grad students. I hope it takes off and becomes widely used. Fingers crossed! Disclaimer: After Nicolas EuroSciPy talk I offered to help, and joined as a reviewer, so I'm obviously very biased.

streptomycin 11y ago

Like many other people, I have wasted a lot of time trying to reimplement models in the literature. Sometimes it succeeds, sometimes it fails, but the time it takes is always significant. I'm thinking of 3 examples off the top of my head... 1. An ODE model with ~10 equations. No code. I reimplemented it and exactly reproduced the plots from the paper. I assume this would be good for ReScience. I'll try to clean it up and submit it this weekend. 2. Another ODE model with ~10 equations. No code. I reimplemented it as best I could, but it doesn't quite reproduce the figures in the paper. Typo in my code? Typo in the paper? No way to know. Is there any place for something like that? 3. A stochastic model with ~4 equations. No code. I reimplemented it as best I could, but because it's stochastic, it's hard to say if it's exactly what the authors originally made. Now here's the interesting part... if I run the model 10 times, naturally I get 10 different outputs. Maybe 1 or 2 will be nice and pretty like the plots in the paper. The other ones look kind of shitty. Is that evidence that the authors ran their model a bunch of times and picked only the nicest looking results for the paper? Or is it evidence that my implementation isn't quite right? I don't know. Is there anything I can do with that code?

mbrundle 11y ago
I've also had issues replicating ODE models from papers in the past. I assume you've tried contacting the authors to ask for clarification and haven't had a reply? If so, I think you should submit all 3, because without any original code available, your effort is likely to be representative of anyone else's. (And it's easily vettable as it'll be on GitHub). The onus is then on the original author to point out how (if at all) your implementation differs from theirs.
Fede_V 11y ago
We would definitely be very interested. If the code is in Python, I can even sign up to review it.
- streptomycin 11y ago
  Here's another problem: the code is in Matlab. What do you guys think about publishing reproductions in proprietary languages?

justifier 11y ago

this looks great this is a sort of research version of an effort i am making re: journalism.. shamefully, i call it 'death to the editorial' the nytimes has the slogan 'all the news that's fit to print' it frustrates me that this mentality has, for the most part, bled over into digital media as well where the potential strings of 0s and 1s seems permissibly infinite i think journalism should also embrace the notion: an article about computational result is advertising, not scholarship. The actual scholarship is the full software environment, code and data that produced the result state a fact? let me see the source reference a quote? let me see the source posit an opinion? let me see the method of discovery i want more information stead less if the journalist, or in this case the researcher, is unable or unwilling then i suppose crowdsourcing is a fine alternative

dalke 11y ago

> Buckheit & Donoho (1995) explained almost 20 years ago that, an article about computational result is advertising, not scholarship. It's been years since I read a PNAS paper but I still remember the disclaimer for each paper that since the author contributed money towards the publication, it must legally be described as "advertisement." Thus, advertising and scholarship are not incompatible. The original quote is slightly different: "An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instruction which generate the figures." My two, relatively minor but still important, qualms are: 1) the important thing is to be able to verify a claim, not reproduce the method to get the claim. In most cases those are the same thing. But consider the RSA Factoring Challenge. It's easy to verify that RSA640 was correctly factored, even without access to the underlying hardware or algorithms. At that point you know that only a fraction of the world's compute power is needed to factor that number. I've seen this in my own field of computational chemistry where one paper proposed a way to categorize molecules and a later paper published counter-examples which showed failure cases. The details of how to find the failure case aren't always important - they might even be done with only brain power. Thus there can be good scholarship without knowing the complete compute environment of the paper. 2) Not everyone has images which are machine reproducible from the raw data. For example, https://en.wikipedia.org/wiki/Ribbon_diagram#History https://en.wikipedia.org/wiki/Ribbon_diagram#History describes how Richardson converted 3D protein structures to hand-drawn diagrams: > These drawings were made in pen on tracing paper over a printout of a Cα trace of the atomic coordinates; they preserved positions, smoothed the backbone path, and incorporated small local shifts to disambiguate the visual appearance. While there are many computer programs which can do most of those steps, the final presentation may still be tweaked manually when disambiguation is more important than absolute fidelity. (The coordinates are usually available, so it's still possible to get the fidelity.)