4 ms·

Like many other people, I have wasted a lot of time trying to reimplement models in the literature. Sometimes it succeeds, sometimes it fails, but the time it t

Like many other people, I have wasted a lot of time trying to reimplement models in the literature. Sometimes it succeeds, sometimes it fails, but the time it takes is always significant. I'm thinking of 3 examples off the top of my head... 1. An ODE model with ~10 equations. No code. I reimplemented it and exactly reproduced the plots from the paper. I assume this would be good for ReScience. I'll try to clean it up and submit it this weekend. 2. Another ODE model with ~10 equations. No code. I reimplemented it as best I could, but it doesn't quite reproduce the figures in the paper. Typo in my code? Typo in the paper? No way to know. Is there any place for something like that? 3. A stochastic model with ~4 equations. No code. I reimplemented it as best I could, but because it's stochastic, it's hard to say if it's exactly what the authors originally made. Now here's the interesting part... if I run the model 10 times, naturally I get 10 different outputs. Maybe 1 or 2 will be nice and pretty like the plots in the paper. The other ones look kind of shitty. Is that evidence that the authors ran their model a bunch of times and picked only the nicest looking results for the paper? Or is it evidence that my implementation isn't quite right? I don't know. Is there anything I can do with that code?

mbrundle 11y ago
I've also had issues replicating ODE models from papers in the past. I assume you've tried contacting the authors to ask for clarification and haven't had a reply? If so, I think you should submit all 3, because without any original code available, your effort is likely to be representative of anyone else's. (And it's easily vettable as it'll be on GitHub). The onus is then on the original author to point out how (if at all) your implementation differs from theirs.
Fede_V 11y ago
We would definitely be very interested. If the code is in Python, I can even sign up to review it.
- streptomycin 11y ago
  Here's another problem: the code is in Matlab. What do you guys think about publishing reproductions in proprietary languages?
  - ogrisel 11y ago
    ReScience explicitly does not accept submissions that are not written in an open source programming language. It you can make your code work with Octave I assume it's fine. https://www.gnu.org/software/octave/ https://www.gnu.org/software/octave/ Although I am sure you will find more reviewers ready to review code written with Python / NumPy / SciPy.
    - streptomycin 11y ago
      Guess I'll try to port. IMHO Matlab is still the best thing I've seen for numerical analysis of differential equations.
      - plus 11y ago
        While it's a bit of a pain to install, Assimulo provides very nice Python interface to the robust Sundials library of ODE/DAE solvers. I'm currently using it in a project that requires the solution of DAEs and it behaves much more nicely than any other Python ODE/DAE solver I've come across. Edit: Here's a link: http://www.jmodelica.org/assimulo_home/index.html http://www.jmodelica.org/assimulo_home/index.html
        Fede_V 11y ago
        I benchmarked Assimulo vs LSODA (the function wrapped by SciPy odeint) and LSODA was significantly faster for bigger problems. Assimulo has a lot more features though, like adjoint sensitivity, etc...
seiji 11y ago
If you go deep enough, sometimes you can find the models on one author's university website. Then you'll scream. Often you'll find the source program, but then discover it's a 12,000 line matlab file with no comments, no spaces between lines, and lots of copy/paste. Step one of reproducible results: stop using matlab. Step two: give every author a "code reviewer" or "code tidier" to fix their rush-rush-rush-hurry-theres-a-deadline code into something others can at least evaluate against.
Nicolas_Rougier 11y ago
I think we would happily review the three submissions. Case 1. ok Case 2. this would need to extra precaution (maybe 3 reviewers) + public challenge to spot any error. Then ask directly the original author. If nothing works, we can declare the paper non reproducible. But it might be heavy responsibility for the you. Case 3. also ok and you can specify in the accompanying article there are several cases where things does not run as expected.

mbrundle 11y ago

I've also had issues replicating ODE models from papers in the past. I assume you've tried contacting the authors to ask for clarification and haven't had a reply? If so, I think you should submit all 3, because without any original code available, your effort is likely to be representative of anyone else's. (And it's easily vettable as it'll be on GitHub). The onus is then on the original author to point out how (if at all) your implementation differs from theirs.

Fede_V 11y ago

We would definitely be very interested. If the code is in Python, I can even sign up to review it.

streptomycin 11y ago
Here's another problem: the code is in Matlab. What do you guys think about publishing reproductions in proprietary languages?
- ogrisel 11y ago
  ReScience explicitly does not accept submissions that are not written in an open source programming language. It you can make your code work with Octave I assume it's fine. https://www.gnu.org/software/octave/ https://www.gnu.org/software/octave/ Although I am sure you will find more reviewers ready to review code written with Python / NumPy / SciPy.
  - streptomycin 11y ago
    Guess I'll try to port. IMHO Matlab is still the best thing I've seen for numerical analysis of differential equations.
    - plus 11y ago
      While it's a bit of a pain to install, Assimulo provides very nice Python interface to the robust Sundials library of ODE/DAE solvers. I'm currently using it in a project that requires the solution of DAEs and it behaves much more nicely than any other Python ODE/DAE solver I've come across. Edit: Here's a link: http://www.jmodelica.org/assimulo_home/index.html http://www.jmodelica.org/assimulo_home/index.html
      - Fede_V 11y ago
        I benchmarked Assimulo vs LSODA (the function wrapped by SciPy odeint) and LSODA was significantly faster for bigger problems. Assimulo has a lot more features though, like adjoint sensitivity, etc...

seiji 11y ago

If you go deep enough, sometimes you can find the models on one author's university website. Then you'll scream. Often you'll find the source program, but then discover it's a 12,000 line matlab file with no comments, no spaces between lines, and lots of copy/paste. Step one of reproducible results: stop using matlab. Step two: give every author a "code reviewer" or "code tidier" to fix their rush-rush-rush-hurry-theres-a-deadline code into something others can at least evaluate against.

Nicolas_Rougier 11y ago

I think we would happily review the three submissions. Case 1. ok Case 2. this would need to extra precaution (maybe 3 reviewers) + public challenge to spot any error. Then ask directly the original author. If nothing works, we can declare the paper non reproducible. But it might be heavy responsibility for the you. Case 3. also ok and you can specify in the accompanying article there are several cases where things does not run as expected.