4 ms·

This is spot on, thank you for this summary! I just finished working on an exploratory data science paper that was in the making for the last 5 years. In the b

This is spot on, thank you for this summary! I just finished working on an exploratory data science paper that was in the making for the last 5 years. In the beginning, I set the goal to version the whole process in git. It was incredibly difficult to not accumulate an immense mountain of tech debt during this whole time. We had 4 internal review rounds (5 researchers from two universities), then 2 immediate rejections ("not fitting the scope of journal"), which I considered Major Revisions (because you need to significantly adjust the paper to each new journal) and at the final journal now 1 major revision, maybe another one (but I am not sure yet). All in all, there were about 7 Major Revisions, each taking about 3 to 6 months of 8 hour shifts. The paper now consists of 10 (chained) Jupyter Notebooks and about 16000 lines of code with 1200 git revisions (the notebooks are versioned in Markdown using Jupytext). I cannot account for the fundamental changes that were necessary due to all the external input - this is really very difficult to plan for in the beginning, when you set out for such a project. I hear my IT colleague arguing "you need tests for all of this!" and then I face another day where I need to write 25 new functions in 6 hours, while preparing lectures, workshops, assess master theses etc. - impossible to me, without the flexibility of python. It is at least pretty different to how things work in production at companies, where you have at least some clear goals/rules.

periheli0n 4y ago
> The paper now consists of 10 Jupyter Notebooks and about 16000 lines of code with 1200 git revisions I‘ve been there too! And about 1 year after the paper was finally published, fresh starters in my lab couldn’t run the code anymore because of an obscure Pandas error caused by a version change. Then Python 2.7 was shelved and created more problems. In conclusion the considerable effort I put in to create Jupyter notebooks recreating every single figure from the paper was not worth it.
- Helmut10001 4y ago
  I created Docker Container Images for every versioned "release" of the paper that we maintain in our Gitlab Registry (a CI builds those images automatically from `.gitlab-ci.yml`, `docker-compose.yml` using `docker:dind`), so you can pull a specific Docker Image for every version of the paper that will definitely have the correct dependencies (because the Jupyter Notebooks were tested with each specific version), including Jupyter itself.
  - ogarten 4y ago
    That is awesome and should be the minimum standard, but even just getting the entire infrastructure set up is something almost all scientists are not willing to do. In my experience, one should be happy if code is version controlled in a proper way. Too often, it isn't. In ML this might be a little different but in my field at least (electrical engineering) this is not the case at all.
    - Helmut10001 4y ago
      Yes, I know and I was pretty stubborn with my initial goals to go through with this.. (probably will also cost my employment, but I learned a lot and it was thus still worth it).

periheli0n 4y ago

> The paper now consists of 10 Jupyter Notebooks and about 16000 lines of code with 1200 git revisions I‘ve been there too! And about 1 year after the paper was finally published, fresh starters in my lab couldn’t run the code anymore because of an obscure Pandas error caused by a version change. Then Python 2.7 was shelved and created more problems. In conclusion the considerable effort I put in to create Jupyter notebooks recreating every single figure from the paper was not worth it.

Helmut10001 4y ago
I created Docker Container Images for every versioned "release" of the paper that we maintain in our Gitlab Registry (a CI builds those images automatically from `.gitlab-ci.yml`, `docker-compose.yml` using `docker:dind`), so you can pull a specific Docker Image for every version of the paper that will definitely have the correct dependencies (because the Jupyter Notebooks were tested with each specific version), including Jupyter itself.
- ogarten 4y ago
  That is awesome and should be the minimum standard, but even just getting the entire infrastructure set up is something almost all scientists are not willing to do. In my experience, one should be happy if code is version controlled in a proper way. Too often, it isn't. In ML this might be a little different but in my field at least (electrical engineering) this is not the case at all.
  - Helmut10001 4y ago
    Yes, I know and I was pretty stubborn with my initial goals to go through with this.. (probably will also cost my employment, but I learned a lot and it was thus still worth it).