4 ms·

I've tried a few other techniques, but there is something that is really hard to beat about the plain Makefile. Eventually, it can become unwieldy, but for smal

I've tried a few other techniques, but there is something that is really hard to beat about the plain Makefile. Eventually, it can become unwieldy, but for small-medium projects, it's a life saver. Combine that with git, and you've got a pretty robust system. The only real downside is that when you have very large initial inputs, it isn't practical to store those in the same repository. That is my goto combinations for bioinformatics workflows that don't need to run on a cluster. (A Makefile variant of mine is my go-to for clusters). I've experimented with Docker workflows, which while in theory are nice, you'll always require some sort of organizing script to document the analysis. (Docker is still nice for capturing program versioning/configuration though!)

cryptarch 9y ago
You could use Git-LFS for managing huge data blobs with Git?
paulgb 9y ago
I like the Cookiecutter Data Science approach of keeping data out of the repo: https://drivendata.github.io/cookiecutter-data-science/ https://drivendata.github.io/cookiecutter-data-science/ I usually work with data on an internal cloud anyway, so instead of storing data on my local disk I just keep text files that contain the URI of the cloud resource. I treat the cloud resources as immutable once created so that the "pointer" file's modified time is consistent with the modified time of the data on the cloud.
- in9 9y ago
  I really like their approach, but their template is too opinionated towards python + makefiles. At my company we use mixes of R, python and Scala, together with airflow. We have been toying with our own templates, and should release them eventually
  - paulgb 9y ago
    Yes, I actually only use Python for notebooks, so I mostly ignore the Python specific parts. I think their data/ directory structure (along with keeping it out of the repo) is a sane approach though. (I don't actually use their template, I just use the directory layout as guidance)

cryptarch 9y ago

You could use Git-LFS for managing huge data blobs with Git?

paulgb 9y ago

I like the Cookiecutter Data Science approach of keeping data out of the repo: https://drivendata.github.io/cookiecutter-data-science/ https://drivendata.github.io/cookiecutter-data-science/ I usually work with data on an internal cloud anyway, so instead of storing data on my local disk I just keep text files that contain the URI of the cloud resource. I treat the cloud resources as immutable once created so that the "pointer" file's modified time is consistent with the modified time of the data on the cloud.

in9 9y ago
I really like their approach, but their template is too opinionated towards python + makefiles. At my company we use mixes of R, python and Scala, together with airflow. We have been toying with our own templates, and should release them eventually
- paulgb 9y ago
  Yes, I actually only use Python for notebooks, so I mostly ignore the Python specific parts. I think their data/ directory structure (along with keeping it out of the repo) is a sane approach though. (I don't actually use their template, I just use the directory layout as guidance)