3 ms·

I see it can be used to define and run workflows. But for reproducibility on top of the execution of operations on data, I'm wondering if there's any way to ver

I see it can be used to define and run workflows. But for reproducibility on top of the execution of operations on data, I'm wondering if there's any way to version the underlying data? Or would you typically use this -in addition- to a tool like dvc? I've used dvc a bit and while it's quite good for data versioning, I find the workflow aspect is clunky.

jlduan 3y ago
I am very fond of https://zenodo.org https://zenodo.org, especially for small datasets for scientific publications.
samesense 3y ago
I use snakemake along with dvc. It detects changes in files, and reruns steps to produce downstream files as needed.
samuell 3y ago
The way a workflow like Snakemake can help here is generally by letting the filenames pretty much describe how each particular output was created, meaning data outputs can act as immutable in a sense. What I mean is that rather than create a new version of a file, if you run the same analysis with different sets of parameters, it should generate a new file with a different name rather than a new version of the old one. This also helps comparing differences between output from different parameters etc. That said, there are workflow platforms which support data versioning, such as Pachyderm (https://pachyderm.com https://pachyderm.com), but it is a bit more heavyweight as it runs on top of Kubernetes.
- bafe 3y ago
  The reliance on filenames to define (parametric) dependencies was among the reasons I later adopted nextflow. The model fit the type of computation dependencies better for my case. In the mean time snakemake grew and many DAG that were hard to describe back then are now expressed directly with snakemake primitives
troymc 3y ago
Pachyderm has data versioning / data version control built-in. I guess other tools do too.
bafe 3y ago
There is indeed an old issue asking whether they could return a provenance graph using the prov ontology in JSON format https://github.com/snakemake/snakemake/issues/2077 https://github.com/snakemake/snakemake/issues/2077 I think it's a good task to work on if you like to contribute

jlduan 3y ago

I am very fond of https://zenodo.org https://zenodo.org, especially for small datasets for scientific publications.

samesense 3y ago

I use snakemake along with dvc. It detects changes in files, and reruns steps to produce downstream files as needed.

samuell 3y ago

The way a workflow like Snakemake can help here is generally by letting the filenames pretty much describe how each particular output was created, meaning data outputs can act as immutable in a sense. What I mean is that rather than create a new version of a file, if you run the same analysis with different sets of parameters, it should generate a new file with a different name rather than a new version of the old one. This also helps comparing differences between output from different parameters etc. That said, there are workflow platforms which support data versioning, such as Pachyderm (https://pachyderm.com https://pachyderm.com), but it is a bit more heavyweight as it runs on top of Kubernetes.

bafe 3y ago
The reliance on filenames to define (parametric) dependencies was among the reasons I later adopted nextflow. The model fit the type of computation dependencies better for my case. In the mean time snakemake grew and many DAG that were hard to describe back then are now expressed directly with snakemake primitives

troymc 3y ago

Pachyderm has data versioning / data version control built-in. I guess other tools do too.

bafe 3y ago

There is indeed an old issue asking whether they could return a provenance graph using the prov ontology in JSON format https://github.com/snakemake/snakemake/issues/2077 https://github.com/snakemake/snakemake/issues/2077 I think it's a good task to work on if you like to contribute