3 ms·

Snakemake seems really interesting (in a good way). I was going to say I wasn't aware of it but then in watching the Intro video I realized I had seen it before

Snakemake seems really interesting (in a good way). I was going to say I wasn't aware of it but then in watching the Intro video I realized I had seen it before. However, I find myself scratching my head about when I would use it, or what need it's fulfilling. It seems like most of what it brings to the table should be directly inferrable from the analysis and presentation code and comments. If you're working with multiple systems, isn't this what scripting languages were originally for? It seems like another layer of complexity on top of what's already there, which ... just adds complexity. Or is the idea to make a data analysis-specific scripting DSL?

teekert 3y ago
However, I find myself scratching my head about when I would use it, or what need it's fulfilling. When you are in Next Generation Sequencing data analysis, it is basically THE tool. Snakemake comes with anything I ever needed, since it uses conda to install workflow dependencies, tools like BWA, fastqc, multiqc etc, are all present and fresh. I feel that Snakemake was made for NGS data analysis. Many Snakemake users, like me, are biologists that know some Python and bash, I have never used Make for example. The people pointing out weaknesses here have complex, CS related use cases it seems, for bio-informaticians in the NGS field it's a godsend.
matthew_stone 3y ago
Yes, it's a DSL. Here's a simple scatter-gather example. Let's say you want to count the number of lines in each file for a list of samples, and report a table of counts collected from each sample. Define a rule to process each input file, and a rule to collect the results. I find this much less complex than an equivalent bash workflow. Additionally, these rules can be easily containerized, the workflow can be parallelized, and the workflow is robust to interruption and the addition of new samples. Snakemake manages checking for existing files and running rules as necessary to create missing files, logic that is much more finicky to implement by hand in bash. with open('data/samples.txt') as slist: SAMPLES = [l.strip() for l in slist.readlines()] rule all: input: "results/line_counts.txt" rule count_lines: input: "data/lines/{sample}.txt" output: "processed/count_lines/{sample}.txt" shell: """ cat {input} | wc -l | paste <(echo -e {wildcards.sample}) - > {output} """ rule collect_counts: input: expand("processed/count_lines/{sample}.txt", sample=SAMPLES) output: "results/line_counts.txt" shell: """ cat <(echo -e "sample\tn_lines") {input} > {output} """
- c0l0 3y ago
  This looks like... an unconstrained amalgamation of YAML, python, and zsh/bash. Knowing all these building blocks quite well, I cannot say I find this attractive at all. Seems like it is bound to incur all the problems that having everything in YAML cursed configuration management and deployment orchestration with in ansible.

teekert 3y ago

However, I find myself scratching my head about when I would use it, or what need it's fulfilling. When you are in Next Generation Sequencing data analysis, it is basically THE tool. Snakemake comes with anything I ever needed, since it uses conda to install workflow dependencies, tools like BWA, fastqc, multiqc etc, are all present and fresh. I feel that Snakemake was made for NGS data analysis. Many Snakemake users, like me, are biologists that know some Python and bash, I have never used Make for example. The people pointing out weaknesses here have complex, CS related use cases it seems, for bio-informaticians in the NGS field it's a godsend.

matthew_stone 3y ago

Yes, it's a DSL. Here's a simple scatter-gather example. Let's say you want to count the number of lines in each file for a list of samples, and report a table of counts collected from each sample. Define a rule to process each input file, and a rule to collect the results. I find this much less complex than an equivalent bash workflow. Additionally, these rules can be easily containerized, the workflow can be parallelized, and the workflow is robust to interruption and the addition of new samples. Snakemake manages checking for existing files and running rules as necessary to create missing files, logic that is much more finicky to implement by hand in bash. with open('data/samples.txt') as slist: SAMPLES = [l.strip() for l in slist.readlines()] rule all: input: "results/line_counts.txt" rule count_lines: input: "data/lines/{sample}.txt" output: "processed/count_lines/{sample}.txt" shell: """ cat {input} | wc -l | paste <(echo -e {wildcards.sample}) - > {output} """ rule collect_counts: input: expand("processed/count_lines/{sample}.txt", sample=SAMPLES) output: "results/line_counts.txt" shell: """ cat <(echo -e "sample\tn_lines") {input} > {output} """

c0l0 3y ago
This looks like... an unconstrained amalgamation of YAML, python, and zsh/bash. Knowing all these building blocks quite well, I cannot say I find this attractive at all. Seems like it is bound to incur all the problems that having everything in YAML cursed configuration management and deployment orchestration with in ansible.