4 ms·
Snakemake is a beautiful project and evolves and improves so fast. Years ago I realized I needed to up my game from the usual bash based NGS data processing pip
by teekert 3y ago
Snakemake is a beautiful project and evolves and improves so fast. Years ago I realized I needed to up my game from the usual bash based NGS data processing pipelines I was writing. Based on several recommendation I choose Snakemake. I have never regretted it, It worked perfectly on our PBS cluster then on our Slurm cluster. I made some steps to make it run on K8s, which it supports, and most recently, I'm still/again happy with my choice for Snakemake because it (together with Nextflow) seems to be the chosen framework for GA4GH's cloud work stream's "products" like WES and TES [0]. This seems to be the tech stack where Amazon Omics and Microsoft Genomics focus on [1]. It enables many cool things, like "data visiting": Just submit your Snakefile (the definition of a workflow, a DAG basically) to a WES API in the same data center where your data lives, and data analysis starts, near the data. Brilliant.
I owe a lot to Snakemake and Johannes Köster, I hope some day I can repay him and his project.
[0] https://www.ga4gh.org/work_stream/cloud/ https://www.ga4gh.org/work_stream/cloud/
[1] https://github.com/Microsoft/tes-azure https://github.com/Microsoft/tes-azure
- matthew_stone 3y ago100% agree, and it's wonderful to see Snakemake on the top of HN. Snakemake is an invaluable tool in bioinformatics analysis. It's a testament to Johannes' talent and dedication that, even with the relatively limited resources of an academic developer, Snakemake has remained broadly useful and popular. Super nice guy too, he's always been remarkably responsive and helpful. I saw him present on Snakemake back when he was a postdoc, and it really changed my approach to pipeline development.
- bsmith89 3y agoI too owe a lot of my PhD and postdoc productivity to Snakemake. It's my bioinformatics super-power, allowing me to run a complex analysis, including downloading containers (Singularity/Apptainer) and other dependencies (conda), with one command. Great for reproducibility. Great for development. Great for scaling analyses. Snakemake is vital infrastructure for my work.
- tetris11 3y agoIts fantastic but it doesn't scale laterelly particularly well, compared to just Make.
- ta988 3y agoWhat dimension are you referring to?
- tetris11 3y agoLarge scale reproducibility was a problem a few years back for one. Conda and containers were a constant problem for us back then, especially if you had multiple NGS tools running in different environments. This has probably been solved by now, but we went with another workflow system