4 ms·

I use snakemake[0] as a replacement for make. The debugging tools in snakemake are really good - it can output the DAG of your workflow as a dotfile, and you ca

I use snakemake[0] as a replacement for make. The debugging tools in snakemake are really good - it can output the DAG of your workflow as a dotfile, and you can add breakpoints and use pdb to debug your workflow. It also integrates really nicely with conda, so you can have different environments for different rules. The killer feature of snakemake for me is that it can run a workflow on a single core or as many as you give it, or in the case of a cluster it can actually submit jobs to a queue and wait for them to finish before submitting more. The tradeoff is that you need to install it (conda install -c bioconda snakemake), and that it can be very slow if you have thousands of target files. But for most workflows it's the right tool for me. [0] http://snakemake.readthedocs.io/en/stable/ http://snakemake.readthedocs.io/en/stable/

blcArmadillo 9y ago
> The killer feature of snakemake for me is that it can run a workflow on a single core or as many as you give it, or in the case of a cluster it can actually submit jobs to a queue and wait for them to finish before submitting more. Other than your cluster example isn't this the same as passing -j to make to run jobs in parallel?
- rflrob 9y ago
  Yeah, -j just about does it, though with Snakemake you can also specify other resources (memory, I/o, etc) for each task, and it will keep all of them under quota. So if you tell snakemake that task A takes six cores, but B-K take only one each, it will solve the knapsack problem to come up with some kind of optimal order for them, and on your 8-core machine you could have A, B, and C running, or B-K. For a while, I muddled through with Make and a custom cluster job-submitter script, but the real Snakemake killer feature for me was multiple, independent wildcards. So I can, in one rule, have dir/{databasename}/{query}.tsv and rapidly hit multiple databases at once. If you are very clever (and rule one of Make is don’t be clever), you can have a GNUMake wildcard get parsed into multiple wildcards with the wimpy pattern substitution rules, but this is not a fun way to relax.
  - jschwartzi 9y ago
    Oh, cool, so I can finally run make -j target without worrying about the LMK in Linux killing my window manager!

blcArmadillo 9y ago

> The killer feature of snakemake for me is that it can run a workflow on a single core or as many as you give it, or in the case of a cluster it can actually submit jobs to a queue and wait for them to finish before submitting more. Other than your cluster example isn't this the same as passing -j to make to run jobs in parallel?

rflrob 9y ago
Yeah, -j just about does it, though with Snakemake you can also specify other resources (memory, I/o, etc) for each task, and it will keep all of them under quota. So if you tell snakemake that task A takes six cores, but B-K take only one each, it will solve the knapsack problem to come up with some kind of optimal order for them, and on your 8-core machine you could have A, B, and C running, or B-K. For a while, I muddled through with Make and a custom cluster job-submitter script, but the real Snakemake killer feature for me was multiple, independent wildcards. So I can, in one rule, have dir/{databasename}/{query}.tsv and rapidly hit multiple databases at once. If you are very clever (and rule one of Make is don’t be clever), you can have a GNUMake wildcard get parsed into multiple wildcards with the wimpy pattern substitution rules, but this is not a fun way to relax.
- jschwartzi 9y ago
  Oh, cool, so I can finally run make -j target without worrying about the LMK in Linux killing my window manager!