4 ms·

A while back I went through several of these kinds of build systems. The one I eventually settled on was Snakemake[1], with which I finally was able to put toge

A while back I went through several of these kinds of build systems. The one I eventually settled on was Snakemake[1], with which I finally was able to put together an entire complex bioinformatics workflow[2]. I would say that Snakemake's strength is one of Perl's old mottos: "making easy things easy and hard things possible." On the easy side, writing a simple rule with fixed inputs and outputs is just as easy as Make (trading the weird whitespace rules of Make for the less weird whitespace rules of Python). On the hard side, it supports stuff like computing inputs at runtime, or using regex capture groups to determine output file names from input file names. So for example you can have your rule for LaTeX to PDF conversion read the source TeX file, determine all the external images, bib files, etc. referenced from it, and add those files as input dependencies so that when any of them changes, Snakemake will know to rebuild the PDF. See e.g. this rule[3] that builds my resume PDF from the LyX source. [1]: https://snakemake.readthedocs.io/en/stable/ https://snakemake.readthedocs.io/en/stable/ [2]: https://github.com/DarwinAwardWinner/CD4-csaw https://github.com/DarwinAwardWinner/CD4-csaw [3]: https://github.com/DarwinAwardWinner/resume/blob/main/Snakefile#L169-L175 https://github.com/DarwinAwardWinner/resume/blob/main/Snakef...

nerdponx 4y ago
I really tried my best to use Snakemake in ~2018-2019 for day-to-day data science work. I found that its DSL was a confusing and (at least at the time) under-documented hybrid of Python and custom syntax. It also suffered from some of of the primary limitations of traditional Make: • Targets/outputs must be files, without even the opportunity to extend the space of possible targets e.g. using your own hash comparison function. This is not suitable for a workflow where the output is a directory of filenames that aren't known in advance, or a database table. • By default, can only run commands through a shell, not directly as a subprocess. At the same time, it also lacks support for nontrivial chaining and composition of commands. In your CD4 example you hack around this by literally writing your own pipelining command runner in Python; hard to say if that's any better than a big blob of shell script in a string! I agree that it definitely "makes hard things possible", but my experience did not give me the impression that it "makes easy things easy". It was frustratingly deficient in what I believe are obvious areas for improvement, and some of the most obnoxious Make pain points that I encounter on a regular basis, even for regular day-to-day software development. I ended up switching from Snakemake to DVC basically as soon as it came out, because it at least supports using directories as outputs, and has a few other nice features that specifically work well for data science project version control and artifact management/sharing for collaboration. That said, I had/have much less of a need for complicated workflow orchestration than for easy tracking & sharing of datasets and large artifacts (fitted models, processed tabular datasets), and for tracking directories as outputs (partitioned Parquet datasets). It would be interesting to go back and re-evaluate the space of workflow DSLs and task runners, because DVC does not have the complete feature set that Snakemake has, and IMO nor should it (I am already getting worried about its feature creep). I'm also not sure if or how well Snakemake and DVC could be made to work together; the former for task running and the latter for task input/output tracking with version control. The perfect tool (or better yet, the perfect combination of tools) remains elusive.
- rcthompson 4y ago
  Hehe, I forgot about the pipeline function. I had originally written that for something else and then repurposed it here. It always felt like something that ought to be built in to python. And I guess I'm kind of grading "easy" on a curve here, because the first build tool I tried for bioinformatics was GNU Make, and it was not up to the task, even when I exploited all the special GNU features to the hilt.
- kvnhn 4y ago
  I very much agree with you about DVC's feature creep. The other issue I have with it is speed. DVC has left me scratching my head at its sluggishness many times. Because of these factors, I've been working on an alternative that focuses on simplicity and speed[0]. My tool is often five to ten times faster than DVC[1]. I'd love to hear what you think. [0]: https://github.com/kevin-hanselman/dud https://github.com/kevin-hanselman/dud [1]: https://kevin-hanselman.github.io/dud/benchmarks/ https://kevin-hanselman.github.io/dud/benchmarks/
  - nerdponx 4y ago
    Thanks! I really like your clear explanation of how Dud differs from DVC (and I prefer your version in all cases). Would it be possible for Dud to push/pull from a DVC remote and use the DVC shared cache? That would be really useful so I (iconoclastic free software user) could use Dud on my machine/acocunt, but still share data and artifacts with other people (who don't give a shit what tool they use) using DVC on their machines/accounts. Also: Does Dud support reflinks at all? Or does it only support symlinks?
    - kvnhn 4y ago
      Unfortunately, there's a few things that currently hinder compatibility with DVC caches. First, Dud uses the Blake3 checksum algorithm, and DVC uses md5. This means the content-addressed caches will have completely different file names. Second, directories are committed to DVC differently than they are in Dud. For directories, not only will the committed file names not match (due to point 1), but the contents will not match either. Both of these things could be addressed, but it would take a lot of effort and would likely cost Dud in terms of its two main goals, speed and simplicity. I'm not opposed to this if we can make it work, though. Dud currently does not support reflinks, but I think adding reflink support would be fairly straight-forward. Just curious: What filesystem and OS are you using for reflinks? I'd be happy to chat more about this. Feel free to open GitHub issues for these items. I welcome contributions as well. ;)

nerdponx 4y ago

I really tried my best to use Snakemake in ~2018-2019 for day-to-day data science work. I found that its DSL was a confusing and (at least at the time) under-documented hybrid of Python and custom syntax. It also suffered from some of of the primary limitations of traditional Make: • Targets/outputs must be files, without even the opportunity to extend the space of possible targets e.g. using your own hash comparison function. This is not suitable for a workflow where the output is a directory of filenames that aren't known in advance, or a database table. • By default, can only run commands through a shell, not directly as a subprocess. At the same time, it also lacks support for nontrivial chaining and composition of commands. In your CD4 example you hack around this by literally writing your own pipelining command runner in Python; hard to say if that's any better than a big blob of shell script in a string! I agree that it definitely "makes hard things possible", but my experience did not give me the impression that it "makes easy things easy". It was frustratingly deficient in what I believe are obvious areas for improvement, and some of the most obnoxious Make pain points that I encounter on a regular basis, even for regular day-to-day software development. I ended up switching from Snakemake to DVC basically as soon as it came out, because it at least supports using directories as outputs, and has a few other nice features that specifically work well for data science project version control and artifact management/sharing for collaboration. That said, I had/have much less of a need for complicated workflow orchestration than for easy tracking & sharing of datasets and large artifacts (fitted models, processed tabular datasets), and for tracking directories as outputs (partitioned Parquet datasets). It would be interesting to go back and re-evaluate the space of workflow DSLs and task runners, because DVC does not have the complete feature set that Snakemake has, and IMO nor should it (I am already getting worried about its feature creep). I'm also not sure if or how well Snakemake and DVC could be made to work together; the former for task running and the latter for task input/output tracking with version control. The perfect tool (or better yet, the perfect combination of tools) remains elusive.

rcthompson 4y ago
Hehe, I forgot about the pipeline function. I had originally written that for something else and then repurposed it here. It always felt like something that ought to be built in to python. And I guess I'm kind of grading "easy" on a curve here, because the first build tool I tried for bioinformatics was GNU Make, and it was not up to the task, even when I exploited all the special GNU features to the hilt.
kvnhn 4y ago
I very much agree with you about DVC's feature creep. The other issue I have with it is speed. DVC has left me scratching my head at its sluggishness many times. Because of these factors, I've been working on an alternative that focuses on simplicity and speed[0]. My tool is often five to ten times faster than DVC[1]. I'd love to hear what you think. [0]: https://github.com/kevin-hanselman/dud https://github.com/kevin-hanselman/dud [1]: https://kevin-hanselman.github.io/dud/benchmarks/ https://kevin-hanselman.github.io/dud/benchmarks/
- nerdponx 4y ago
  Thanks! I really like your clear explanation of how Dud differs from DVC (and I prefer your version in all cases). Would it be possible for Dud to push/pull from a DVC remote and use the DVC shared cache? That would be really useful so I (iconoclastic free software user) could use Dud on my machine/acocunt, but still share data and artifacts with other people (who don't give a shit what tool they use) using DVC on their machines/accounts. Also: Does Dud support reflinks at all? Or does it only support symlinks?
  - kvnhn 4y ago
    Unfortunately, there's a few things that currently hinder compatibility with DVC caches. First, Dud uses the Blake3 checksum algorithm, and DVC uses md5. This means the content-addressed caches will have completely different file names. Second, directories are committed to DVC differently than they are in Dud. For directories, not only will the committed file names not match (due to point 1), but the contents will not match either. Both of these things could be addressed, but it would take a lot of effort and would likely cost Dud in terms of its two main goals, speed and simplicity. I'm not opposed to this if we can make it work, though. Dud currently does not support reflinks, but I think adding reflink support would be fairly straight-forward. Just curious: What filesystem and OS are you using for reflinks? I'd be happy to chat more about this. Feel free to open GitHub issues for these items. I welcome contributions as well. ;)