3 ms·

fabfile.py (Fabric) could be used as a Makefile in Python. If you don't ever need to ssh to other machines ti run your tasks, you could use pyinvoke library dir

by d0mine 7y ago

fabfile.py (Fabric) could be used as a Makefile in Python. If you don't ever need to ssh to other machines ti run your tasks, you could use pyinvoke library directly (tasks.py). https://www.fabfile.org/ https://www.fabfile.org/ It is easy to add command line arguments to the tasks, configure them using files (json, yaml), environment variables, to split the task definitions into several modules/namespaces.

deleted 7y ago
[deleted]
bb88 7y ago
Having used fabric in the past, I've always found it just as easy to use a shell script and make files. There's always some level of bootstrapping a project (installing packages/software, compiling libraries and dependencies) where it's easier to just to write a shell script than to program python to do. E.g. How do you get fabric installed on a system? There's also been this longevity of sorts that Make seems to have gotten right. People just keep going back to it because it's simple.
- theossuary 7y ago
  I've been moving away from using shell scripts in a tools/ directory to using Python Invoke (http://www.pyinvoke.org http://www.pyinvoke.org), which is the library underlying fabric. I used bash scripts for years, but for a lot of reasons made the switch: - It was always painful to create small libraries of functions used across multiple scripts in a project - It's difficult to consistently configure them with per-user settings. I've written bash implementations of settings management, Invoke handles this for me. - I'd still have to reach for Python whenever I needed to do anything with arrays or dicts, etc. - Getting error handling correct can be a chore Invoke has a lot of nice to haves to: - Autogenerated help output - Autocomplete out of the box - Very easy to add tasks, just a Python function - Easy to run shell code when needed - Very powerful config management when needed - Supports namespacing, task dependencies, running multiple tasks at once and deduplicating them It's not perfect, but it's a lot better than my hand rolled scripts were.
  - Groxx 7y ago
    it's also far, far, far easier to make it work predictably on multiple platforms. and easier to understand and change later. that can get nightmarishly hard in make/bash, once you go outside the super-simple realm.
  - noobermin 7y ago
    Groxx replies hits the point. I might work with a small number of platforms, but the "super-simple" qualifier is the point. The point at which you need dictionaries (associative arrays) in your install script, not to mention settings management beyond a make include is the point at which you've outgrown make.
    - deleted 7y ago
      [deleted]
anewhnaccount2 7y ago
Snakemake is a quite nice improvement on make for data munging stuff.
- kabacha 7y ago
  Wow I've never seen such bloated python project before. It has 10 dependancies with 2 additional optional dependancies and the introduction/tutorial is absurdly overspecific. The first example they use to describe the tool is: > Cufflinks is a tool to assemble transcripts, calculate abundance and conduct a differential expression analysis on RNA-Seq data. his example shows how to create a typical Cufflinks workflow with Snakemake. It assumes that mapped RNA-Seq data for four samples 101-104 is given as bam files. This is epitome of non-programmer programming, colour me disappointed.
  - anewhnaccount2 7y ago
    And yet it's still less crufty than the "by hackers for hackers" GNU Automake and less over engineered than the "made by real professional programmers at a real big tech company" Luigi. Would love the hear if you have any suggestions for actual alternatives for doing this type of automation beyond what make can neatly deal with rather than just going "eww.. it has dependencies"; "eww.. it's made by bioinformaticists".

by d0mine 7y ago

deleted 7y ago

[deleted]

bb88 7y ago

Having used fabric in the past, I've always found it just as easy to use a shell script and make files. There's always some level of bootstrapping a project (installing packages/software, compiling libraries and dependencies) where it's easier to just to write a shell script than to program python to do. E.g. How do you get fabric installed on a system? There's also been this longevity of sorts that Make seems to have gotten right. People just keep going back to it because it's simple.

theossuary 7y ago
I've been moving away from using shell scripts in a tools/ directory to using Python Invoke (http://www.pyinvoke.org http://www.pyinvoke.org), which is the library underlying fabric. I used bash scripts for years, but for a lot of reasons made the switch: - It was always painful to create small libraries of functions used across multiple scripts in a project - It's difficult to consistently configure them with per-user settings. I've written bash implementations of settings management, Invoke handles this for me. - I'd still have to reach for Python whenever I needed to do anything with arrays or dicts, etc. - Getting error handling correct can be a chore Invoke has a lot of nice to haves to: - Autogenerated help output - Autocomplete out of the box - Very easy to add tasks, just a Python function - Easy to run shell code when needed - Very powerful config management when needed - Supports namespacing, task dependencies, running multiple tasks at once and deduplicating them It's not perfect, but it's a lot better than my hand rolled scripts were.
- Groxx 7y ago
  it's also far, far, far easier to make it work predictably on multiple platforms. and easier to understand and change later. that can get nightmarishly hard in make/bash, once you go outside the super-simple realm.
- noobermin 7y ago
  Groxx replies hits the point. I might work with a small number of platforms, but the "super-simple" qualifier is the point. The point at which you need dictionaries (associative arrays) in your install script, not to mention settings management beyond a make include is the point at which you've outgrown make.
  - deleted 7y ago
    [deleted]

anewhnaccount2 7y ago

Snakemake is a quite nice improvement on make for data munging stuff.

kabacha 7y ago
Wow I've never seen such bloated python project before. It has 10 dependancies with 2 additional optional dependancies and the introduction/tutorial is absurdly overspecific. The first example they use to describe the tool is: > Cufflinks is a tool to assemble transcripts, calculate abundance and conduct a differential expression analysis on RNA-Seq data. his example shows how to create a typical Cufflinks workflow with Snakemake. It assumes that mapped RNA-Seq data for four samples 101-104 is given as bam files. This is epitome of non-programmer programming, colour me disappointed.
- anewhnaccount2 7y ago
  And yet it's still less crufty than the "by hackers for hackers" GNU Automake and less over engineered than the "made by real professional programmers at a real big tech company" Luigi. Would love the hear if you have any suggestions for actual alternatives for doing this type of automation beyond what make can neatly deal with rather than just going "eww.. it has dependencies"; "eww.. it's made by bioinformaticists".