4 ms·

I have briefly used https://github.com/Factual/drake https://github.com/Factual/drake for this. It is a bit unpolished and still a work in progress, but has so

I have briefly used https://github.com/Factual/drake https://github.com/Factual/drake for this. It is a bit unpolished and still a work in progress, but has some features that are useful for data science workflows that most software build tools do not cover: * Steps can produce multiple named outputs (for example full.csv and summary.csv), dependency graph can branch in both ways, * Outputs that are directories with many files, * Built-in input/output file sharding (for example by date or month) and HDFS support, * Lots of control over which steps to execute: not just "rebuild target X and any required upstream dependencies". Can select a target and all that depend on it downstream (if you know that a remote file changed), skip rebuilding some steps, etc, * Can put very short (text processing) scripts directly in the Drakefile in Python, Ruby.

makapuf 10y ago
I'm not commenting about Drake or its usefulness, but concerning (GNU) make : * you can specify several inputs & outputs per rule * you can specify which script interpreter is used ($SHELL), and python can be it. See https://www.gnu.org/software/make/manual/html_node/One-Shell.html https://www.gnu.org/software/make/manual/html_node/One-Shell... * downstream make can indeed be interesting. however you can try to build all ultimate targets ans let make sort out which ones can be avoided * fs support is best done at the os level (https://wiki.apache.org/hadoop/MountableHDFS https://wiki.apache.org/hadoop/MountableHDFS, not sure about timestamps) and * "everyone" "hates" gnu make, which means everyone knows it. (edit :format)
- Arkanosis 10y ago
  AFAIK, you can't specify several outputs per rule in GNU Make without relying on horribly complex workarounds. It's actually the feature I miss the most in it.
  - lgas 10y ago
    Is this too complex and/or insufficient? https://www.gnu.org/software/automake/manual/html_node/Multiple-Outputs.html https://www.gnu.org/software/automake/manual/html_node/Multi...
    - Arkanosis 10y ago
      That's the kind of workaround I had in mind and is probably sufficient, but yes, ~25 lines of cryptic and error-prone interleaved GNU make and shell script just to express multiple outputs correctly is way above my personal threshold for “too complex” or even “sane”.
    - qznc 10y ago
      GNU Make works intuitively in pattern rules, but differently for normal rules? Uh oh.
      - Arkanosis 10y ago
        It even has builtin implicit rules. Enabled by default. Checking for the existence of files that are not found in any new project for maybe ten or twenty years (such as RCS and SCSS version control files). On. Every. Single. Run. Yup.
        xyzzy_plugh 10y ago
        But I love the built-in implicit rules. I use these frequently. Often, I write a little test program in C or C++. I name the file "foo.c". Then, to build it I simply type `make foo`. Sure the rules may be a bit stale, but there isn't much software i use today that I could use in a near identical manner 40 years ago. Make has truly stood the test of time.
    - lomnakkus 10y ago
      That doesn't really look like "handling it" by any sensible definition. That looks like a huge kludgy workaround, almost like user implementing a build system inside the build system. (I mean, wtf, asking the user to implement a lock-and-set operation inside their build?)

makapuf 10y ago

I'm not commenting about Drake or its usefulness, but concerning (GNU) make : * you can specify several inputs & outputs per rule * you can specify which script interpreter is used ($SHELL), and python can be it. See https://www.gnu.org/software/make/manual/html_node/One-Shell.html https://www.gnu.org/software/make/manual/html_node/One-Shell... * downstream make can indeed be interesting. however you can try to build all ultimate targets ans let make sort out which ones can be avoided * fs support is best done at the os level (https://wiki.apache.org/hadoop/MountableHDFS https://wiki.apache.org/hadoop/MountableHDFS, not sure about timestamps) and * "everyone" "hates" gnu make, which means everyone knows it. (edit :format)

Arkanosis 10y ago
AFAIK, you can't specify several outputs per rule in GNU Make without relying on horribly complex workarounds. It's actually the feature I miss the most in it.
- lgas 10y ago
  Is this too complex and/or insufficient? https://www.gnu.org/software/automake/manual/html_node/Multiple-Outputs.html https://www.gnu.org/software/automake/manual/html_node/Multi...
  - Arkanosis 10y ago
    That's the kind of workaround I had in mind and is probably sufficient, but yes, ~25 lines of cryptic and error-prone interleaved GNU make and shell script just to express multiple outputs correctly is way above my personal threshold for “too complex” or even “sane”.
  - qznc 10y ago
    GNU Make works intuitively in pattern rules, but differently for normal rules? Uh oh.
    - Arkanosis 10y ago
      It even has builtin implicit rules. Enabled by default. Checking for the existence of files that are not found in any new project for maybe ten or twenty years (such as RCS and SCSS version control files). On. Every. Single. Run. Yup.
      - xyzzy_plugh 10y ago
        But I love the built-in implicit rules. I use these frequently. Often, I write a little test program in C or C++. I name the file "foo.c". Then, to build it I simply type `make foo`. Sure the rules may be a bit stale, but there isn't much software i use today that I could use in a near identical manner 40 years ago. Make has truly stood the test of time.
  - lomnakkus 10y ago
    That doesn't really look like "handling it" by any sensible definition. That looks like a huge kludgy workaround, almost like user implementing a build system inside the build system. (I mean, wtf, asking the user to implement a lock-and-set operation inside their build?)