4 ms·

So far I've also always found `make` to be the best option for what I need, especially because it's already available everywhere (albeit sadly in very subtly in

So far I've also always found `make` to be the best option for what I need, especially because it's already available everywhere (albeit sadly in very subtly incompatible versions). Anyways, one thing that has always bothered me about make is that it so much depends on file modification dates. Imagine if it would instead use a very optimised hashing algorithm over the input content. Content can be a file or any URI, so it uses wget/curl and ssh as a dependency. Hashing is optimized such that it fails early - e.g. hash in n kB increments, mark as new and return if changed. Imagine how well this would now suddenly integrate into our modern web service based landscape. You could hook together services quite easily: all: report.csv report.csv: ${customers_json_uri} ${earnings_json_uri} ${report_executable} $^ > $@ Has anyone built something like this already?

LegNeato 10y ago
Buck (and Bazel) essentially do what you want and more: - Don't use modification times and instead do intelligent hashing (see https://buckbuild.com/concept/rule_keys.html https://buckbuild.com/concept/rule_keys.html) - Include the compiler and a bunch of other stuff in the hash. This means you can share cached artifacts safely via a http cache (https://buckbuild.com/concept/http_cache_api.html https://buckbuild.com/concept/http_cache_api.html) - Can refer to remote files (see https://buckbuild.com/rule/remote_file.html https://buckbuild.com/rule/remote_file.html) - Can use any script to do anything as long as it has a single consistent output (see https://buckbuild.com/rule/genrule.html https://buckbuild.com/rule/genrule.html) Because their model is so much better, you get crazy fast local caching and remote shared caching of artifacts. Please don't use make, the newer build tools are better in virtually every way. (disclosure: my team created Buck when I was at FB)
ProblemFactory 10y ago
I have briefly used https://github.com/Factual/drake https://github.com/Factual/drake for this. It is a bit unpolished and still a work in progress, but has some features that are useful for data science workflows that most software build tools do not cover: * Steps can produce multiple named outputs (for example full.csv and summary.csv), dependency graph can branch in both ways, * Outputs that are directories with many files, * Built-in input/output file sharding (for example by date or month) and HDFS support, * Lots of control over which steps to execute: not just "rebuild target X and any required upstream dependencies". Can select a target and all that depend on it downstream (if you know that a remote file changed), skip rebuilding some steps, etc, * Can put very short (text processing) scripts directly in the Drakefile in Python, Ruby.
- makapuf 10y ago
  I'm not commenting about Drake or its usefulness, but concerning (GNU) make : * you can specify several inputs & outputs per rule * you can specify which script interpreter is used ($SHELL), and python can be it. See https://www.gnu.org/software/make/manual/html_node/One-Shell.html https://www.gnu.org/software/make/manual/html_node/One-Shell... * downstream make can indeed be interesting. however you can try to build all ultimate targets ans let make sort out which ones can be avoided * fs support is best done at the os level (https://wiki.apache.org/hadoop/MountableHDFS https://wiki.apache.org/hadoop/MountableHDFS, not sure about timestamps) and * "everyone" "hates" gnu make, which means everyone knows it. (edit :format)
  - Arkanosis 10y ago
    AFAIK, you can't specify several outputs per rule in GNU Make without relying on horribly complex workarounds. It's actually the feature I miss the most in it.
    - lgas 10y ago
      Is this too complex and/or insufficient? https://www.gnu.org/software/automake/manual/html_node/Multiple-Outputs.html https://www.gnu.org/software/automake/manual/html_node/Multi...
      - Arkanosis 10y ago
        That's the kind of workaround I had in mind and is probably sufficient, but yes, ~25 lines of cryptic and error-prone interleaved GNU make and shell script just to express multiple outputs correctly is way above my personal threshold for “too complex” or even “sane”.
      - qznc 10y ago
        GNU Make works intuitively in pattern rules, but differently for normal rules? Uh oh.
        10y ago

LegNeato 10y ago

Buck (and Bazel) essentially do what you want and more: - Don't use modification times and instead do intelligent hashing (see https://buckbuild.com/concept/rule_keys.html https://buckbuild.com/concept/rule_keys.html) - Include the compiler and a bunch of other stuff in the hash. This means you can share cached artifacts safely via a http cache (https://buckbuild.com/concept/http_cache_api.html https://buckbuild.com/concept/http_cache_api.html) - Can refer to remote files (see https://buckbuild.com/rule/remote_file.html https://buckbuild.com/rule/remote_file.html) - Can use any script to do anything as long as it has a single consistent output (see https://buckbuild.com/rule/genrule.html https://buckbuild.com/rule/genrule.html) Because their model is so much better, you get crazy fast local caching and remote shared caching of artifacts. Please don't use make, the newer build tools are better in virtually every way. (disclosure: my team created Buck when I was at FB)

ProblemFactory 10y ago

I have briefly used https://github.com/Factual/drake https://github.com/Factual/drake for this. It is a bit unpolished and still a work in progress, but has some features that are useful for data science workflows that most software build tools do not cover: * Steps can produce multiple named outputs (for example full.csv and summary.csv), dependency graph can branch in both ways, * Outputs that are directories with many files, * Built-in input/output file sharding (for example by date or month) and HDFS support, * Lots of control over which steps to execute: not just "rebuild target X and any required upstream dependencies". Can select a target and all that depend on it downstream (if you know that a remote file changed), skip rebuilding some steps, etc, * Can put very short (text processing) scripts directly in the Drakefile in Python, Ruby.

makapuf 10y ago
I'm not commenting about Drake or its usefulness, but concerning (GNU) make : * you can specify several inputs & outputs per rule * you can specify which script interpreter is used ($SHELL), and python can be it. See https://www.gnu.org/software/make/manual/html_node/One-Shell.html https://www.gnu.org/software/make/manual/html_node/One-Shell... * downstream make can indeed be interesting. however you can try to build all ultimate targets ans let make sort out which ones can be avoided * fs support is best done at the os level (https://wiki.apache.org/hadoop/MountableHDFS https://wiki.apache.org/hadoop/MountableHDFS, not sure about timestamps) and * "everyone" "hates" gnu make, which means everyone knows it. (edit :format)
- Arkanosis 10y ago
  AFAIK, you can't specify several outputs per rule in GNU Make without relying on horribly complex workarounds. It's actually the feature I miss the most in it.
  - lgas 10y ago
    Is this too complex and/or insufficient? https://www.gnu.org/software/automake/manual/html_node/Multiple-Outputs.html https://www.gnu.org/software/automake/manual/html_node/Multi...
    - Arkanosis 10y ago
      That's the kind of workaround I had in mind and is probably sufficient, but yes, ~25 lines of cryptic and error-prone interleaved GNU make and shell script just to express multiple outputs correctly is way above my personal threshold for “too complex” or even “sane”.
    - qznc 10y ago
      GNU Make works intuitively in pattern rules, but differently for normal rules? Uh oh.
      - 10y ago