4 ms·

Djb redo[1], a make alternative, feels like a good fit for these type of data manipulation and dependency representations. Below is a port of the first example.

Djb redo[1], a make alternative, feels like a good fit for these type of data manipulation and dependency representations. Below is a port of the first example. The build script is just shell, so you can do stuff like embed python with a heredoc. One bit of syntactic sugar is that redo assumes stdout is the desired contents of the generated file, so you don't need to explicitly pipe to an OUTPUT variable. #!/bin/sh case $1 in contracts.csv) curl http://www.ferc.gov/docs-filing/eqr/soft-tools/sample-csv/contract.txt ;; evergreens.csv) redo-ifchange contracts.csv grep Evergreen contracts.csv ;; report.txt) input=evergreens.csv redo-ifchange $input python2 <<-EOF linecount = len(file("$input").readlines()) print("File $input has {0} lines.\n".format(linecount)) EOF ;; esac [1] https://github.com/apenwarr/redo https://github.com/apenwarr/redo

aboytsov 14y ago
Please see my response to Make comparison: http://news.ycombinator.com/item?id=5111527 http://news.ycombinator.com/item?id=5111527 I suspect most of the points I made would be applicable to redo as well, if not more so. Trivial things don't require Drake. Heck, they often times don't require Make as well - just put it in a linear shell script if the steps are not too expensive. It's when things are getting complicated you need something like Drake.
- moonboots 14y ago
  Redo lacks features baked into Drake, especially the Hadoop integration, but I believe it would be easier to incorporate custom functionality into redo versus hacking Make or writing a custom build system. I haven't used Drake, so I would be interested in a small but complicated Drake script which tackles an intractable problem in Make. I don't claim redo can provide a cleaner solution than a purpose-built system, but I think it will be unexpectedly simple.
  - aboytsov 14y ago
    The most crucial thing that Make lacks is multiple outputs and precise control over execution. When you're debugging/developing a large and expensive workflow, you absolutely must have the ability to say things like: - run only this step, I'm debugging it - I've changed implementation of this step, re-build it and everything that depends on it - build everything except this branch, it's expensive and I don't need to rebuild it that often (example: model training) Other examples of intractable problems in Make would be timestamped dependency resolution between local and HDFS files. If Make can't look at HDFS, it can't say if the step needs to be built or not. I don't think you can fix it with external commands. But generally, search for intractable problems is a futile one. Remember, everything you can code in Java, you can code in a Turing machine. :)
    - lars512 14y ago
      Make can certainly generate multiple outputs, and can trivially be coerced to redo any step you like. Provided you add your code as a dependency in the analysis, then it will happily redo only what's changed, giving you nice tight iterations. I think it's real limitations are with multi-machine setups, as in the HDFS problem you're mentioning. Then you need a new tool.
      - aboytsov 14y ago
        Sorry, I might be very ignorant of make - could you please give me a command to re-build a particular target and everything that depends on it?
        lars512 14y ago
        So make's default behaviour "make somefile.csv" is to build the whole tree of dependencies. To force rebuild of everything, run "make -B somefile.csv". It then assumes everything is out of date. To force rebuild of one step, just delete its output or run "touch" on one of its dependencies before running make. Then that step will get redone. I like to have generated data in a separate folder, say "output/" which you can then snapshot, blow away, or do what you like with. Basically though, I keep it separate from data and code inputs.
groby_b 14y ago
Taking off my programmer hat, and putting on my "I just want to get data moved along" hat - drake is much more readable than this. The ideas behind redo are brilliant, but the way to express them in this particular implementation is not so fun.

aboytsov 14y ago

Please see my response to Make comparison: http://news.ycombinator.com/item?id=5111527 http://news.ycombinator.com/item?id=5111527 I suspect most of the points I made would be applicable to redo as well, if not more so. Trivial things don't require Drake. Heck, they often times don't require Make as well - just put it in a linear shell script if the steps are not too expensive. It's when things are getting complicated you need something like Drake.

moonboots 14y ago
Redo lacks features baked into Drake, especially the Hadoop integration, but I believe it would be easier to incorporate custom functionality into redo versus hacking Make or writing a custom build system. I haven't used Drake, so I would be interested in a small but complicated Drake script which tackles an intractable problem in Make. I don't claim redo can provide a cleaner solution than a purpose-built system, but I think it will be unexpectedly simple.
- aboytsov 14y ago
  The most crucial thing that Make lacks is multiple outputs and precise control over execution. When you're debugging/developing a large and expensive workflow, you absolutely must have the ability to say things like: - run only this step, I'm debugging it - I've changed implementation of this step, re-build it and everything that depends on it - build everything except this branch, it's expensive and I don't need to rebuild it that often (example: model training) Other examples of intractable problems in Make would be timestamped dependency resolution between local and HDFS files. If Make can't look at HDFS, it can't say if the step needs to be built or not. I don't think you can fix it with external commands. But generally, search for intractable problems is a futile one. Remember, everything you can code in Java, you can code in a Turing machine. :)
  - lars512 14y ago
    Make can certainly generate multiple outputs, and can trivially be coerced to redo any step you like. Provided you add your code as a dependency in the analysis, then it will happily redo only what's changed, giving you nice tight iterations. I think it's real limitations are with multi-machine setups, as in the HDFS problem you're mentioning. Then you need a new tool.
    - aboytsov 14y ago
      Sorry, I might be very ignorant of make - could you please give me a command to re-build a particular target and everything that depends on it?
      - lars512 14y ago
        So make's default behaviour "make somefile.csv" is to build the whole tree of dependencies. To force rebuild of everything, run "make -B somefile.csv". It then assumes everything is out of date. To force rebuild of one step, just delete its output or run "touch" on one of its dependencies before running make. Then that step will get redone. I like to have generated data in a separate folder, say "output/" which you can then snapshot, blow away, or do what you like with. Basically though, I keep it separate from data and code inputs.

groby_b 14y ago

Taking off my programmer hat, and putting on my "I just want to get data moved along" hat - drake is much more readable than this. The ideas behind redo are brilliant, but the way to express them in this particular implementation is not so fun.