4 ms·

It looks like all of the drakefiles could be replaced pretty trivially with Makefiles. Replacing '<-' with ':', ';' with '#', and '$INPUT', '$OUTPUT' with '$<'

by ori_b 14y ago

It looks like all of the drakefiles could be replaced pretty trivially with Makefiles. Replacing '<-' with ':', ';' with '#', and '$INPUT', '$OUTPUT' with '$<' and '$@', and inserting shell invocations of the Python interpreter looks like it would do the job. The major differences I see are: - Inline support for Python et al. - Confirming the steps that will be taken. - HDFS support. Are there any other big differences?

dirtyvagabond 14y ago
Make was a major inspiration for us, and so Drake definitely has similarities to Make. The differences you list were non-trivial to us in usefulness, but of course YMMV. Also, there are a lot of (possibly) interesting future features described in the spec. https://docs.google.com/document/d/1bF-OKNLIG10v_lMes_m4yyaJtAaJKtdK0Jizvi_MNsg/edit# https://docs.google.com/document/d/1bF-OKNLIG10v_lMes_m4yyaJ...
JoshTriplett 14y ago
Make can support Python, or any other language you'd like. Just set ONESHELL to avoid splitting commands by line, and then set SHELL to your preferred language interpreter. Make will then hand that interpreter the entire body of commands to rebuild a target.
- aboytsov 14y ago
  Drake supports "protocol" abstraction, which is much more than just specifying an interpreter. Python is a trivial protocol, not much more complicated than shell. There are slightly more complicated protocols, for example, "eval", which runs the first line as a shell command before putting everything else in $CMDS environment variable. There could be protocols for running an HBase query, a Pig query, Cascalog query, or an SQL query. Some of these things could involve building a JAR file and giving it to Hadoop binary. Currently only a handful of protocols is implemented, but more are described in the spec.
aboytsov 14y ago
The example in the blogpost is understandably trivial, and it can be implemented in almost any Make-like system. The concept of Make is not unique. Everything that has dependencies and executes steps is similar to Make in concept. Drake is no exception, and it can be replaced with Make, but no more so than Rake, Ant or Maven can be replaced by Make. That is, if it's trivial - yes. Just a bit more complicated - no. Some things are merely painful to implement with Make, some are just impossible: - multiple outputs - no-input and no-output steps - HDFS support - Hadoop's partial files support (part-?????) - forced execution of any subbranch, up or down the tree or any individual targets (crucial for debugging and development) - target exclusions - protocol abstraction - inline Python is just one example - tags - branching - methods These are just what's implemented already. Other things are planned such as: - automated data versioning (backup and revert) - parallelization - real-time status console - retries, email notifications - etc. Requirements for building executables and working with large, complicated and expensive data workflows are quite visible different, and the most important thing about Drake is that it provides the platform for convenient features (such as versioning or email notifications) to be implemented. And once they are, every data workflow can take advantage of them. I guess, if Make was really, really extendable, we could have considered it as a platform for all this. But it's not, and hacking all of that into Make's source code in C would be, I'm sure, a much greater pain than writing Drake. Artem.
- blablabla123 14y ago
  retries and email notifications is a good one. Currently I do something similar with cronjobs, rsync, shell scripts and some custom tools -- on multiple boxes. (Email notification with mailx) Works in theory pretty well, in practice race conditions become a problem, making it sometimes annoying because I need to run things manually when I need up to date processed data. If I had retries, this would be an improvement.
  - aboytsov 14y ago
    Got ya. Please voice your opinion about the priority in which features should be implemented by submitting a feature request at https://github.com/Factual/drake/issues https://github.com/Factual/drake/issues, or +1'ing an existing one. There are so many potential features to be added to Drake, and a lot of them have already been thought about and spec'ed out, that we need some sort of a way to figure out what to do first. Of course, if you'd like to actively contribute, we'd be ecstatic. Artem.
andrewflnr 14y ago
Does it have to have big differences? It's a slightly nicer system with a fairly shallow initial learning curve. If you're on a new project, what's the problem? I'm wondering how well it would work as an actual make replacement.

by ori_b 14y ago

dirtyvagabond 14y ago

Make was a major inspiration for us, and so Drake definitely has similarities to Make. The differences you list were non-trivial to us in usefulness, but of course YMMV. Also, there are a lot of (possibly) interesting future features described in the spec. https://docs.google.com/document/d/1bF-OKNLIG10v_lMes_m4yyaJtAaJKtdK0Jizvi_MNsg/edit# https://docs.google.com/document/d/1bF-OKNLIG10v_lMes_m4yyaJ...

JoshTriplett 14y ago

Make can support Python, or any other language you'd like. Just set ONESHELL to avoid splitting commands by line, and then set SHELL to your preferred language interpreter. Make will then hand that interpreter the entire body of commands to rebuild a target.

aboytsov 14y ago
Drake supports "protocol" abstraction, which is much more than just specifying an interpreter. Python is a trivial protocol, not much more complicated than shell. There are slightly more complicated protocols, for example, "eval", which runs the first line as a shell command before putting everything else in $CMDS environment variable. There could be protocols for running an HBase query, a Pig query, Cascalog query, or an SQL query. Some of these things could involve building a JAR file and giving it to Hadoop binary. Currently only a handful of protocols is implemented, but more are described in the spec.

aboytsov 14y ago

The example in the blogpost is understandably trivial, and it can be implemented in almost any Make-like system. The concept of Make is not unique. Everything that has dependencies and executes steps is similar to Make in concept. Drake is no exception, and it can be replaced with Make, but no more so than Rake, Ant or Maven can be replaced by Make. That is, if it's trivial - yes. Just a bit more complicated - no. Some things are merely painful to implement with Make, some are just impossible: - multiple outputs - no-input and no-output steps - HDFS support - Hadoop's partial files support (part-?????) - forced execution of any subbranch, up or down the tree or any individual targets (crucial for debugging and development) - target exclusions - protocol abstraction - inline Python is just one example - tags - branching - methods These are just what's implemented already. Other things are planned such as: - automated data versioning (backup and revert) - parallelization - real-time status console - retries, email notifications - etc. Requirements for building executables and working with large, complicated and expensive data workflows are quite visible different, and the most important thing about Drake is that it provides the platform for convenient features (such as versioning or email notifications) to be implemented. And once they are, every data workflow can take advantage of them. I guess, if Make was really, really extendable, we could have considered it as a platform for all this. But it's not, and hacking all of that into Make's source code in C would be, I'm sure, a much greater pain than writing Drake. Artem.

blablabla123 14y ago
retries and email notifications is a good one. Currently I do something similar with cronjobs, rsync, shell scripts and some custom tools -- on multiple boxes. (Email notification with mailx) Works in theory pretty well, in practice race conditions become a problem, making it sometimes annoying because I need to run things manually when I need up to date processed data. If I had retries, this would be an improvement.
- aboytsov 14y ago
  Got ya. Please voice your opinion about the priority in which features should be implemented by submitting a feature request at https://github.com/Factual/drake/issues https://github.com/Factual/drake/issues, or +1'ing an existing one. There are so many potential features to be added to Drake, and a lot of them have already been thought about and spec'ed out, that we need some sort of a way to figure out what to do first. Of course, if you'd like to actively contribute, we'd be ecstatic. Artem.

andrewflnr 14y ago

Does it have to have big differences? It's a slightly nicer system with a fairly shallow initial learning curve. If you're on a new project, what's the problem? I'm wondering how well it would work as an actual make replacement.