3 ms·

I like the idea that the tasks can be implemented in any language, but I feel like this has limitations compared to something like Rake, where the step definiti

I like the idea that the tasks can be implemented in any language, but I feel like this has limitations compared to something like Rake, where the step definition is code, too. What this means is that in Rake I am not just limited to defining new task bodies, but new ways of defining tasks themselves. I see that Drake is implemented in Clojure, so I'd imagine you understand the value of homoiconicity and extensible languages. So I wonder why you didn't just use Clojure all the way through?

aboytsov 14y ago
This is a great question. Our approach to this is described here: http://www.youtube.com/watch?feature=player_detailpage&v=BUgxmvpuKAs#t=2393s http://www.youtube.com/watch?feature=player_detailpage&v... In short, we don't feel like it's an either or question. We want to have Drake as a command-line frontend to the core functionality, but we would love to see/have other frontends developed as well. Currently, there's no Clojure DSL for Drake, but I think it'd be totally awesome. The reason we started from command-line is because our workflows are heterogenous, and we also didn't want to limit Drake to developers and associate it with coding. Clojure can be quite a big learning curve if you only need it to specify steps and link them together through file dependencies. We had an important design goal in mind: Drake should be as simple as writing a shell script. If it's not, our experience shows that most workflow start as trivial shell-scripts with one or two steps, and by the time it grows into something unmanageable, it's kinda too late. :) On a related note, Drake supports Clojure code inlining for manipulation of the parse tree. It's not an equivalent, just a somewhat related feature. It allows you to modify the steps, dependencies, and anything else in the parse tree directly from Clojure.
jboggan 14y ago
I'm glad the step definitions are not in Clojure or a unified programming language. It makes it much easier to pull in data specialists, product managers, and other non-engineers to help build and maintain a data workflow while leaving them the autonomy to run and troubleshoot the steps of the build specific to their skillsets.

aboytsov 14y ago

This is a great question. Our approach to this is described here: http://www.youtube.com/watch?feature=player_detailpage&v=BUgxmvpuKAs#t=2393s http://www.youtube.com/watch?feature=player_detailpage&v... In short, we don't feel like it's an either or question. We want to have Drake as a command-line frontend to the core functionality, but we would love to see/have other frontends developed as well. Currently, there's no Clojure DSL for Drake, but I think it'd be totally awesome. The reason we started from command-line is because our workflows are heterogenous, and we also didn't want to limit Drake to developers and associate it with coding. Clojure can be quite a big learning curve if you only need it to specify steps and link them together through file dependencies. We had an important design goal in mind: Drake should be as simple as writing a shell script. If it's not, our experience shows that most workflow start as trivial shell-scripts with one or two steps, and by the time it grows into something unmanageable, it's kinda too late. :) On a related note, Drake supports Clojure code inlining for manipulation of the parse tree. It's not an equivalent, just a somewhat related feature. It allows you to modify the steps, dependencies, and anything else in the parse tree directly from Clojure.

jboggan 14y ago

I'm glad the step definitions are not in Clojure or a unified programming language. It makes it much easier to pull in data specialists, product managers, and other non-engineers to help build and maintain a data workflow while leaving them the autonomy to run and troubleshoot the steps of the build specific to their skillsets.