3 ms·
I'm not talking about software builds, I'm talking about data. You tell the build system all the steps involved in turning your input data into your output data
by rspeer 11y ago
I'm not talking about software builds, I'm talking about data. You tell the build system all the steps involved in turning your input data into your output data. You don't need extra commands to run things in parallel because any good build system can already run things in parallel.
This isn't "poor man's MapReduce" anymore than a car is a poor man's space shuttle. A space shuttle is a difficult and expensive way to get around if you don't need to leave the surface of the Earth. A MapReduce framework is a computationally expensive way to get data around if it doesn't need to leave your computer.
Look at Ninja sometime. It's not a hack.
- srean 11y agoWe are in agreement. I perhaps did not phrase it well enough. Although make can be used to parallelize data workflows (guilty as charged), it was not particularly designed for this use. It has lots of other things baked in that specifically target building software which for the purpose of data work flows are extra cruft. Effective ? yes, quirky? yes, recommended ? perhaps not if one is not very familiar with make. Ninja is lovely its like a library of building blocks to make your own build system. BTW as long as we are trading favorites, may I suggest tup.