3 ms·

GNU parallel would be one step in such a build, right? I tend to face problems more difficult than "run exactly the same code in N places". Perhaps I should kno

by rspeer 11y ago

GNU parallel would be one step in such a build, right? I tend to face problems more difficult than "run exactly the same code in N places". Perhaps I should know more about GNU parallel but it seems like it would actually be kind of hard to fit into a larger build. This is not to say that there's a problem with GNU parallel, it just seems like it's designed for the "embarrassingly parallel" case with no dependencies between steps, whereas Make and the system under discussion are for the case where you have dependencies between steps. I've also never found myself needing to use xargs in a Make or Ninja build, incidentally.

srean 11y ago
Correct. I was talking about general parallel workflows, not software builds in particular. In any case I would call these as hacks rather than solid solutions. Xargs us great for chunking up arguments and submitting them to a parallel set of consumers: Poor man's mapreduce if you will
- rspeer 11y ago
  I'm not talking about software builds, I'm talking about data. You tell the build system all the steps involved in turning your input data into your output data. You don't need extra commands to run things in parallel because any good build system can already run things in parallel. This isn't "poor man's MapReduce" anymore than a car is a poor man's space shuttle. A space shuttle is a difficult and expensive way to get around if you don't need to leave the surface of the Earth. A MapReduce framework is a computationally expensive way to get data around if it doesn't need to leave your computer. Look at Ninja sometime. It's not a hack.
  - srean 11y ago
    We are in agreement. I perhaps did not phrase it well enough. Although make can be used to parallelize data workflows (guilty as charged), it was not particularly designed for this use. It has lots of other things baked in that specifically target building software which for the purpose of data work flows are extra cruft. Effective ? yes, quirky? yes, recommended ? perhaps not if one is not very familiar with make. Ninja is lovely its like a library of building blocks to make your own build system. BTW as long as we are trading favorites, may I suggest tup.

by rspeer 11y ago

srean 11y ago

Correct. I was talking about general parallel workflows, not software builds in particular. In any case I would call these as hacks rather than solid solutions. Xargs us great for chunking up arguments and submitting them to a parallel set of consumers: Poor man's mapreduce if you will

rspeer 11y ago
I'm not talking about software builds, I'm talking about data. You tell the build system all the steps involved in turning your input data into your output data. You don't need extra commands to run things in parallel because any good build system can already run things in parallel. This isn't "poor man's MapReduce" anymore than a car is a poor man's space shuttle. A space shuttle is a difficult and expensive way to get around if you don't need to leave the surface of the Earth. A MapReduce framework is a computationally expensive way to get data around if it doesn't need to leave your computer. Look at Ninja sometime. It's not a hack.
- srean 11y ago
  We are in agreement. I perhaps did not phrase it well enough. Although make can be used to parallelize data workflows (guilty as charged), it was not particularly designed for this use. It has lots of other things baked in that specifically target building software which for the purpose of data work flows are extra cruft. Effective ? yes, quirky? yes, recommended ? perhaps not if one is not very familiar with make. Ninja is lovely its like a library of building blocks to make your own build system. BTW as long as we are trading favorites, may I suggest tup.