4 ms·

No, make -B mytarget rebuilds either mytarget only or mytarget and everything mytarget depends on. A more common scenario is when you need to rebuild mytarget a

No, make -B mytarget rebuilds either mytarget only or mytarget and everything mytarget depends on. A more common scenario is when you need to rebuild mytarget and everything that depends on it. Without rebuilding other parts of the workflow that you don't need.

tedunangst 14y ago
This is a really weird request. make won't rebuild things that haven't changed, so the default make all rule will only rebuild the things depending on mytarget. Every time you change mytarget, just run make (all) and everything that depends on mytarget (and only those things) will be rebuilt.
- aboytsov 14y ago
  Really? This is not a weird request, this is one of the most common things we do when we're developing a workflow. You need to do this every time you make changes to code and you want these changes to propagate. You can't run "make all", because it literally builds everything. You might be working on a specific branch of the workflow, and the overall workflow could be huge. And out-of-date in a lot of places. Or it could contain steps that are very expensive, but not necessary to build for your development purposes (for example, generating a model). This is why exclusions are also important, and make also does not support them. Make also does not support multiple outputs, and I gave you a prooflink before. And a lot of other things which we think are important, too (I could make a list. I did, actually). If you like Make, you should continue using it. I think it is a little arrogant on your part to try to explain to us that we simply wasted our time. We built the tool to address the problems we were facing. If you do not face similar problems, by all means, use Make.
  - tedunangst 14y ago
    Sorry, I didn't mean to imply "you're doing it wrong". Didn't even realize you made the tool. Oops. Personally, if large chunks of my output are out of date, I don't like the idea of commingling them with new stuff, but obviously I don't know a whole lot about what you're doing.
    - aboytsov 14y ago
      Thanks. I think you're missing the point. Imagine a big, complicated data workflow, like the one the diagram for which I showed at my video (real-life workflow): http://www.youtube.com/watch?feature=player_detailpage&v=BUgxmvpuKAs#t=1016s http://www.youtube.com/watch?feature=player_detailpage&v... Now imagine you're not the only one working on it. You may have even never run it in its entirety, since it takes 10 hours. Imagine there's a branch which you, a developer, is currently working on. This branch depends on some other files in the workflow. Let's say, generate synonyms from the sentence dataset. Or, some complicated cleaning of some intermediate data. This is not a small task and you will spend a couple of days doing it, re-running your code dozens of times in the process. You don't care about other parts of the workflow. You only care about what you're developing and how it propagates. Does it propagate? Does it break something down the road? What is the final output? Did all this synonym collection help? Did the changes you made in learning code improve the results? When you're done, you may commit your code and somewhere else somebody will build a nice new dataset, but while you're working on it, you really need to be able to run any target individually, with dependencies or without, as well as forcibly rebuild all steps down the tree to see the final result. Makes sense?
      - tedunangst 14y ago
        Thanks.
lars512 14y ago
This is basically the case where you don't (or it's infeasible to) capture the dependences fully, so you want to rebuild everything from target onwards after some change. In that case: "rm target; make target"
- aboytsov 14y ago
  "make target" will not make stuff down the tree. It will stop at target. Please, try again. :)
  - lars512 14y ago
    Haha, well pointed out. I'm clearly having trouble parsing today. "rm target; make" can work, but only if you're using a pattern for data pipelines where there is only one default set of downstream targets. If the one Makefile supports a range of downstream targets, then this won't work. I concede, make doesn't support that operation out of the box :)
    - aboytsov 14y ago
      Phew. Thank you.

tedunangst 14y ago

This is a really weird request. make won't rebuild things that haven't changed, so the default make all rule will only rebuild the things depending on mytarget. Every time you change mytarget, just run make (all) and everything that depends on mytarget (and only those things) will be rebuilt.

aboytsov 14y ago
Really? This is not a weird request, this is one of the most common things we do when we're developing a workflow. You need to do this every time you make changes to code and you want these changes to propagate. You can't run "make all", because it literally builds everything. You might be working on a specific branch of the workflow, and the overall workflow could be huge. And out-of-date in a lot of places. Or it could contain steps that are very expensive, but not necessary to build for your development purposes (for example, generating a model). This is why exclusions are also important, and make also does not support them. Make also does not support multiple outputs, and I gave you a prooflink before. And a lot of other things which we think are important, too (I could make a list. I did, actually). If you like Make, you should continue using it. I think it is a little arrogant on your part to try to explain to us that we simply wasted our time. We built the tool to address the problems we were facing. If you do not face similar problems, by all means, use Make.
- tedunangst 14y ago
  Sorry, I didn't mean to imply "you're doing it wrong". Didn't even realize you made the tool. Oops. Personally, if large chunks of my output are out of date, I don't like the idea of commingling them with new stuff, but obviously I don't know a whole lot about what you're doing.
  - aboytsov 14y ago
    Thanks. I think you're missing the point. Imagine a big, complicated data workflow, like the one the diagram for which I showed at my video (real-life workflow): http://www.youtube.com/watch?feature=player_detailpage&v=BUgxmvpuKAs#t=1016s http://www.youtube.com/watch?feature=player_detailpage&v... Now imagine you're not the only one working on it. You may have even never run it in its entirety, since it takes 10 hours. Imagine there's a branch which you, a developer, is currently working on. This branch depends on some other files in the workflow. Let's say, generate synonyms from the sentence dataset. Or, some complicated cleaning of some intermediate data. This is not a small task and you will spend a couple of days doing it, re-running your code dozens of times in the process. You don't care about other parts of the workflow. You only care about what you're developing and how it propagates. Does it propagate? Does it break something down the road? What is the final output? Did all this synonym collection help? Did the changes you made in learning code improve the results? When you're done, you may commit your code and somewhere else somebody will build a nice new dataset, but while you're working on it, you really need to be able to run any target individually, with dependencies or without, as well as forcibly rebuild all steps down the tree to see the final result. Makes sense?
    - tedunangst 14y ago
      Thanks.

lars512 14y ago

This is basically the case where you don't (or it's infeasible to) capture the dependences fully, so you want to rebuild everything from target onwards after some change. In that case: "rm target; make target"

aboytsov 14y ago
"make target" will not make stuff down the tree. It will stop at target. Please, try again. :)
- lars512 14y ago
  Haha, well pointed out. I'm clearly having trouble parsing today. "rm target; make" can work, but only if you're using a pattern for data pipelines where there is only one default set of downstream targets. If the one Makefile supports a range of downstream targets, then this won't work. I concede, make doesn't support that operation out of the box :)
  - aboytsov 14y ago
    Phew. Thank you.