2 ms·
Sorry, I didn't mean to imply "you're doing it wrong". Didn't even realize you made the tool. Oops. Personally, if large chunks of my output are out of date,
by tedunangst 14y ago
Sorry, I didn't mean to imply "you're doing it wrong". Didn't even realize you made the tool. Oops. Personally, if large chunks of my output are out of date, I don't like the idea of commingling them with new stuff, but obviously I don't know a whole lot about what you're doing.
- aboytsov 14y agoThanks. I think you're missing the point. Imagine a big, complicated data workflow, like the one the diagram for which I showed at my video (real-life workflow): http://www.youtube.com/watch?feature=player_detailpage&v=BUgxmvpuKAs#t=1016s http://www.youtube.com/watch?feature=player_detailpage&v... Now imagine you're not the only one working on it. You may have even never run it in its entirety, since it takes 10 hours. Imagine there's a branch which you, a developer, is currently working on. This branch depends on some other files in the workflow. Let's say, generate synonyms from the sentence dataset. Or, some complicated cleaning of some intermediate data. This is not a small task and you will spend a couple of days doing it, re-running your code dozens of times in the process. You don't care about other parts of the workflow. You only care about what you're developing and how it propagates. Does it propagate? Does it break something down the road? What is the final output? Did all this synonym collection help? Did the changes you made in learning code improve the results? When you're done, you may commit your code and somewhere else somebody will build a nice new dataset, but while you're working on it, you really need to be able to run any target individually, with dependencies or without, as well as forcibly rebuild all steps down the tree to see the final result. Makes sense?
- tedunangst 14y agoThanks.