7 ms·

Heh, all the bioinformaticians come out of the woodwork :-) Here's yet-another-project for bioinformatics workflows that I've been involved in. This one based

by zmmmmm 14y ago

Heh, all the bioinformaticians come out of the woodwork :-) Here's yet-another-project for bioinformatics workflows that I've been involved in. This one based on Groovy: http://bpipe.org http://bpipe.org I agree with your sentiments about the nature of pipelines vs build system a la make. Many many people start down the path of putting the classic DAG dependency analysis as the foundation of their needs when in fact, this isn't so much of a problem in real situations, and is even somewhat counterproductive because it forces you to declare a lot of things in a static way that actually aren't static at all. I've found tools like this completely break down when your data starts determining your workflow (eg: if the file is bigger than X I will break it in n parts and run them in parallel, otherwise I will continue on and do it using a different command entirely in memory). In my experience the problems in big data analysis are more about the complexity of managing the process, achieving as much parallelization with as little effort and craziness as possible (don't see any mention of that in Drake), documenting what actually happened when something ran so you can figure it out later, and most of all, flexibility in modifying it since it changes every day of the week. One mistake that Drake appears to make (again, from my quick skim), is interweaving the declaration of the "stages" of the pipeline (what they do) and the dependencies between them (the order they run in). This makes your pipeline stages less reusable and the pipeline harder to maintain. Bpipe completely separates these things out, which is something I like about it.

aboytsov 14y ago
Thanks for your feedback. We do mention parallelization in the designdoc, it's just not implemented yet. It's quite easy to add though. We have a lot of features spec'ed out, but not implemented. I would appreciate if you elaborated on separating step definitions from dependency definitions. In my mind, they are the same thing. If you mean that steps might not be connected by input-output relationship, but still have dependencies, Drake fully supports that via tags. If you mean that steps might be connected through input-output files, but not depend upon each other, I don't frankly see how it's possible. And if you mean some other syntax which more clearly separates the two, Drake supports methods which achieves exactly that. If you mean something else, I would love to see an example. Thanks!
- zmmmmm 14y ago
  > I would appreciate if you elaborated on separating step definitions from dependency definitions As I said, I only very quickly skimmed since I'm busy, I might have overlooked information, and apologies in that case. But take the example from the front page: evergreens.csv <- contracts.csv grep Evergreen $INPUT > $OUTPUT So now suppose a new requirement comes along - Evergreen is also called "Neverbrown" sometimes. It's decided the best way is to convert all references at input so nothing else gets confused downstream. So I need an extra step, now renamed.csv <- contracts.csv: sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT evergreens.csv <- renamed.csv: grep Evergreen $INPUT > $OUTPUT Adding this step forced me to modify the declaration of the original command, even though what I added had nothing to do with that command. With Bpipe, for example, you say extract_evergreens = { exec "grep Evergreen $input > $output" } fix_names = { exec "sed 's/Neverbrown/Evergreen/g' $input > $output" } Then you define your pipeline order separately - run { fix_names + extract_evergreens } If I get contracts from a different source that don't need the renaming, I can still run my old version and I'm not changing the definition of anything: run { extract_evergreens } Hope this explains what I mean, and again apologies if this is all clearly explained in your docs and I just jumped to conclusions from the simple examples!
  - aboytsov 14y ago
    I see. Thank you very much. I think this is very cool. I can see several problems with this approach, and I would greatly appreciate it if you could comment on that. After all, I don't know Bpipe. The fundamental issue is why do you have to repeat the filename, and I did give it some thought. 1. What your example does is allows to allocate dependencies based on positions. It's pretty cool. This seems to be easily reproducible in Drake, if we add a special symbol that would just mean "a temporary file" for the output, and "last temporary output" for the input (by the way, you don't need colons): _ <- contracts.csv sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT evergreens.csv <- _ grep Evergreen $INPUT > $OUTPUT or even: <<- contracts.csv sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT evergreens.csv <<- grep Evergreen $INPUT > $OUTPUT 2. One of the problems, as you can see, that it only works if you don't care about the filenames, i.e. you use a temporary file. Similarly, your Bpipe expression: run { fix_names + extract_evergreens } doesn't care about filenames as well. How do you add it there? What if you need this file for debugging purposes, or if it's an input to some further step down the road? In this case, you'd have to do what you want to avoid doing (i.e. modify the original step). 3. I'm even more concerned with multiple inputs and multiple outputs. As long as your workflow is simple, you can get away with a + b. But when it's more complicated, you would have to do something like: run { (((fix_names + extract_evergreens) * and_some_otheroutput) + some_other_step) * some_other_output } (I used * as an operator that puts two outputs together to create an input with two files for the next command. Mathematically, + is better for that and * is for what + is used in your examples. :)) As you can see, it gets unreadable so fast, that you'd want to use some sort of identifiers to specify dependencies, and would end up with a scheme pretty much equivalent to filenames. The fact that some file might be a temporary is a related, but parallel problem. 4. Now even worse, I'm not quite sure how this syntax could accomodate multiple outputs. If fix_name creates several outputs, and extract_evergreens uses only one, you can't get around it without some weird syntax and specifying a numeric position. It also gets out of hand pretty quickly and you're back to using some sort of identifiers, be it filenames or not. 5. Speaking of identifiers, you can use variables in Drake instead of filenames, so you can abstract filenames away. But it seems to me there's a more fundamental problem in play. 6. If you're concerned with coupling implementation and input and output names, Drake has methods for this: fix_names() sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT extract_evergreens() grep Evergreen $INPUT > $OUTPUT renamed.csv <- contracts.csv [method:fix_names] evergreens.csv <- renamed.csv [method:extract_evergreens] or even, as discussed above: <<- contracts.csv [method:fix_names] evergreens.csv <<- [method:extract_evergreens] To summarize, I think your example is cool, but it seems to only be practical for rather simple workflows. And I can also see how Drake can easily be extended to support such syntactic sugar. For more complicated dependencies though, I don't really see a better approach. I would love to hear your further thoughts on the matter, and whether you'd like to see something similar to what I proposed in Drake. Or something else. Artem.

by zmmmmm 14y ago

aboytsov 14y ago

Thanks for your feedback. We do mention parallelization in the designdoc, it's just not implemented yet. It's quite easy to add though. We have a lot of features spec'ed out, but not implemented. I would appreciate if you elaborated on separating step definitions from dependency definitions. In my mind, they are the same thing. If you mean that steps might not be connected by input-output relationship, but still have dependencies, Drake fully supports that via tags. If you mean that steps might be connected through input-output files, but not depend upon each other, I don't frankly see how it's possible. And if you mean some other syntax which more clearly separates the two, Drake supports methods which achieves exactly that. If you mean something else, I would love to see an example. Thanks!

zmmmmm 14y ago
> I would appreciate if you elaborated on separating step definitions from dependency definitions As I said, I only very quickly skimmed since I'm busy, I might have overlooked information, and apologies in that case. But take the example from the front page: evergreens.csv <- contracts.csv grep Evergreen $INPUT > $OUTPUT So now suppose a new requirement comes along - Evergreen is also called "Neverbrown" sometimes. It's decided the best way is to convert all references at input so nothing else gets confused downstream. So I need an extra step, now renamed.csv <- contracts.csv: sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT evergreens.csv <- renamed.csv: grep Evergreen $INPUT > $OUTPUT Adding this step forced me to modify the declaration of the original command, even though what I added had nothing to do with that command. With Bpipe, for example, you say extract_evergreens = { exec "grep Evergreen $input > $output" } fix_names = { exec "sed 's/Neverbrown/Evergreen/g' $input > $output" } Then you define your pipeline order separately - run { fix_names + extract_evergreens } If I get contracts from a different source that don't need the renaming, I can still run my old version and I'm not changing the definition of anything: run { extract_evergreens } Hope this explains what I mean, and again apologies if this is all clearly explained in your docs and I just jumped to conclusions from the simple examples!
- aboytsov 14y ago
  I see. Thank you very much. I think this is very cool. I can see several problems with this approach, and I would greatly appreciate it if you could comment on that. After all, I don't know Bpipe. The fundamental issue is why do you have to repeat the filename, and I did give it some thought. 1. What your example does is allows to allocate dependencies based on positions. It's pretty cool. This seems to be easily reproducible in Drake, if we add a special symbol that would just mean "a temporary file" for the output, and "last temporary output" for the input (by the way, you don't need colons): _ <- contracts.csv sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT evergreens.csv <- _ grep Evergreen $INPUT > $OUTPUT or even: <<- contracts.csv sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT evergreens.csv <<- grep Evergreen $INPUT > $OUTPUT 2. One of the problems, as you can see, that it only works if you don't care about the filenames, i.e. you use a temporary file. Similarly, your Bpipe expression: run { fix_names + extract_evergreens } doesn't care about filenames as well. How do you add it there? What if you need this file for debugging purposes, or if it's an input to some further step down the road? In this case, you'd have to do what you want to avoid doing (i.e. modify the original step). 3. I'm even more concerned with multiple inputs and multiple outputs. As long as your workflow is simple, you can get away with a + b. But when it's more complicated, you would have to do something like: run { (((fix_names + extract_evergreens) * and_some_otheroutput) + some_other_step) * some_other_output } (I used * as an operator that puts two outputs together to create an input with two files for the next command. Mathematically, + is better for that and * is for what + is used in your examples. :)) As you can see, it gets unreadable so fast, that you'd want to use some sort of identifiers to specify dependencies, and would end up with a scheme pretty much equivalent to filenames. The fact that some file might be a temporary is a related, but parallel problem. 4. Now even worse, I'm not quite sure how this syntax could accomodate multiple outputs. If fix_name creates several outputs, and extract_evergreens uses only one, you can't get around it without some weird syntax and specifying a numeric position. It also gets out of hand pretty quickly and you're back to using some sort of identifiers, be it filenames or not. 5. Speaking of identifiers, you can use variables in Drake instead of filenames, so you can abstract filenames away. But it seems to me there's a more fundamental problem in play. 6. If you're concerned with coupling implementation and input and output names, Drake has methods for this: fix_names() sed 's/Neverbrown/Evergreen/g' $INPUT > $OUTPUT extract_evergreens() grep Evergreen $INPUT > $OUTPUT renamed.csv <- contracts.csv [method:fix_names] evergreens.csv <- renamed.csv [method:extract_evergreens] or even, as discussed above: <<- contracts.csv [method:fix_names] evergreens.csv <<- [method:extract_evergreens] To summarize, I think your example is cool, but it seems to only be practical for rather simple workflows. And I can also see how Drake can easily be extended to support such syntactic sugar. For more complicated dependencies though, I don't really see a better approach. I would love to hear your further thoughts on the matter, and whether you'd like to see something similar to what I proposed in Drake. Or something else. Artem.