5 ms·

The attempt to cleanly separate out workflow is nice in theory but an extremely leaky abstraction when it comes to real world operations for DAGs larger than 10

by prpl 7y ago

The attempt to cleanly separate out workflow is nice in theory but an extremely leaky abstraction when it comes to real world operations for DAGs larger than 10s of nodes. You will end up with lots of code that will duplicate functionality that exists in a workflow engine just to have something common (e.g. notifications, database ingest, etc...) There are no operational concerns in CWL.

deleted 7y ago
[deleted]
mbreese 7y ago
That’s my main issue with CWL. It’s a nice execution language, but not something I like to write an analysis pipeline in. Because it is YAML, using runtime logic is difficult (impossible?). I much rather use a pipeline tool that includes some flow control and ability to adapt to different arguments. Last I looked at CWL, this wasn’t the case.
- tetron 7y ago
  Adding conditional execution to the dataflow graph is something being worked on for the next version of CWL.
  - mbreese 7y ago
    After seeing the horror that was conditionals and flow control in Ant's build.xml format, I don't see anything based on a text file format working well...