7 ms·

Common Workflow Language

68 points by vyuh 7y ago

fowl2 7y ago
"Workflow" is such a loaded/broad/bifurcated term. In this case we seem to be talking about "Scientific Computing Workflow" or some sort of data pipeline. (As opposed to some sort of "Enterprise" workflow)
- oeuviz 7y ago
  Fully agree. It seems nobody understands it in the sense of Business Processes/Workflow modelling anymore and it is arbitrary used for "something to do".
  - liotier 7y ago
    I did expect business process modeling... I was confused !
- lewisjoe 7y ago
  Yes, "Workflow" has to be one of the most abused word in the industry. People seem to use workflow for all of these things: #1 Business process modeling and automating platforms - Camunda, BPMN, jBPM etc. #2 Assembling together software functions (also called tasks, sometimes) for getting an output data - Airflow, Cadence, CWL is related too. #3 Moving one entity (bug, document, deal) through different stages of a state machine, until the entity reaches an end state - Sharepoint workflow, Zoho Orchestly, Comala workflows #4 Orchestrating several apps together for automating something, usually as reactions for events - Zapier When developers talk about workflows, it's mostly about #2. When business people talk about workflows it's mostly about #1. When managers/leads of a team talk about workflows it's mostly about #3. We should accept that Workflow as a word is too ambiguous, and come up with what kind of workflow category the tool belongs to, before branding it as an "one stop solution for all things workflow".
  - zekrioca 7y ago
    Couldn't agree more. I do work with workflows (mostly #2 and #4), but in the beginning of my journey I had problems understanding what researchers were actually trying to tell me, and I always got confused. I suspect that they themselves do not fully understand how ambiguous this word can be..
- hhas01 7y ago
  “we seem to be talking about "Scientific Computing Workflow" or some sort of data pipeline” So, more along the lines of https://en.wikipedia.org/wiki/Dataflow_programming https://en.wikipedia.org/wiki/Dataflow_programming. A good problem space to bring declarative/functional/logic/concatenative concepts and tools, IMO, as opposed to the ubiquitous “hit it with a rock” of the usual C* suspects. Be great to see real progress here, but my bullshit detector is immediately dialed to MAX. The great problem with “workflow systems” is that they very easily disappear up their own ass: “Potemkin solutions” that solve every problem except the customer’s own. (See: https://thedailywtf.com/articles/The_Inner-Platform_Effect https://thedailywtf.com/articles/The_Inner-Platform_Effect). “Enterprise workflows” in particular I think are just a way for indolent code monkeys to shirk learning how the business works for themselves. Looks all very impressive without ever solving shit. .. FWIW I do think Alan Kay was onto something with his Nile strategy (https://news.ycombinator.com/item?id=10535364 https://news.ycombinator.com/item?id=10535364). Pity there wasn’t followthrough (bit of a mayfly, Kay). Ditto Iverson and APL, although APL’s sidelining was more down to a nascent Programming Profession redefining Programming™ to be what they do for users rather than what users (mathematicians in this case) do for themselves.
  - tetron 7y ago
    CWL is a declarative/functional language for describing how to execute command line tools (staging input files, mapping arguments, collecting outputs) and how to connect the outputs of one tool to the inputs of the next. It is HPC and cloud agnostic. The same workflow description can run on a laptop or on 1000 cloud compute nodes. Lots of people are already using it to solve their problems, for some examples see https://github.com/search?q=extension%3Acwl+cwlVersion https://github.com/search?q=extension%3Acwl+cwlVersion
    - hhas01 7y ago
      Quickly skimming the CWL User Guide, calling that a “language” is a huge stretch. Looks like writing a bunch of YAML config files which let you write more YAML files to construct a UNIX pipeline and run it. In which case, why didn’t you just use bash? Yeah, I know bash stinks. But this is not my fist rodeo* so I’m struggling here to see how CWL stinks less. See again: Inner-Platform Effect, Greenspun’s Tenth Rule. -- * i.e. I already know how easily “workflow engines” go up their own arse because I’ve done it myself. And CWL fails the same sniff test. Not a good start.
eiro 7y ago
it really feels like a sophisticated bad joke to me but i'm curious: any real world example ?
- jkh1 7y ago
  It's used in bioinformatics. for example by the Global Alliance for Genomics and Health[0]. You can find some CWL workflows on dockstore [1]. [0] https://www.ga4gh.org/ https://www.ga4gh.org/ [1] https://dockstore.org/search?descriptorType=CWL&searchMode=files https://dockstore.org/search?descriptorType=CWL&searchMode=f...
- Xophmeister 7y ago
  We've used it and competing workflow management "languages" -- there are a few of them -- in our bioinformatic processing pipelines. "Sophisticated bad joke" is a pretty accurate description.
  - hhas01 7y ago
    Disappointing, but unsurprising. See also: Greenspun’s Tenth Rule, “Yo Dawg…”.
- tetron 7y ago
  https://github.com/search?q=extension%3Acwl+cwlVersion https://github.com/search?q=extension%3Acwl+cwlVersion
smartmic 7y ago
There are so many alternatives, see https://news.ycombinator.com/item?id=20410848 https://news.ycombinator.com/item?id=20410848 Any information about which language is leading the field?
- jkh1 7y ago
  Which field? In the absence of any constraint, the choice of workflow management system is down to individual preferences. Note that the goal of CWL is to be a standard workflow language not a workflow management system. That is, it is concerned purely by how to precisely describe a workflow, not by how it should be implemented which is the task of a workflow management system. The diversity of workflow management systems starts to be recognized as an obstacle to re-use and reproducibility of scientific workflows. Having different systems implementing a standard language would help get around this issue. Several workflow management systems used in bioinformatics have or are implementing support for CWL.
- maddyboo 7y ago
  https://xkcd.com/927/ https://xkcd.com/927/
  - jkh1 7y ago
    This is not exactly true in this case as before CWL there was no standard, only languages tied to particular implementations.
  - snak 7y ago
    Came to the comments looking for this.
- ibern 7y ago
  Right now I'd say it's in competition with Workflow Description Language (WDL), Nextflow, and maybe Snakemake. CWL seems to be more popular than WDL, as far as I can tell, but Nextflow is probably more popular than both.
  - 7y ago
amir01 7y ago
it was valuable. if you want to practice real time industry projects. click here http://bit.ly/2KqTKiR http://bit.ly/2KqTKiR
- hhas01 7y ago
  Spammy account is spamming. Where’s a `flag` link when you need it.
  - the-dude 7y ago
    You click the timestamp, and the flag button appears.
    - hhas01 7y ago
      Hah, thanks. Obvious brainfart is obvious. Never did figure why family got me one of these: https://i.pinimg.com/originals/5a/a7/95/5aa795b87c41d0f4d595c075042b1716.jpg https://i.pinimg.com/originals/5a/a7/95/5aa795b87c41d0f4d595...
hhas01 7y ago
Those who can, implement solutions. Those who can’t, implement Standards.
scottious 7y ago
Imagine a programming language except it also has to be valid YAML. Yeah. Now sprinkle some JavaScript in there just for good measure. Look no further than their own hand-crafted examples (confusingly called "Episodes") for compelling arguments for why to NOT use Common Workflow Language.
- dang 7y ago
  "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." Your comment almost gets there, but not quite—and then it sinks itself lower by breaking this one: "Don't be snarky." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html Can you please take the guidelines a bit more to heart when posting here? We'd appreciate it.
  - scottious 7y ago
    Sorry
prpl 7y ago
The attempt to cleanly separate out workflow is nice in theory but an extremely leaky abstraction when it comes to real world operations for DAGs larger than 10s of nodes. You will end up with lots of code that will duplicate functionality that exists in a workflow engine just to have something common (e.g. notifications, database ingest, etc...) There are no operational concerns in CWL.
- deleted 7y ago
  [deleted]
- mbreese 7y ago
  That’s my main issue with CWL. It’s a nice execution language, but not something I like to write an analysis pipeline in. Because it is YAML, using runtime logic is difficult (impossible?). I much rather use a pipeline tool that includes some flow control and ability to adapt to different arguments. Last I looked at CWL, this wasn’t the case.
  - tetron 7y ago
    Adding conditional execution to the dataflow graph is something being worked on for the next version of CWL.
    - mbreese 7y ago
      After seeing the horror that was conditionals and flow control in Ant's build.xml format, I don't see anything based on a text file format working well...
tetron 7y ago
CWL project co-founder here. It's funny, I have been trying to start a discussion of CWL on HN for years, obviously someone else had to do it. The goal is to provide a way to describe dataflow processes that is highly portable, auditable, and reproducible. This is incredibly important in research, clinical, and regulatory domains where you need to be able to show how you came up with a result. It's not a general purpose language on purpose, and operational concerns like notifications are the domain of specific implementations (engines). I agree the syntax is horrible (and I designed most of it) but it also makes it easy to write programs that read and write CWL, enabling an ecosystem. For example, here is a transpiled languages that emits CWL: https://github.com/common-workflow-language/cwl-ex https://github.com/common-workflow-language/cwl-ex
- Edmond 7y ago
  something worth looking at: https://en.wikipedia.org/wiki/Business_Process_Execution_Language https://en.wikipedia.org/wiki/Business_Process_Execution_Lan...
hhas01 7y ago
Bless. https://media3.giphy.com/media/49zC0Bm1kbu36/source.gif https://media3.giphy.com/media/49zC0Bm1kbu36/source.gif
PeterStuer 7y ago
But is it SPEM™[1] compliant? /s [1] https://www.omg.org/spec/SPEM/2.0 https://www.omg.org/spec/SPEM/2.0
drocer88 7y ago
Snakemake, CWL, WDL, Nextflow are used in bioinformatics workflows. Just plain old bash seems to work just fine, though. It is simple and gives you a lot of portability to HPC clusters and general purpose cloud. Not hardwiring to a configuration file and not using an, in practice, un-portable workflow language saves you a lot of pain. A truly portable cloud/hpc module for python would be a good solution.
- RocketSyntax 7y ago
  Wondering why airflow and prefect haven't crept into space yet
  - tetron 7y ago
    https://github.com/Barski-lab/cwl-airflow https://github.com/Barski-lab/cwl-airflow
JaDogg 7y ago
https://xkcd.com/927/ https://xkcd.com/927/

68 points by vyuh 7y ago

"Workflow" is such a loaded/broad/bifurcated term. In this case we seem to be talking about "Scientific Computing Workflow" or some sort of data pipeline. (As opposed to some sort of "Enterprise" workflow)

oeuviz 7y ago
Fully agree. It seems nobody understands it in the sense of Business Processes/Workflow modelling anymore and it is arbitrary used for "something to do".
- liotier 7y ago
  I did expect business process modeling... I was confused !
lewisjoe 7y ago
Yes, "Workflow" has to be one of the most abused word in the industry. People seem to use workflow for all of these things: #1 Business process modeling and automating platforms - Camunda, BPMN, jBPM etc. #2 Assembling together software functions (also called tasks, sometimes) for getting an output data - Airflow, Cadence, CWL is related too. #3 Moving one entity (bug, document, deal) through different stages of a state machine, until the entity reaches an end state - Sharepoint workflow, Zoho Orchestly, Comala workflows #4 Orchestrating several apps together for automating something, usually as reactions for events - Zapier When developers talk about workflows, it's mostly about #2. When business people talk about workflows it's mostly about #1. When managers/leads of a team talk about workflows it's mostly about #3. We should accept that Workflow as a word is too ambiguous, and come up with what kind of workflow category the tool belongs to, before branding it as an "one stop solution for all things workflow".
- zekrioca 7y ago
  Couldn't agree more. I do work with workflows (mostly #2 and #4), but in the beginning of my journey I had problems understanding what researchers were actually trying to tell me, and I always got confused. I suspect that they themselves do not fully understand how ambiguous this word can be..
hhas01 7y ago
“we seem to be talking about "Scientific Computing Workflow" or some sort of data pipeline” So, more along the lines of https://en.wikipedia.org/wiki/Dataflow_programming https://en.wikipedia.org/wiki/Dataflow_programming. A good problem space to bring declarative/functional/logic/concatenative concepts and tools, IMO, as opposed to the ubiquitous “hit it with a rock” of the usual C* suspects. Be great to see real progress here, but my bullshit detector is immediately dialed to MAX. The great problem with “workflow systems” is that they very easily disappear up their own ass: “Potemkin solutions” that solve every problem except the customer’s own. (See: https://thedailywtf.com/articles/The_Inner-Platform_Effect https://thedailywtf.com/articles/The_Inner-Platform_Effect). “Enterprise workflows” in particular I think are just a way for indolent code monkeys to shirk learning how the business works for themselves. Looks all very impressive without ever solving shit. .. FWIW I do think Alan Kay was onto something with his Nile strategy (https://news.ycombinator.com/item?id=10535364 https://news.ycombinator.com/item?id=10535364). Pity there wasn’t followthrough (bit of a mayfly, Kay). Ditto Iverson and APL, although APL’s sidelining was more down to a nascent Programming Profession redefining Programming™ to be what they do for users rather than what users (mathematicians in this case) do for themselves.
- tetron 7y ago
  CWL is a declarative/functional language for describing how to execute command line tools (staging input files, mapping arguments, collecting outputs) and how to connect the outputs of one tool to the inputs of the next. It is HPC and cloud agnostic. The same workflow description can run on a laptop or on 1000 cloud compute nodes. Lots of people are already using it to solve their problems, for some examples see https://github.com/search?q=extension%3Acwl+cwlVersion https://github.com/search?q=extension%3Acwl+cwlVersion
  - hhas01 7y ago
    Quickly skimming the CWL User Guide, calling that a “language” is a huge stretch. Looks like writing a bunch of YAML config files which let you write more YAML files to construct a UNIX pipeline and run it. In which case, why didn’t you just use bash? Yeah, I know bash stinks. But this is not my fist rodeo* so I’m struggling here to see how CWL stinks less. See again: Inner-Platform Effect, Greenspun’s Tenth Rule. -- * i.e. I already know how easily “workflow engines” go up their own arse because I’ve done it myself. And CWL fails the same sniff test. Not a good start.

eiro 7y ago

it really feels like a sophisticated bad joke to me but i'm curious: any real world example ?

jkh1 7y ago
It's used in bioinformatics. for example by the Global Alliance for Genomics and Health[0]. You can find some CWL workflows on dockstore [1]. [0] https://www.ga4gh.org/ https://www.ga4gh.org/ [1] https://dockstore.org/search?descriptorType=CWL&searchMode=files https://dockstore.org/search?descriptorType=CWL&searchMode=f...
Xophmeister 7y ago
We've used it and competing workflow management "languages" -- there are a few of them -- in our bioinformatic processing pipelines. "Sophisticated bad joke" is a pretty accurate description.
- hhas01 7y ago
  Disappointing, but unsurprising. See also: Greenspun’s Tenth Rule, “Yo Dawg…”.
tetron 7y ago
https://github.com/search?q=extension%3Acwl+cwlVersion https://github.com/search?q=extension%3Acwl+cwlVersion

smartmic 7y ago

There are so many alternatives, see https://news.ycombinator.com/item?id=20410848 https://news.ycombinator.com/item?id=20410848 Any information about which language is leading the field?

jkh1 7y ago
Which field? In the absence of any constraint, the choice of workflow management system is down to individual preferences. Note that the goal of CWL is to be a standard workflow language not a workflow management system. That is, it is concerned purely by how to precisely describe a workflow, not by how it should be implemented which is the task of a workflow management system. The diversity of workflow management systems starts to be recognized as an obstacle to re-use and reproducibility of scientific workflows. Having different systems implementing a standard language would help get around this issue. Several workflow management systems used in bioinformatics have or are implementing support for CWL.
maddyboo 7y ago
https://xkcd.com/927/ https://xkcd.com/927/
- jkh1 7y ago
  This is not exactly true in this case as before CWL there was no standard, only languages tied to particular implementations.
- snak 7y ago
  Came to the comments looking for this.
ibern 7y ago
Right now I'd say it's in competition with Workflow Description Language (WDL), Nextflow, and maybe Snakemake. CWL seems to be more popular than WDL, as far as I can tell, but Nextflow is probably more popular than both.
- 7y ago

amir01 7y ago

it was valuable. if you want to practice real time industry projects. click here http://bit.ly/2KqTKiR http://bit.ly/2KqTKiR

hhas01 7y ago
Spammy account is spamming. Where’s a `flag` link when you need it.
- the-dude 7y ago
  You click the timestamp, and the flag button appears.
  - hhas01 7y ago
    Hah, thanks. Obvious brainfart is obvious. Never did figure why family got me one of these: https://i.pinimg.com/originals/5a/a7/95/5aa795b87c41d0f4d595c075042b1716.jpg https://i.pinimg.com/originals/5a/a7/95/5aa795b87c41d0f4d595...

hhas01 7y ago

Those who can, implement solutions. Those who can’t, implement Standards.

scottious 7y ago

Imagine a programming language except it also has to be valid YAML. Yeah. Now sprinkle some JavaScript in there just for good measure. Look no further than their own hand-crafted examples (confusingly called "Episodes") for compelling arguments for why to NOT use Common Workflow Language.

dang 7y ago
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." Your comment almost gets there, but not quite—and then it sinks itself lower by breaking this one: "Don't be snarky." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html Can you please take the guidelines a bit more to heart when posting here? We'd appreciate it.
- scottious 7y ago
  Sorry

prpl 7y ago

The attempt to cleanly separate out workflow is nice in theory but an extremely leaky abstraction when it comes to real world operations for DAGs larger than 10s of nodes. You will end up with lots of code that will duplicate functionality that exists in a workflow engine just to have something common (e.g. notifications, database ingest, etc...) There are no operational concerns in CWL.

deleted 7y ago
[deleted]
mbreese 7y ago
That’s my main issue with CWL. It’s a nice execution language, but not something I like to write an analysis pipeline in. Because it is YAML, using runtime logic is difficult (impossible?). I much rather use a pipeline tool that includes some flow control and ability to adapt to different arguments. Last I looked at CWL, this wasn’t the case.
- tetron 7y ago
  Adding conditional execution to the dataflow graph is something being worked on for the next version of CWL.
  - mbreese 7y ago
    After seeing the horror that was conditionals and flow control in Ant's build.xml format, I don't see anything based on a text file format working well...

tetron 7y ago

CWL project co-founder here. It's funny, I have been trying to start a discussion of CWL on HN for years, obviously someone else had to do it. The goal is to provide a way to describe dataflow processes that is highly portable, auditable, and reproducible. This is incredibly important in research, clinical, and regulatory domains where you need to be able to show how you came up with a result. It's not a general purpose language on purpose, and operational concerns like notifications are the domain of specific implementations (engines). I agree the syntax is horrible (and I designed most of it) but it also makes it easy to write programs that read and write CWL, enabling an ecosystem. For example, here is a transpiled languages that emits CWL: https://github.com/common-workflow-language/cwl-ex https://github.com/common-workflow-language/cwl-ex

Edmond 7y ago
something worth looking at: https://en.wikipedia.org/wiki/Business_Process_Execution_Language https://en.wikipedia.org/wiki/Business_Process_Execution_Lan...

hhas01 7y ago

Bless. https://media3.giphy.com/media/49zC0Bm1kbu36/source.gif https://media3.giphy.com/media/49zC0Bm1kbu36/source.gif

PeterStuer 7y ago

But is it SPEM™[1] compliant? /s [1] https://www.omg.org/spec/SPEM/2.0 https://www.omg.org/spec/SPEM/2.0

drocer88 7y ago

Snakemake, CWL, WDL, Nextflow are used in bioinformatics workflows. Just plain old bash seems to work just fine, though. It is simple and gives you a lot of portability to HPC clusters and general purpose cloud. Not hardwiring to a configuration file and not using an, in practice, un-portable workflow language saves you a lot of pain. A truly portable cloud/hpc module for python would be a good solution.

RocketSyntax 7y ago
Wondering why airflow and prefect haven't crept into space yet
- tetron 7y ago
  https://github.com/Barski-lab/cwl-airflow https://github.com/Barski-lab/cwl-airflow

JaDogg 7y ago

https://xkcd.com/927/ https://xkcd.com/927/