3 ms·

It seems the domain specific aspect is important. When I read that sentence, I was surprised not to see the classic suggestions from tech/data science: Prefec

It seems the domain specific aspect is important. When I read that sentence, I was surprised not to see the classic suggestions from tech/data science: Prefect, Airflow, Luigi, Apache NiFi, Jenkins, AWS Step Functions, or Pachyderm

MontyCarloHall 5y ago
The reason they’re domain specific is that they were intended to be usable by non-programmers, namely wetlab biologists. Of course, that didn’t end up being the case. If you can’t program at an elementary enough level to define simple workflow graphs using an API in a non-DSL (like those you mention), then the language is irrelevant—you’re just not technically sophisticated enough to be writing your own workflows. So now we have these DSLs that non-programmers still can’t use and programmers all hate. A lose-lose.
- geoffjentry 5y ago
  That's a big part of it. And I say this as someone who played a heavy role in the development of one of the named DSLs. Yes we had that exact intention and exact unintended outcome. There's another part of it as well. Bioinformatics workflows tend to be just enough different from more standard workflows that there is friction using off the shelf tooling. For one, the DAG nodes are often mostly/fully represented by command line tools expecting a POSIX style file system and making assumptions/asserting opinions on where files live, to where it writes outputs, etc. Bioinformatics workflow orchestrators can understand this and provide optimizations in the DSL to express how to manage the file movement. In contrast, I find many people with a more standard data/biz workflow mentality think of DAG nodes as being queries, running blocks of code, etc. Lifecycles vary as well. The institutions who have these workflows will either be pure research or some degree of research/production hybrid. There's an advantage to being able to use the same software on both side of that hybrid. When your research workflow is ready to be blessed to production, having to translate it into a different system can be expensive in terms of both time and bugs. Another aspect is that these workflows, until not too many years ago, were often run on HPC compute clusters using software like SGE, SLURM, PBS, etc. These DSLs can provide optimizations for tweaking parameters in a way that more mainstream tools do not.
- epistasis 5y ago
  I don't think that a single one of these was meant to be used by wetlab biologists, and every single one assumes that the user will be learning an entirely new text based language to express the workflow semantics. In contrast, I see useless GUIs as fairly common on non-bioinformatics workflow systems, and when I see that I know that it is unlikely to be able to scale what I need it to do. Manually drawing every box and line assumes simple workflows that are unlikely to scale very far. Bioinformatics and Other workflow management systems just have different focuses, more than they have different end user skills. In the 2000s I tried to shoe-horn so many CS Systems type workflow onto using clusters, and every single one had abstractions that simply did not fit what I needed. Typically they seem to require knowledge of how many tasks a job must be split up into, for example, instead of dynamically adapting the workflow to the inputs as desired. Or they hard code the method does launching processes to a certain type of compute environment. So when somebody says "workflow," I assume they mean something entirely different then what I mean until proven otherwise. For example, if I go to the Apache NiFi docs to see if it's suitable, the Getting Started page starts with a GUI, which is a huge red flag, I don't want to do workflow entry with a GUI because a pre-defined graph will only cover a fraction of my use cases. But worse then that, when it describes where to start creating a processor, it says you need to select from a list of processors. Now this is an even BIGGER red flag because I know absolutely none of the commands I need will be available, and I'll need to add tons of metadata for whatever I want, most likely in some sort of bad GUI, or even worse, XML or some other bad data standard that software engineering adopted due to cargo-cult design. The bigger danger is that the system assumes that processors must be written in some certain language, or assumes that data itself will be introspectable, or something that will require massive amounts of work. Every one of these systems embodies a huuuuuuge number of assumptions about what it means to run a workflow, nobody ever states what those assumptions are, or probably even realizes how different their needs are from others' needs. At least with the workflow systems from other people in bioinformatics, I know that the assumptions they make will work with the tools and data sizes in the field, and won't have some deal-breaker I only find out about after three hours of making workflows.
  - geoffjentry 5y ago
    > I don't think that a single one of these was meant to be used by wetlab biologists Speaking for myself, we didn't target wetlab biologists with WDL but we did target "non-programmers" as MontyCarloHall described. This was influenced by the groups from which the original developers originated prior to working on the DSL. The degree to which we were successful can be debated, but hey that was the idea at least ;)
nerdponx 5y ago
Also DVC, which has gotten really good in the last 2 years.

MontyCarloHall 5y ago

The reason they’re domain specific is that they were intended to be usable by non-programmers, namely wetlab biologists. Of course, that didn’t end up being the case. If you can’t program at an elementary enough level to define simple workflow graphs using an API in a non-DSL (like those you mention), then the language is irrelevant—you’re just not technically sophisticated enough to be writing your own workflows. So now we have these DSLs that non-programmers still can’t use and programmers all hate. A lose-lose.

geoffjentry 5y ago
That's a big part of it. And I say this as someone who played a heavy role in the development of one of the named DSLs. Yes we had that exact intention and exact unintended outcome. There's another part of it as well. Bioinformatics workflows tend to be just enough different from more standard workflows that there is friction using off the shelf tooling. For one, the DAG nodes are often mostly/fully represented by command line tools expecting a POSIX style file system and making assumptions/asserting opinions on where files live, to where it writes outputs, etc. Bioinformatics workflow orchestrators can understand this and provide optimizations in the DSL to express how to manage the file movement. In contrast, I find many people with a more standard data/biz workflow mentality think of DAG nodes as being queries, running blocks of code, etc. Lifecycles vary as well. The institutions who have these workflows will either be pure research or some degree of research/production hybrid. There's an advantage to being able to use the same software on both side of that hybrid. When your research workflow is ready to be blessed to production, having to translate it into a different system can be expensive in terms of both time and bugs. Another aspect is that these workflows, until not too many years ago, were often run on HPC compute clusters using software like SGE, SLURM, PBS, etc. These DSLs can provide optimizations for tweaking parameters in a way that more mainstream tools do not.
epistasis 5y ago
I don't think that a single one of these was meant to be used by wetlab biologists, and every single one assumes that the user will be learning an entirely new text based language to express the workflow semantics. In contrast, I see useless GUIs as fairly common on non-bioinformatics workflow systems, and when I see that I know that it is unlikely to be able to scale what I need it to do. Manually drawing every box and line assumes simple workflows that are unlikely to scale very far. Bioinformatics and Other workflow management systems just have different focuses, more than they have different end user skills. In the 2000s I tried to shoe-horn so many CS Systems type workflow onto using clusters, and every single one had abstractions that simply did not fit what I needed. Typically they seem to require knowledge of how many tasks a job must be split up into, for example, instead of dynamically adapting the workflow to the inputs as desired. Or they hard code the method does launching processes to a certain type of compute environment. So when somebody says "workflow," I assume they mean something entirely different then what I mean until proven otherwise. For example, if I go to the Apache NiFi docs to see if it's suitable, the Getting Started page starts with a GUI, which is a huge red flag, I don't want to do workflow entry with a GUI because a pre-defined graph will only cover a fraction of my use cases. But worse then that, when it describes where to start creating a processor, it says you need to select from a list of processors. Now this is an even BIGGER red flag because I know absolutely none of the commands I need will be available, and I'll need to add tons of metadata for whatever I want, most likely in some sort of bad GUI, or even worse, XML or some other bad data standard that software engineering adopted due to cargo-cult design. The bigger danger is that the system assumes that processors must be written in some certain language, or assumes that data itself will be introspectable, or something that will require massive amounts of work. Every one of these systems embodies a huuuuuuge number of assumptions about what it means to run a workflow, nobody ever states what those assumptions are, or probably even realizes how different their needs are from others' needs. At least with the workflow systems from other people in bioinformatics, I know that the assumptions they make will work with the tools and data sizes in the field, and won't have some deal-breaker I only find out about after three hours of making workflows.
- geoffjentry 5y ago
  > I don't think that a single one of these was meant to be used by wetlab biologists Speaking for myself, we didn't target wetlab biologists with WDL but we did target "non-programmers" as MontyCarloHall described. This was influenced by the groups from which the original developers originated prior to working on the DSL. The degree to which we were successful can be debated, but hey that was the idea at least ;)

nerdponx 5y ago

Also DVC, which has gotten really good in the last 2 years.