7 ms·

Nextflow: Data-Driven Computational Pipelines

90 points by brianzelip 3y ago

adolph 3y ago
Another day, another workflow DSL. * looks like yaml * has curly braces to look programmery * whitespace might be meaningful * has pipes like a bash script https://www.commonwl.org/ https://www.commonwl.org/ https://github.com/common-workflow-language/common-workflow-language/wiki/Existing-Workflow-systems https://github.com/common-workflow-language/common-workflow-... mea culpa: The above was based on a first look at something titled "A DSL for parallel and and scalable computational pipelines"a as opposed to "Java workflow manager with Groovy language scripting." The presented screenshot still looks to me like an unholy union of yaml, js, py and sh. If that sounds groovy to you; have fun.
- bafe 3y ago
  If you refer to nextflow,the syntax is basically groovy
  - notQuiteEither 3y ago
    Indeed, it's a fully functional scripting language that is an extension of groovy. So "looks programmery" is more than a little reductive. Edit: functional as in, it's not half-hearted, not functional as in functional programming.
  - adolph 3y ago
    Interesting. From the docs: The Nextflow scripting language is an extension of the Groovy programming language. Groovy is a powerful programming language for the Java virtual machine. The Nextflow syntax has been specialized to ease the writing of computational pipelines in a declarative manner. https://www.nextflow.io/docs/latest/script.html?highlight=groovy https://www.nextflow.io/docs/latest/script.html?highlight=gr...
    - joysofpi 3y ago
      Its really just a layer on top that uses the groovy DSL features http://docs.groovy-lang.org/docs/latest/html/documentation/core-domain-specific-languages.html http://docs.groovy-lang.org/docs/latest/html/documentation/c...
- esafak 3y ago
  In all fairness, they predate the competition (2013): https://github.com/nextflow-io/nextflow/releases?page=25 https://github.com/nextflow-io/nextflow/releases?page=25
  - geoffjentry 3y ago
    As GP referenced CWL, while NF had appeared first in terms of the bioinformatics world Nextflow, CWL, Snakelike, and WDL all erupted close enough to each other to be equal-ish. The people were aware of each other but they were all so nascent that it wasn't clear if it was worth joining in or not. At the end of the day these all came from groups trying to scratch particular itches, and not everyone agreed on the right way to scratch. However all of them were rejections of prior models as well as the workflow solutions prominent in the business space.
    - adolph 3y ago
      Yeah, the thing that I find disappointing is that there is a lot of science value locked into the different systems of describing a workflow, pipeline or DAG. Like you said, they all had different itches to scratch and even some barebones "standards" like csv have flavors/extensions/etc.
    - bafe 3y ago
      They try to address similar solutions, but comparing snakemake and nextflow doesn't do either tool a favour. They use different computation models, nextflow is based on dataflow programming and therefore schedules processes dynamically as new data comes in, while snakemake is pull-based and schedules the processes based on the dag defined by the dependencies. Anyhow they are both great tools.
      - geoffjentry 3y ago
        While true that’s a minor distinction when comparing the clusters of bioinformatics workflow systems vs workflow systems aimed at different domains
suslik 3y ago
I develop bioinformatics pipelines for a living and am very opinionated on the topic. Having enough experience with snakemake as well as nextflow in production for many years now, I would always opt out for snakemake for anything but extremely large DAgs (which is quite rare for for bioinformatics pipelines). The fact that nextflow still doesn't allow deleting temporary files during execution or re-rerunning the workflow from a set of intermediary files is an insane deal breaker (not mentioning other strange things like an arbitrary limit of 1000 parallel jobs etc.). AWS runners that once were nextflow's selling point is not an advantage anymore give Amazon Genomics Cli. Subjectively, writing pipelines that incorporate conditional logic is much nicer in python+snakemake than groovy+nextflow, but maybe there is someone out there who prefers groovy. Is a proper dry run possible in nextflow already btw?
- mbreese 3y ago
  I do too.. and have similar opinions. I wrote my own tool years back for pipelines because it was always frustrating (started roughly around the same time as Nextflow). Allowing for files to be marked as transient (temp) and re-running from arbitrary time points are definitely one of the things I support... as is conditional logic within the pipeline for job definition and resource usage. For me though, one of the biggest things is that I like having composable pipelines, so each part of the larger workflow can be developed independently. They can interact with each other (DAG) and use existing dependencies, but they don't have to exist in the same document/script. I work on large WGS datasets, so 1000's of jobs per patient isn't uncommon. Happy to talk more if you're interested. https://github.com/compgen-io/cgpipe https://github.com/compgen-io/cgpipe And yes, you can dry run the entire thing. It will write out a bash script if you want to see exactly what is going to run without submitting jobs. It's a full language for pipelines, but heavily inspired by Makefiles (with conditional logic).
  - suslik 3y ago
    Interesting, thanks for sharing. I'll definitely take a look, although at this point I am so comfortable with Snakemake, it is a bit hard to imagine what would convince me to move to another tool. But I like the idea of composable pipelines: I am building a tool (too early to share) that would allow to lay Snakemake pipelines on top of each other using semi-automatic data annotations similar to how it is done in kedro (https://github.com/kedro-org/kedro https://github.com/kedro-org/kedro).
radus 3y ago
I've considered using Nextflow for bioinformatics pipelines but have yet to take the plunge. At work, I develop a proteomics pipeline that is composed of huey¹ tasks (Python library; simple alternative to Celery) which either use subprocess to call out to some external tool, or are just pure python. It runs in a worker container which is managed by Docker swarm, and all containers pull jobs from redis. For our scale, it works great. However, I don't have control over the resource utilization of individual steps, and in the past I've had issues with the pipeline blocking as a result of how I was chaining tasks together. I think something like Nextflow would remove these limitations, but one thing I think I would miss is the ability to debug individual pipeline steps locally with an interactive debugger. As far as I can tell, Nextflow has logging/tracing facilities but nothing quite like an interactive debugger. I'd be happy to be told I'm wrong, or even that I'm doing it wrong. Other reasons I'd like to start using Nextflow: - my homebrew pipeline would be easier to setup/share - there are some efforts in the proteomics community to develop Nextflow pipelines (eg. QuantMS²). I think it would to have a shared language to express pipelines, and it would make benchmarking simpler. ___ ¹ https://github.com/coleifer/huey/ https://github.com/coleifer/huey/ ² https://docs.quantms.org/en/latest/ https://docs.quantms.org/en/latest/
- nonrepeating 3y ago
  The closest I’ve gotten to local debugging is having the Python scripts that are launched by NextFlow steps connect to a remote debugger process (“remote” but running on the same workstation). PyCharm makes this fairly painless to orchestrate. I’ve never been able to debug thr Groovy script in a Nextflow pipeline itself; I think you’d need a debug build of the nextflow executable for that.
- a_bonobo 3y ago
  As someone who dips into nextflow from time to time, I'd strongly suggest by developing your pipeline based on an existing nf-core pipeline or the nf-core templates. nf-core comes with a bunch of nicer defaults like profiles for SLURM, Singularity, Docker that help you abstract some of the headaches away, plus you could get lucky and can just glue some of their modules together.
anyoneamous 3y ago
It's kind of a shame this is based on Groovy, rather than Python which is much more familiar to people in the HCLS space. I've always been stuck on the fence between wanting to use NF (since it's the most popular) and Snakemake (which feels like less of an oddity development-wise).
- bafe 3y ago
  Perhaps it is not as popular, but I found the groovy syntax ideal for DSL like that
  - geoffjentry 3y ago
    I feel like groovy pushes towards the worst of both worlds between an internal dsl and external dsl. It’s an internal dsl so you get the language but oh man groovy sucks
dekhn 3y ago
In theory I want to like Nextflow, but my main criticism is that it's really, really hard to debug programs that pass around lists of Promises (nextflow's dag uses promises as handles on future computations, and functions receive promises and can't easily materialize them and print them. The caching is often more trouble than it's worth. Also, the little bash scripts that integrate with AWS break if your AWS environment isn't vanilla (our enterprise AWS has a lot of restrictions).
- nonrepeating 3y ago
  Using Tower for Nextflow can help streamline it on AWS, it’s pretty powerful (but costs money for anything beyond trivial use cases): tower.nf
firecraker 3y ago
So my question to the non bioinformatics - is this already a solved problem? You have tasks which require resources based on the input parameters, these are run in docker containers to ensure the environment and you want to track the output of each step. Often these are embarrassingly parallel operations (e.g. I have 200 samples to do the same thing on). Something like dask perhaps,but can specify a docker image for the task? What is the goto in DevOps for similar tasks? GitHub actions comes pretty close... To bioinformatics what is the unique selling point of next flow over say wdl/Cromwell?
- geoffjentry 3y ago
  The big difference when comparing bioinformatics systems with non are what the typical payload of a DAG node is and what optimizations that indicates. Most other domains don’t have DAG nodes that assume the payload is a crappy command line call and expecting inputs/outputs to magically be in specific places on a POSIX file system. You can do this on other systems but it’s nice to have the headache abstracted away for you. The other major difference is assumption of lifecycle. In most biz domains you don’t have researchers iterating on these things the way you do in bioinf. The newer ML/DS systems do solve this problem than say Aorflow
  - samuell 3y ago
    I for one have started to appreciate the fact that the shell/commandline interface means: - We have an interface that very strongly imposes composability, that is rarely seen in other parts of IT, and making people actually "follow the rules" :D - Data is (mostly) treated as immutable, except perhaps inside tools - Data is cached - The cli boundaries means that at least one can inspect inputs/outputs as a way to debug. - Etc... Personally, the biggest frustration is all the inconsistencies in how people design the commandline interfaces. Primarily that output filenames are so often created based on non-obvious and sometimes arbitrary rules, rather than being specified by the user. If all filenames were specified (or at least possible to specify) via the CLI, pipeline managers would have such an enormously easier time. What happens now is that you basically need a mechanism like Nextflow has, where all commands are executed in a temp directory, and the pipeline tool just globs up all the generated files afterwards. This works, but opens a lot of possibilities for mistakes in how files are tracked (might be routed to the wrong downstream output, if you do something funny with the naming, such that two output path patterns overlap).
jromma 3y ago
I'm personally a huge fan of redun¹ for running computational pipelines. It's pure python, it's easy to learn/debug, it has automatic caching, retry, provenance logging, and a great integration with AWS Batch for running large jobs. I've been really impressed with how easy it is to run a job to completion that fans out to thousands of AWS spot instances at once. I've used nextflow in the past, and I've found it to be much harder to use. Learning another DSL is annoying, documentation was sparse, I constantly ran into bugs, and it was hard to debug in general. I don't know how much it's changed over the past 3 years though. ¹https://github.com/insitro/redun https://github.com/insitro/redun
- radus 3y ago
  This is the first time I've seen this and it looks very interesting -- thank you for sharing! I described my workflow in a different comment on this post, and this seems like something I could port to with minimal changes in code since every step is already a Python function and even decorated by @task.
jkh1 3y ago
How many workflow management systems do we need? Over 300[1]. If that's not reinventing the wheel. [1] https://github.com/common-workflow-language/common-workflow-language/wiki/Existing-Workflow-systems https://github.com/common-workflow-language/common-workflow-...
- samuell 3y ago
  Pretty much everyone underestimates the complexity of workflow scheduling. This means like 99% of tools have had various random limitations and quirks that make them not applicable to a lot of use cases. This is something that people using these tools mostly never realize (and so complain about the "reinvention"). The folks I know who tried to implement a pipeline tool are often a bit more aware of the challenges and how hard it is to make something that is really general. I say this as someone who evaluated a dozen tools, and finally extended an existing tool to fix some limitations (Luigi, with our SciLuigi extension), and finally developing our own tool (SciPipe). It has gotten better, and a tool like Nextflow is pretty generic these days, although they also might have limitations. For example, before DSL2 we needed re-usable modules, which is why we developed SciPipe, which otherwise has a very similar scheduling mechanism to Nextflow (Dataflow/Flow-based). Still today, I'm having mixed feelings about using extremely complex tools that are dependent on a single organisation to keep updating. Not being able to easily debug execution and a few other things, which is why we wanted a simple library that we could understand ourselves and run through a debugger. (And it didn't hurt that we could get complete audit logs per output file, which can be very useful both for provenance and debugging, and is not found in almost any other tool.) Just to give some examples of why someone might still entertain thoughts about developing separate tools. All in all, the widely used ones like Nextflow (and Snakemake) are great tools. They just aren't optimal for every usecase and situation.

90 points by brianzelip 3y ago

Another day, another workflow DSL. * looks like yaml * has curly braces to look programmery * whitespace might be meaningful * has pipes like a bash script https://www.commonwl.org/ https://www.commonwl.org/ https://github.com/common-workflow-language/common-workflow-language/wiki/Existing-Workflow-systems https://github.com/common-workflow-language/common-workflow-... mea culpa: The above was based on a first look at something titled "A DSL for parallel and and scalable computational pipelines"a as opposed to "Java workflow manager with Groovy language scripting." The presented screenshot still looks to me like an unholy union of yaml, js, py and sh. If that sounds groovy to you; have fun.

bafe 3y ago
If you refer to nextflow,the syntax is basically groovy
- notQuiteEither 3y ago
  Indeed, it's a fully functional scripting language that is an extension of groovy. So "looks programmery" is more than a little reductive. Edit: functional as in, it's not half-hearted, not functional as in functional programming.
- adolph 3y ago
  Interesting. From the docs: The Nextflow scripting language is an extension of the Groovy programming language. Groovy is a powerful programming language for the Java virtual machine. The Nextflow syntax has been specialized to ease the writing of computational pipelines in a declarative manner. https://www.nextflow.io/docs/latest/script.html?highlight=groovy https://www.nextflow.io/docs/latest/script.html?highlight=gr...
  - joysofpi 3y ago
    Its really just a layer on top that uses the groovy DSL features http://docs.groovy-lang.org/docs/latest/html/documentation/core-domain-specific-languages.html http://docs.groovy-lang.org/docs/latest/html/documentation/c...
esafak 3y ago
In all fairness, they predate the competition (2013): https://github.com/nextflow-io/nextflow/releases?page=25 https://github.com/nextflow-io/nextflow/releases?page=25
- geoffjentry 3y ago
  As GP referenced CWL, while NF had appeared first in terms of the bioinformatics world Nextflow, CWL, Snakelike, and WDL all erupted close enough to each other to be equal-ish. The people were aware of each other but they were all so nascent that it wasn't clear if it was worth joining in or not. At the end of the day these all came from groups trying to scratch particular itches, and not everyone agreed on the right way to scratch. However all of them were rejections of prior models as well as the workflow solutions prominent in the business space.
  - adolph 3y ago
    Yeah, the thing that I find disappointing is that there is a lot of science value locked into the different systems of describing a workflow, pipeline or DAG. Like you said, they all had different itches to scratch and even some barebones "standards" like csv have flavors/extensions/etc.
  - bafe 3y ago
    They try to address similar solutions, but comparing snakemake and nextflow doesn't do either tool a favour. They use different computation models, nextflow is based on dataflow programming and therefore schedules processes dynamically as new data comes in, while snakemake is pull-based and schedules the processes based on the dag defined by the dependencies. Anyhow they are both great tools.
    - geoffjentry 3y ago
      While true that’s a minor distinction when comparing the clusters of bioinformatics workflow systems vs workflow systems aimed at different domains

suslik 3y ago

I develop bioinformatics pipelines for a living and am very opinionated on the topic. Having enough experience with snakemake as well as nextflow in production for many years now, I would always opt out for snakemake for anything but extremely large DAgs (which is quite rare for for bioinformatics pipelines). The fact that nextflow still doesn't allow deleting temporary files during execution or re-rerunning the workflow from a set of intermediary files is an insane deal breaker (not mentioning other strange things like an arbitrary limit of 1000 parallel jobs etc.). AWS runners that once were nextflow's selling point is not an advantage anymore give Amazon Genomics Cli. Subjectively, writing pipelines that incorporate conditional logic is much nicer in python+snakemake than groovy+nextflow, but maybe there is someone out there who prefers groovy. Is a proper dry run possible in nextflow already btw?

mbreese 3y ago
I do too.. and have similar opinions. I wrote my own tool years back for pipelines because it was always frustrating (started roughly around the same time as Nextflow). Allowing for files to be marked as transient (temp) and re-running from arbitrary time points are definitely one of the things I support... as is conditional logic within the pipeline for job definition and resource usage. For me though, one of the biggest things is that I like having composable pipelines, so each part of the larger workflow can be developed independently. They can interact with each other (DAG) and use existing dependencies, but they don't have to exist in the same document/script. I work on large WGS datasets, so 1000's of jobs per patient isn't uncommon. Happy to talk more if you're interested. https://github.com/compgen-io/cgpipe https://github.com/compgen-io/cgpipe And yes, you can dry run the entire thing. It will write out a bash script if you want to see exactly what is going to run without submitting jobs. It's a full language for pipelines, but heavily inspired by Makefiles (with conditional logic).
- suslik 3y ago
  Interesting, thanks for sharing. I'll definitely take a look, although at this point I am so comfortable with Snakemake, it is a bit hard to imagine what would convince me to move to another tool. But I like the idea of composable pipelines: I am building a tool (too early to share) that would allow to lay Snakemake pipelines on top of each other using semi-automatic data annotations similar to how it is done in kedro (https://github.com/kedro-org/kedro https://github.com/kedro-org/kedro).

radus 3y ago

I've considered using Nextflow for bioinformatics pipelines but have yet to take the plunge. At work, I develop a proteomics pipeline that is composed of huey¹ tasks (Python library; simple alternative to Celery) which either use subprocess to call out to some external tool, or are just pure python. It runs in a worker container which is managed by Docker swarm, and all containers pull jobs from redis. For our scale, it works great. However, I don't have control over the resource utilization of individual steps, and in the past I've had issues with the pipeline blocking as a result of how I was chaining tasks together. I think something like Nextflow would remove these limitations, but one thing I think I would miss is the ability to debug individual pipeline steps locally with an interactive debugger. As far as I can tell, Nextflow has logging/tracing facilities but nothing quite like an interactive debugger. I'd be happy to be told I'm wrong, or even that I'm doing it wrong. Other reasons I'd like to start using Nextflow: - my homebrew pipeline would be easier to setup/share - there are some efforts in the proteomics community to develop Nextflow pipelines (eg. QuantMS²). I think it would to have a shared language to express pipelines, and it would make benchmarking simpler. ___ ¹ https://github.com/coleifer/huey/ https://github.com/coleifer/huey/ ² https://docs.quantms.org/en/latest/ https://docs.quantms.org/en/latest/

nonrepeating 3y ago
The closest I’ve gotten to local debugging is having the Python scripts that are launched by NextFlow steps connect to a remote debugger process (“remote” but running on the same workstation). PyCharm makes this fairly painless to orchestrate. I’ve never been able to debug thr Groovy script in a Nextflow pipeline itself; I think you’d need a debug build of the nextflow executable for that.
a_bonobo 3y ago
As someone who dips into nextflow from time to time, I'd strongly suggest by developing your pipeline based on an existing nf-core pipeline or the nf-core templates. nf-core comes with a bunch of nicer defaults like profiles for SLURM, Singularity, Docker that help you abstract some of the headaches away, plus you could get lucky and can just glue some of their modules together.

anyoneamous 3y ago

It's kind of a shame this is based on Groovy, rather than Python which is much more familiar to people in the HCLS space. I've always been stuck on the fence between wanting to use NF (since it's the most popular) and Snakemake (which feels like less of an oddity development-wise).

bafe 3y ago
Perhaps it is not as popular, but I found the groovy syntax ideal for DSL like that
- geoffjentry 3y ago
  I feel like groovy pushes towards the worst of both worlds between an internal dsl and external dsl. It’s an internal dsl so you get the language but oh man groovy sucks

dekhn 3y ago

In theory I want to like Nextflow, but my main criticism is that it's really, really hard to debug programs that pass around lists of Promises (nextflow's dag uses promises as handles on future computations, and functions receive promises and can't easily materialize them and print them. The caching is often more trouble than it's worth. Also, the little bash scripts that integrate with AWS break if your AWS environment isn't vanilla (our enterprise AWS has a lot of restrictions).

nonrepeating 3y ago
Using Tower for Nextflow can help streamline it on AWS, it’s pretty powerful (but costs money for anything beyond trivial use cases): tower.nf

firecraker 3y ago

So my question to the non bioinformatics - is this already a solved problem? You have tasks which require resources based on the input parameters, these are run in docker containers to ensure the environment and you want to track the output of each step. Often these are embarrassingly parallel operations (e.g. I have 200 samples to do the same thing on). Something like dask perhaps,but can specify a docker image for the task? What is the goto in DevOps for similar tasks? GitHub actions comes pretty close... To bioinformatics what is the unique selling point of next flow over say wdl/Cromwell?

geoffjentry 3y ago
The big difference when comparing bioinformatics systems with non are what the typical payload of a DAG node is and what optimizations that indicates. Most other domains don’t have DAG nodes that assume the payload is a crappy command line call and expecting inputs/outputs to magically be in specific places on a POSIX file system. You can do this on other systems but it’s nice to have the headache abstracted away for you. The other major difference is assumption of lifecycle. In most biz domains you don’t have researchers iterating on these things the way you do in bioinf. The newer ML/DS systems do solve this problem than say Aorflow
- samuell 3y ago
  I for one have started to appreciate the fact that the shell/commandline interface means: - We have an interface that very strongly imposes composability, that is rarely seen in other parts of IT, and making people actually "follow the rules" :D - Data is (mostly) treated as immutable, except perhaps inside tools - Data is cached - The cli boundaries means that at least one can inspect inputs/outputs as a way to debug. - Etc... Personally, the biggest frustration is all the inconsistencies in how people design the commandline interfaces. Primarily that output filenames are so often created based on non-obvious and sometimes arbitrary rules, rather than being specified by the user. If all filenames were specified (or at least possible to specify) via the CLI, pipeline managers would have such an enormously easier time. What happens now is that you basically need a mechanism like Nextflow has, where all commands are executed in a temp directory, and the pipeline tool just globs up all the generated files afterwards. This works, but opens a lot of possibilities for mistakes in how files are tracked (might be routed to the wrong downstream output, if you do something funny with the naming, such that two output path patterns overlap).

jromma 3y ago

I'm personally a huge fan of redun¹ for running computational pipelines. It's pure python, it's easy to learn/debug, it has automatic caching, retry, provenance logging, and a great integration with AWS Batch for running large jobs. I've been really impressed with how easy it is to run a job to completion that fans out to thousands of AWS spot instances at once. I've used nextflow in the past, and I've found it to be much harder to use. Learning another DSL is annoying, documentation was sparse, I constantly ran into bugs, and it was hard to debug in general. I don't know how much it's changed over the past 3 years though. ¹https://github.com/insitro/redun https://github.com/insitro/redun

radus 3y ago
This is the first time I've seen this and it looks very interesting -- thank you for sharing! I described my workflow in a different comment on this post, and this seems like something I could port to with minimal changes in code since every step is already a Python function and even decorated by @task.

jkh1 3y ago

How many workflow management systems do we need? Over 300[1]. If that's not reinventing the wheel. [1] https://github.com/common-workflow-language/common-workflow-language/wiki/Existing-Workflow-systems https://github.com/common-workflow-language/common-workflow-...

samuell 3y ago
Pretty much everyone underestimates the complexity of workflow scheduling. This means like 99% of tools have had various random limitations and quirks that make them not applicable to a lot of use cases. This is something that people using these tools mostly never realize (and so complain about the "reinvention"). The folks I know who tried to implement a pipeline tool are often a bit more aware of the challenges and how hard it is to make something that is really general. I say this as someone who evaluated a dozen tools, and finally extended an existing tool to fix some limitations (Luigi, with our SciLuigi extension), and finally developing our own tool (SciPipe). It has gotten better, and a tool like Nextflow is pretty generic these days, although they also might have limitations. For example, before DSL2 we needed re-usable modules, which is why we developed SciPipe, which otherwise has a very similar scheduling mechanism to Nextflow (Dataflow/Flow-based). Still today, I'm having mixed feelings about using extremely complex tools that are dependent on a single organisation to keep updating. Not being able to easily debug execution and a few other things, which is why we wanted a simple library that we could understand ourselves and run through a debugger. (And it didn't hurt that we could get complete audit logs per output file, which can be very useful both for provenance and debugging, and is not found in almost any other tool.) Just to give some examples of why someone might still entertain thoughts about developing separate tools. All in all, the widely used ones like Nextflow (and Snakemake) are great tools. They just aren't optimal for every usecase and situation.