5 ms·

Reflow, a language for distributed, incremental data processing in the cloud

99 points by krab 5y ago

The_Amp_Walrus 5y ago
Is reflow go only? is the .rf file format a DSL or an existing language?
- prb2 5y ago
  Reflow is implemented in Go, but it can be used to run programs in any language. The Reflow language (.rf files) is a DSL, the language is described in more detail here: https://github.com/grailbio/reflow/blob/master/LANGUAGE.md https://github.com/grailbio/reflow/blob/master/LANGUAGE.md
- mariusae 5y ago
  If you want to work with Go, check out bigslice [1] (and bigmachine [2]), which is built on a similar architecture. [1] https://github.com/grailbio/bigslice/ https://github.com/grailbio/bigslice/ [2] https://github.com/grailbio/bigmachine https://github.com/grailbio/bigmachine
all2 5y ago
From the README: Reflow comprises: - a functional, lazy, type-safe domain specific language for writing workflow programs; - a runtime for evaluating Reflow programs incrementally, coordinating cluster execution, and transparent memoization; - a cluster scheduler to dynamically provision and tear down resources from a cloud provider (AWS currently supported). and Reflow was designed to support sophisticated, large-scale bioinformatics workflows, but should be widely applicable to scientific and engineering computing workloads. It was built using Go. Reflow joins a long list of systems designed to tackle bioinformatics workloads, but differ from these in important ways: - it is a vertically integrated system with a minimal set of external dependencies; this allows Reflow to be "plug-and-play": bring your cloud credentials, and you're off to the races; - it defines a strict data model which is used for transparent memoization and other optimizations; - it takes workflow software seriously: the Reflow DSL provides type checking, modularity, and other constructors that are commonplace in general purpose programming languages; because of its high level data model and use of caching, Reflow computes incrementally: it is always able to compute the smallest set of operations given what has been computed previously.
epistasis 5y ago
Interesting to see Grail share this, I'm excited to try it out. I'm perpetually unsatisfied with bioinformatics workflow software. Snakemake and GNU make remain my favorites so far in terms of developing novel analysis. However, making GNU make into a reusable pipeline always feels like an awful and ugly hack. And GNU make requires a shared file system among nodes, which is problematic on AWS... This seems to have potential for both recording the steps for reproducible science, but also turning those set of steps into a reusable pipeline easily.
- fwip 5y ago
  My personal favorite is Nextflow (http://nextflow.io/ http://nextflow.io/). Quick to start up a one-off script in, and it's ready to run in production without too much tweaking. Edit: I especially appreciate the wide range of supported systems for both dependency management (running the gamut from GNU modules or conda to docker/singularity containers) and execution environments (local, SLURM, SGE, AWS, Azure, etc.)
  - davidmasp 5y ago
    Completely agree, nextflow is a beast!
  - epistasis 5y ago
    I'm glad it works well for you! I haven't had a great experience with it, though I tried it because it does work so well for so many people.
dannykwells 5y ago
If you're into workflow runners, Reflow and Cromwell (https://github.com/broadinstitute/cromwell https://github.com/broadinstitute/cromwell) are the only two really to consider. Having tried them all, these two are by far the best and most supported (and there are 100s!) Cromwell is great because it is google cloud native and supported within the Terra ecosystem (https://app.terra.bio/ https://app.terra.bio/) meaning you do not need to host it yourself - you can just connect your google account and go. Reflow, I've heard, is a little more "professional" given that the Grail team is heavily ex-Google. But both can scale to massively parallel (1000+ parallel analyses).
- aednichols 5y ago
  Thanks for the shoutout. Cromwell/Terra developer here in an informal capacity, can answer Qs.
ritoune 5y ago
This looks cool, but for companies tied to k8s, probably a no go? I'd be all over this if it could just run on our k8s cluster..
- samuell 5y ago
  For k8s you might want to have a look at pachyderm [1]. We wrote a comparison between Pachy, Grail and some other Go workflow tools some time ago [2] [1] https://pachyderm.io https://pachyderm.io [2] http://gopherdata.io/post/more_go_based_workflow_tools_in_bioinformatics/ http://gopherdata.io/post/more_go_based_workflow_tools_in_bi...
anonymousDan 5y ago
One thing I don't get is its claim of being referentially transparent. How can this possibly be the case in the presence of exec commands that call shell scripts (which could do anything, potentially nondeterministically)? Or does the type system take this into account somehow?

99 points by krab 5y ago

The_Amp_Walrus 5y ago

Is reflow go only? is the .rf file format a DSL or an existing language?

prb2 5y ago
Reflow is implemented in Go, but it can be used to run programs in any language. The Reflow language (.rf files) is a DSL, the language is described in more detail here: https://github.com/grailbio/reflow/blob/master/LANGUAGE.md https://github.com/grailbio/reflow/blob/master/LANGUAGE.md
mariusae 5y ago
If you want to work with Go, check out bigslice [1] (and bigmachine [2]), which is built on a similar architecture. [1] https://github.com/grailbio/bigslice/ https://github.com/grailbio/bigslice/ [2] https://github.com/grailbio/bigmachine https://github.com/grailbio/bigmachine

all2 5y ago

From the README: Reflow comprises: - a functional, lazy, type-safe domain specific language for writing workflow programs; - a runtime for evaluating Reflow programs incrementally, coordinating cluster execution, and transparent memoization; - a cluster scheduler to dynamically provision and tear down resources from a cloud provider (AWS currently supported). and Reflow was designed to support sophisticated, large-scale bioinformatics workflows, but should be widely applicable to scientific and engineering computing workloads. It was built using Go. Reflow joins a long list of systems designed to tackle bioinformatics workloads, but differ from these in important ways: - it is a vertically integrated system with a minimal set of external dependencies; this allows Reflow to be "plug-and-play": bring your cloud credentials, and you're off to the races; - it defines a strict data model which is used for transparent memoization and other optimizations; - it takes workflow software seriously: the Reflow DSL provides type checking, modularity, and other constructors that are commonplace in general purpose programming languages; because of its high level data model and use of caching, Reflow computes incrementally: it is always able to compute the smallest set of operations given what has been computed previously.

epistasis 5y ago

Interesting to see Grail share this, I'm excited to try it out. I'm perpetually unsatisfied with bioinformatics workflow software. Snakemake and GNU make remain my favorites so far in terms of developing novel analysis. However, making GNU make into a reusable pipeline always feels like an awful and ugly hack. And GNU make requires a shared file system among nodes, which is problematic on AWS... This seems to have potential for both recording the steps for reproducible science, but also turning those set of steps into a reusable pipeline easily.

fwip 5y ago
My personal favorite is Nextflow (http://nextflow.io/ http://nextflow.io/). Quick to start up a one-off script in, and it's ready to run in production without too much tweaking. Edit: I especially appreciate the wide range of supported systems for both dependency management (running the gamut from GNU modules or conda to docker/singularity containers) and execution environments (local, SLURM, SGE, AWS, Azure, etc.)
- davidmasp 5y ago
  Completely agree, nextflow is a beast!
- epistasis 5y ago
  I'm glad it works well for you! I haven't had a great experience with it, though I tried it because it does work so well for so many people.

dannykwells 5y ago

If you're into workflow runners, Reflow and Cromwell (https://github.com/broadinstitute/cromwell https://github.com/broadinstitute/cromwell) are the only two really to consider. Having tried them all, these two are by far the best and most supported (and there are 100s!) Cromwell is great because it is google cloud native and supported within the Terra ecosystem (https://app.terra.bio/ https://app.terra.bio/) meaning you do not need to host it yourself - you can just connect your google account and go. Reflow, I've heard, is a little more "professional" given that the Grail team is heavily ex-Google. But both can scale to massively parallel (1000+ parallel analyses).

aednichols 5y ago
Thanks for the shoutout. Cromwell/Terra developer here in an informal capacity, can answer Qs.

ritoune 5y ago

This looks cool, but for companies tied to k8s, probably a no go? I'd be all over this if it could just run on our k8s cluster..

samuell 5y ago
For k8s you might want to have a look at pachyderm [1]. We wrote a comparison between Pachy, Grail and some other Go workflow tools some time ago [2] [1] https://pachyderm.io https://pachyderm.io [2] http://gopherdata.io/post/more_go_based_workflow_tools_in_bioinformatics/ http://gopherdata.io/post/more_go_based_workflow_tools_in_bi...

anonymousDan 5y ago

One thing I don't get is its claim of being referentially transparent. How can this possibly be the case in the presence of exec commands that call shell scripts (which could do anything, potentially nondeterministically)? Or does the type system take this into account somehow?