5 ms·

We've found that different users want different levels of wrapping -- some (like you) are experts on the low-level tools, and choose to use our low-level comput

We've found that different users want different levels of wrapping -- some (like you) are experts on the low-level tools, and choose to use our low-level compute, storage, and analysis services directly -- I'm biased, but I believe Google Cloud's low-level services have features, like preemptible VMs and nearline storage, that are a great fit for working with genomic data. Others want gateways into bioinformatics-friendly environments like R Studio and iPython; others want wrapped packaged services; others want point-and-click UIs ,etc. Disclaimer: I work on Google Genomics.

PascLeRasc 11y ago
I'm really excited about moving to R/Python computing for my bioinformatics work, it's a much better workflow than the typical method of emailing matlab files to an in-house Xeon workstation, and it'll be easier to virtualize. This is something I've seen in dozens of labs and there's a huge amount of room to streamline workflows. Disclaimer: I work in a neurobio lab but I wish I worked for Google Genomics.
laurencerowe 11y ago
The problem with spot market and preemptible VMs is that much of the existing bioinformatics code often has run times of days to weeks with current sequencing depths. What's really needed is a way to have a VM suspend instead of shutdown when costs get high so it can resume once prices go down.
- jghn 11y ago
  It depends. If you picture an entire pipeline what often happens is that you end up with both long and short running tasks. Tuning your pipeline to an environment like this involves figuring out what can go where. As an example we turn around a whole genome in just over a day but that's comprised of hundreds of individual units of work
  - jsolson 11y ago
    Assuming you're running multiple pipelines at once you can add robustness in this sort of scenario by running the long-running tasks on guaranteed VMs (complete with live migration on GCE) and the short-running tasks on cheap preemptible VMs. Another features that comes to mind for enabling this sort of work is GCE's support for attaching a persistent disk volume to multiple VMs (read-write for one, read-only for others) -- have each pipeline stage consume from a read-only PD containing the output of the previous stage and write its results to a PD shared with the VM(s) that handle the next stage. Have the VMs communicate with each other for control plane operations ("hey, this directory contains the complete results of stage <x>, one of you stage <y> workers can pick it up" or "I have completed processing the stage <x> outputs for job <5>; they can be removed or archived to GCS."). This strategy also allows independent scaling of the workers responsible for each stage based on the computational requirements of that stage, since a given PD can be shared read-only with many VMs (with limited performance impact in the many-reader case, especially if the readers are not hammering the same parts of the volume). Another option for truly pipeline oriented processing (assuming the underlying workers can generate data suitable for it) is to leverage something like Cloud Dataflow -- then they'll handle the work balancing for you. (note: I work on GCE)
    - i000 11y ago
      I have to say that PDs attached read-only to multiple VMs is a very important part of my pipelines. And also a distinguishing feature of GCE. Clound Dataflow on the other hand is difficult to use in practice - one of the main reasons is the large footprint of intermediate files (sometimes 10x the size of the input) Currently we are running monolithic pipelines on the cloud where one instance does the job from start to finish (if it get's preempted - I restart). I plan the next version to be in some way "split-apart" but have not found a good strategy/tooling - any advice?
      - jkff 11y ago
        Hi! I'm an engineer on the Dataflow team. I'd love to understand the issue you mentioned about size of intermediate files, and help make Dataflow work well for you. Could you send a note to dataflow-feedback@google.com and elaborate?

PascLeRasc 11y ago

I'm really excited about moving to R/Python computing for my bioinformatics work, it's a much better workflow than the typical method of emailing matlab files to an in-house Xeon workstation, and it'll be easier to virtualize. This is something I've seen in dozens of labs and there's a huge amount of room to streamline workflows. Disclaimer: I work in a neurobio lab but I wish I worked for Google Genomics.

laurencerowe 11y ago

The problem with spot market and preemptible VMs is that much of the existing bioinformatics code often has run times of days to weeks with current sequencing depths. What's really needed is a way to have a VM suspend instead of shutdown when costs get high so it can resume once prices go down.

jghn 11y ago
It depends. If you picture an entire pipeline what often happens is that you end up with both long and short running tasks. Tuning your pipeline to an environment like this involves figuring out what can go where. As an example we turn around a whole genome in just over a day but that's comprised of hundreds of individual units of work
- jsolson 11y ago
  Assuming you're running multiple pipelines at once you can add robustness in this sort of scenario by running the long-running tasks on guaranteed VMs (complete with live migration on GCE) and the short-running tasks on cheap preemptible VMs. Another features that comes to mind for enabling this sort of work is GCE's support for attaching a persistent disk volume to multiple VMs (read-write for one, read-only for others) -- have each pipeline stage consume from a read-only PD containing the output of the previous stage and write its results to a PD shared with the VM(s) that handle the next stage. Have the VMs communicate with each other for control plane operations ("hey, this directory contains the complete results of stage <x>, one of you stage <y> workers can pick it up" or "I have completed processing the stage <x> outputs for job <5>; they can be removed or archived to GCS."). This strategy also allows independent scaling of the workers responsible for each stage based on the computational requirements of that stage, since a given PD can be shared read-only with many VMs (with limited performance impact in the many-reader case, especially if the readers are not hammering the same parts of the volume). Another option for truly pipeline oriented processing (assuming the underlying workers can generate data suitable for it) is to leverage something like Cloud Dataflow -- then they'll handle the work balancing for you. (note: I work on GCE)
  - i000 11y ago
    I have to say that PDs attached read-only to multiple VMs is a very important part of my pipelines. And also a distinguishing feature of GCE. Clound Dataflow on the other hand is difficult to use in practice - one of the main reasons is the large footprint of intermediate files (sometimes 10x the size of the input) Currently we are running monolithic pipelines on the cloud where one instance does the job from start to finish (if it get's preempted - I restart). I plan the next version to be in some way "split-apart" but have not found a good strategy/tooling - any advice?
    - jkff 11y ago
      Hi! I'm an engineer on the Dataflow team. I'd love to understand the issue you mentioned about size of intermediate files, and help make Dataflow work well for you. Could you send a note to dataflow-feedback@google.com and elaborate?