4 ms·

At first glance I'm not super into this... this seems like a bunch of opaque, hard-to-use utilities wrapped around already opaque, hard-to-use utilities (like G

At first glance I'm not super into this... this seems like a bunch of opaque, hard-to-use utilities wrapped around already opaque, hard-to-use utilities (like GATK). What am I getting for my effort? I'd rather just use the low-level tool directly. This seems more a way to pipe people into using Google Cloud Services for storing data than it seems a useful way to do bioinformatics. As with most of these cloud-genomics things, it's not useful to have canned analyses that I can't extend (for example, I am so far from giving a shit about transition/transversion ratios and would never think to store that information in my variant table) What's the big plus?

i000 11y ago
I agree the API is at this point is rather limited and it is difficult to use it in a way that would conveniently or efficiently replace any part of a genomic pipeline. But I think google is onto something. If they manage to deconstruct BAM files and store them in "format-free" way that will allow search, aggregation, and computation/reasoning across hundreds of thousands of patients/samples it may accommodate the next big wave of personal genomics data.
- davidglazer 11y ago
  I strongly agree with the idea of deconstructing legacy files (BAM, VCF, etc.) and moving to an API-centric (vs. file-format-centric) world. The Global Alliance for Genomics and Health (GA4GH.org) is working on that, with good early success -- for example, IGV (genome browser) can now work with either files or standard APIs, as can Picard (one of the popular suites of genome analysis tools). See https://www.youtube.com/watch?v=d8EvXtz2uiA https://www.youtube.com/watch?v=d8EvXtz2uiA for details on the Picard support. Disclaimer: I work on Google Genomics, and on the GA4GH APIs.
davidglazer 11y ago
We've found that different users want different levels of wrapping -- some (like you) are experts on the low-level tools, and choose to use our low-level compute, storage, and analysis services directly -- I'm biased, but I believe Google Cloud's low-level services have features, like preemptible VMs and nearline storage, that are a great fit for working with genomic data. Others want gateways into bioinformatics-friendly environments like R Studio and iPython; others want wrapped packaged services; others want point-and-click UIs ,etc. Disclaimer: I work on Google Genomics.
- PascLeRasc 11y ago
  I'm really excited about moving to R/Python computing for my bioinformatics work, it's a much better workflow than the typical method of emailing matlab files to an in-house Xeon workstation, and it'll be easier to virtualize. This is something I've seen in dozens of labs and there's a huge amount of room to streamline workflows. Disclaimer: I work in a neurobio lab but I wish I worked for Google Genomics.
- laurencerowe 11y ago
  The problem with spot market and preemptible VMs is that much of the existing bioinformatics code often has run times of days to weeks with current sequencing depths. What's really needed is a way to have a VM suspend instead of shutdown when costs get high so it can resume once prices go down.
  - jghn 11y ago
    It depends. If you picture an entire pipeline what often happens is that you end up with both long and short running tasks. Tuning your pipeline to an environment like this involves figuring out what can go where. As an example we turn around a whole genome in just over a day but that's comprised of hundreds of individual units of work
    - jsolson 11y ago
      Assuming you're running multiple pipelines at once you can add robustness in this sort of scenario by running the long-running tasks on guaranteed VMs (complete with live migration on GCE) and the short-running tasks on cheap preemptible VMs. Another features that comes to mind for enabling this sort of work is GCE's support for attaching a persistent disk volume to multiple VMs (read-write for one, read-only for others) -- have each pipeline stage consume from a read-only PD containing the output of the previous stage and write its results to a PD shared with the VM(s) that handle the next stage. Have the VMs communicate with each other for control plane operations ("hey, this directory contains the complete results of stage <x>, one of you stage <y> workers can pick it up" or "I have completed processing the stage <x> outputs for job <5>; they can be removed or archived to GCS."). This strategy also allows independent scaling of the workers responsible for each stage based on the computational requirements of that stage, since a given PD can be shared read-only with many VMs (with limited performance impact in the many-reader case, especially if the readers are not hammering the same parts of the volume). Another option for truly pipeline oriented processing (assuming the underlying workers can generate data suitable for it) is to leverage something like Cloud Dataflow -- then they'll handle the work balancing for you. (note: I work on GCE)
aBioGuy 11y ago
Agreed. I'm more interested in the cheap semi-quick data storage ($0.01 / GB / month) than any of the workflows they offer. I'm also not interested in learning a new syntax to replace what I can do with UNIX / python / R (ggplot2) already.
- eggie 11y ago
  If storage is cheap enough, and abstract, we can expect there to be a thriving ecosystem of tools to apply to it. They don't even need to be embedded in a single platform, but could be distributed, streaming, and on-demand. Here are two examples from the iobio project. A heads-up display describing a BAM (alignment) file streaming between two remote servers: http://bam.iobio.io/?bam=http://s3.amazonaws.com/iobio/NA12878/NA12878.autsome.bam&region=1 http://bam.iobio.io/?bam=http://s3.amazonaws.com/iobio/NA128... Streams another BAM from the 1000 Genomes project through two variant callers, with dynamically-updated statistics about the results. You can modify the variant quality threshold for each caller and the results are recomputed in real time: http://iobio.io/demo/variantcomparer/ http://iobio.io/demo/variantcomparer/ Edit: Very nice visualization of interesting statistics from a VCF file, provided by URL or uploaded: http://vcf.iobio.io/?vcf=http://s3.amazonaws.com/vcf.files/ALL.wgs.phase3_shapeit2_mvncall_integrated_v5.20130502.sites.vcf.gz http://vcf.iobio.io/?vcf=http://s3.amazonaws.com/vcf.files/A...

i000 11y ago

I agree the API is at this point is rather limited and it is difficult to use it in a way that would conveniently or efficiently replace any part of a genomic pipeline. But I think google is onto something. If they manage to deconstruct BAM files and store them in "format-free" way that will allow search, aggregation, and computation/reasoning across hundreds of thousands of patients/samples it may accommodate the next big wave of personal genomics data.

davidglazer 11y ago
I strongly agree with the idea of deconstructing legacy files (BAM, VCF, etc.) and moving to an API-centric (vs. file-format-centric) world. The Global Alliance for Genomics and Health (GA4GH.org) is working on that, with good early success -- for example, IGV (genome browser) can now work with either files or standard APIs, as can Picard (one of the popular suites of genome analysis tools). See https://www.youtube.com/watch?v=d8EvXtz2uiA https://www.youtube.com/watch?v=d8EvXtz2uiA for details on the Picard support. Disclaimer: I work on Google Genomics, and on the GA4GH APIs.

davidglazer 11y ago

We've found that different users want different levels of wrapping -- some (like you) are experts on the low-level tools, and choose to use our low-level compute, storage, and analysis services directly -- I'm biased, but I believe Google Cloud's low-level services have features, like preemptible VMs and nearline storage, that are a great fit for working with genomic data. Others want gateways into bioinformatics-friendly environments like R Studio and iPython; others want wrapped packaged services; others want point-and-click UIs ,etc. Disclaimer: I work on Google Genomics.

PascLeRasc 11y ago
I'm really excited about moving to R/Python computing for my bioinformatics work, it's a much better workflow than the typical method of emailing matlab files to an in-house Xeon workstation, and it'll be easier to virtualize. This is something I've seen in dozens of labs and there's a huge amount of room to streamline workflows. Disclaimer: I work in a neurobio lab but I wish I worked for Google Genomics.
laurencerowe 11y ago
The problem with spot market and preemptible VMs is that much of the existing bioinformatics code often has run times of days to weeks with current sequencing depths. What's really needed is a way to have a VM suspend instead of shutdown when costs get high so it can resume once prices go down.
- jghn 11y ago
  It depends. If you picture an entire pipeline what often happens is that you end up with both long and short running tasks. Tuning your pipeline to an environment like this involves figuring out what can go where. As an example we turn around a whole genome in just over a day but that's comprised of hundreds of individual units of work
  - jsolson 11y ago
    Assuming you're running multiple pipelines at once you can add robustness in this sort of scenario by running the long-running tasks on guaranteed VMs (complete with live migration on GCE) and the short-running tasks on cheap preemptible VMs. Another features that comes to mind for enabling this sort of work is GCE's support for attaching a persistent disk volume to multiple VMs (read-write for one, read-only for others) -- have each pipeline stage consume from a read-only PD containing the output of the previous stage and write its results to a PD shared with the VM(s) that handle the next stage. Have the VMs communicate with each other for control plane operations ("hey, this directory contains the complete results of stage <x>, one of you stage <y> workers can pick it up" or "I have completed processing the stage <x> outputs for job <5>; they can be removed or archived to GCS."). This strategy also allows independent scaling of the workers responsible for each stage based on the computational requirements of that stage, since a given PD can be shared read-only with many VMs (with limited performance impact in the many-reader case, especially if the readers are not hammering the same parts of the volume). Another option for truly pipeline oriented processing (assuming the underlying workers can generate data suitable for it) is to leverage something like Cloud Dataflow -- then they'll handle the work balancing for you. (note: I work on GCE)

aBioGuy 11y ago

Agreed. I'm more interested in the cheap semi-quick data storage ($0.01 / GB / month) than any of the workflows they offer. I'm also not interested in learning a new syntax to replace what I can do with UNIX / python / R (ggplot2) already.

eggie 11y ago
If storage is cheap enough, and abstract, we can expect there to be a thriving ecosystem of tools to apply to it. They don't even need to be embedded in a single platform, but could be distributed, streaming, and on-demand. Here are two examples from the iobio project. A heads-up display describing a BAM (alignment) file streaming between two remote servers: http://bam.iobio.io/?bam=http://s3.amazonaws.com/iobio/NA12878/NA12878.autsome.bam&region=1 http://bam.iobio.io/?bam=http://s3.amazonaws.com/iobio/NA128... Streams another BAM from the 1000 Genomes project through two variant callers, with dynamically-updated statistics about the results. You can modify the variant quality threshold for each caller and the results are recomputed in real time: http://iobio.io/demo/variantcomparer/ http://iobio.io/demo/variantcomparer/ Edit: Very nice visualization of interesting statistics from a VCF file, provided by URL or uploaded: http://vcf.iobio.io/?vcf=http://s3.amazonaws.com/vcf.files/ALL.wgs.phase3_shapeit2_mvncall_integrated_v5.20130502.sites.vcf.gz http://vcf.iobio.io/?vcf=http://s3.amazonaws.com/vcf.files/A...