7 ms·

Google Genomics: store, process, explore and share genomic data

rhema 11y ago
I don't have first experience in genetics. However, some professors I met who work with Gene's complained about the high cost of running AWS services. I don't know how much of a market there is for this, but there is some.
united893 11y ago
Google is one of the lead investors in DNANexus[1], this must suck for them. [1] - http://www.fiercebiotech.com/story/google-ventures-backs-dnanexuss-15m-round-rd-cloud-platform/2014-01-03 http://www.fiercebiotech.com/story/google-ventures-backs-dna...
- bitmapbrother 11y ago
  Google is also one of the major investors in Uber.
  - toomuchtodo 11y ago
    Google seems to be the adult who says, "Here kid, have some cash and go work out the kinks for us." Not that I blame them.
    - bitmapbrother 11y ago
      I don't see it that way. The objective of Alphabet's GV is to make as much ROI as possible regardless of what the other companies within Google are doing. So just because Google or some other business within Alphabet has a similar business that's really irrelevant to how GV will be measured. I view it more along the lines of hedging your bets. GV could win or lose with their investment and Google could win or lose with their solution, but in the end Alphabet is usually going to win.
    - beambot 11y ago
      It's more like, "We know this is important. Someone will succeed. We should diversify, and we don't have so much hubris as to believe we will always be better than everyone else."
i000 11y ago
I work at an academic institution that generates and analyzes large amounts of genomic data (clinical sequencing of cancer patients). We were really impressed with the Google cloud and executed several large proof-of-concept projects on it. Honestly, being able to spin up 1000 machines to process 10s of thousands of samples within hours (using cheap preemptible machines) is a huge step forward. The biggest challenge we face is data protection. Sequencing data is not considered protected health information so theoretically HIPAA regulations do not apply, but this may change in future so nobody is willing to take responsibility. If sequencing data is considered PHI, HIPAA requires a BAA between the institution and Google. The process of getting these in place was so convoluted and difficult that it was just easier to use another provider more closely associated with our institution.
- timeu 11y ago
  I am wondering if you did a cost analysis. Last time I checked a dedicated on-premise HPC was still cheaper than using the cloud. Actually the CPU hours are cheaper in the cloud but data-transfer (bandwith) and storage make it more expensive (might not apply to storage anymore but AFAIK data transfer costs are still an issue). What is your experience ?
  - i000 11y ago
    Of course we did this, but cost is not the only consideration. Frankly, CPU-hours and TB-years are still within 5-10% of the total cost providing sequencing results to the patients. What is really important is that the cloud allows us to determine exactly our cost-per-analysis without paying for excess capacity or maintenance. What matters also is reliability, redundancy, and peak capacity, where the cloud (both Google and the other provider) win hands down.
  - boulos 11y ago
    Just a note, you don't pay for sending data to any provider (ingress) only for pulling data out / serving it (egress). You can put your GCS buckets behind Cloudflare to take advantage of "interconnect" pricing of $.04/GB. Disclaimer: I work on Compute Engine.
dlsx 11y ago
G is for Genetics, or Google, or Genomics. An Alphabet Company (TM) (we don't own our domain, so just go to google.com)
astazangasta 11y ago
At first glance I'm not super into this... this seems like a bunch of opaque, hard-to-use utilities wrapped around already opaque, hard-to-use utilities (like GATK). What am I getting for my effort? I'd rather just use the low-level tool directly. This seems more a way to pipe people into using Google Cloud Services for storing data than it seems a useful way to do bioinformatics. As with most of these cloud-genomics things, it's not useful to have canned analyses that I can't extend (for example, I am so far from giving a shit about transition/transversion ratios and would never think to store that information in my variant table) What's the big plus?
- i000 11y ago
  I agree the API is at this point is rather limited and it is difficult to use it in a way that would conveniently or efficiently replace any part of a genomic pipeline. But I think google is onto something. If they manage to deconstruct BAM files and store them in "format-free" way that will allow search, aggregation, and computation/reasoning across hundreds of thousands of patients/samples it may accommodate the next big wave of personal genomics data.
  - davidglazer 11y ago
    I strongly agree with the idea of deconstructing legacy files (BAM, VCF, etc.) and moving to an API-centric (vs. file-format-centric) world. The Global Alliance for Genomics and Health (GA4GH.org) is working on that, with good early success -- for example, IGV (genome browser) can now work with either files or standard APIs, as can Picard (one of the popular suites of genome analysis tools). See https://www.youtube.com/watch?v=d8EvXtz2uiA https://www.youtube.com/watch?v=d8EvXtz2uiA for details on the Picard support. Disclaimer: I work on Google Genomics, and on the GA4GH APIs.
- davidglazer 11y ago
  We've found that different users want different levels of wrapping -- some (like you) are experts on the low-level tools, and choose to use our low-level compute, storage, and analysis services directly -- I'm biased, but I believe Google Cloud's low-level services have features, like preemptible VMs and nearline storage, that are a great fit for working with genomic data. Others want gateways into bioinformatics-friendly environments like R Studio and iPython; others want wrapped packaged services; others want point-and-click UIs ,etc. Disclaimer: I work on Google Genomics.
vonklaus 11y ago
Larry Page has been talking about this for a while. Interesting to see what alphabet does
FreedomToCreate 11y ago
So whose first thought was "So this is how google is going to accelerate there own internal genetics research".
awinter-py 11y ago
is there a privacy policy? the only search results are G's standard policy and scarewords like 'G's expertise in privacy'. Are they mining this cross-client?
- davidglazer 11y ago
  Short answer: it's your data. Period. You decide who can see the data you store, and the default is private. That's spelled out in lots of words in the legal terms somewhere; sorry it wasn't more clear on the website. Disclaimer: I work for Google Genomics.
  - awinter-py 11y ago
    if this is subpoena-proof like you imply, I'm going to start storing all my data in this service.
  - hxrts 11y ago
    This needs to be front and center. Data protections / PHI are the #1 issue for academic groups in the field. – bioinformatician
- username223 11y ago
  There seems to be: https://cloud.google.com/terms/ https://cloud.google.com/terms/ . Section 5.2 seems relevant, but I'm not a lawyer, so I don't know whether it means much.
theophrastus 11y ago
The majority of this data is generated via public funding. So over the crusty decades there have arisen public databases of notably varied quality, curation, and stability; e.g. GenBank, NCBI, SwissProt, EMBL, PIR, SBKB, PDB, ...quite a long list. (I've been involved in direct dealings with several of these sites and sometimes the histories and policies get very opaque) Finally, the question: this is public data, (sometimes held in confidence but technically only for a "short" period), google is a private company, why should we place our trust in a for-profit organization to keep our public data publicly available without charging access fees? Admitting freely that having dealt for decades with the old model, i do have an open mind about this approach.
- roye 11y ago
  I don't think their intent is to replace existing database servers. My impression is that they have copied the data onto their servers to make analysis on their cloud service more attractive. For example, the fact that the 1000 Genomes data is there saves researchers from having to go to the trouble of uploading it themselves.
  - kuschku 11y ago
    > I don't think their intent is to replace existing database servers. We thought the same with Freebase and the Knowledge Graph. It will happen again.
  - jghn 11y ago
    Having worked with them a bit my impression is that at a high level their intent is to get people to spend money using google cloud services. With that in mind it is not in their interest to annoy half of the field by playing shenanigans with things like data ownership
halite 11y ago
Google is getting lot better at design but I can't help notice little things. Like the "Get Started" section has three boxes with first one slightly bigger, how hard was it to make all of them same size (http://i.imgur.com/eGpjRDv.png http://i.imgur.com/eGpjRDv.png). It is minute detail I know.
jamesblonde 11y ago
Here's some good videos on the latest research on Hadoop and NGS data: https://www.youtube.com/playlist?list=PL5oElY7F-znCU_Ppb7YWJ8jifqbDttxOt https://www.youtube.com/playlist?list=PL5oElY7F-znCU_Ppb7YWJ...
gabeiscoding 11y ago
While I think it's great to have Google putting their weight behind standardization efforts like Global Alliance for Genomic Health (GA4GH), I really don't get the need to replace VCF and BAM files with API calls. Ultimately, the "hard part" about genomics is not big-data requiring Spanner and BigTable to get anything done. I actually wrote a blog post about this this week: http://blog.goldenhelix.com/grudy/genomic-data-is-big-data-but-that-is-not-the-hard-part/ http://blog.goldenhelix.com/grudy/genomic-data-is-big-data-b... Both BAM and VCF files can be hosted through a plain HTTP file-server and be meaningfully queried through their BAI/TBI indexes. Visualization tools like our GenomeBrowse or the Broad's IGV can already read S3 hosted genomic files directly without having an API layer and very efficiently (gzip compressed blocks of binary data). So, I see the translation of the exact same data into API-only accessible storage system, where I can't download the VCF and do quick and iterative analysis on it more of a downside that plus. Disclaimer: I build variant interpretation software for NGS data at Golden Helix. Our customers are often small clinical labs who size of data and volume are not driving them to the cloud.
- flux988 11y ago
  How do you think this compares against http://basepair.io http://basepair.io ?
  - gabeiscoding 11y ago
    Looks great, but I can't comment more as I haven't used it. It looks to be solving the same problems as DNAnexus, Seven Bridges, BaseSpace etc as a way to wrap open source tools in more user-friendly ways. But it's orchestrating the production of smaller set of data that still needs the next step of human interpretation, report writing, family-aware algorithms and most complex annotations (the problem space Golden Helix is in). In other words, the automatable bits that is not the hard part that I mentioned in my blog post.

rhema 11y ago

I don't have first experience in genetics. However, some professors I met who work with Gene's complained about the high cost of running AWS services. I don't know how much of a market there is for this, but there is some.

united893 11y ago

Google is one of the lead investors in DNANexus[1], this must suck for them. [1] - http://www.fiercebiotech.com/story/google-ventures-backs-dnanexuss-15m-round-rd-cloud-platform/2014-01-03 http://www.fiercebiotech.com/story/google-ventures-backs-dna...

bitmapbrother 11y ago
Google is also one of the major investors in Uber.
- toomuchtodo 11y ago
  Google seems to be the adult who says, "Here kid, have some cash and go work out the kinks for us." Not that I blame them.
  - bitmapbrother 11y ago
    I don't see it that way. The objective of Alphabet's GV is to make as much ROI as possible regardless of what the other companies within Google are doing. So just because Google or some other business within Alphabet has a similar business that's really irrelevant to how GV will be measured. I view it more along the lines of hedging your bets. GV could win or lose with their investment and Google could win or lose with their solution, but in the end Alphabet is usually going to win.
  - beambot 11y ago
    It's more like, "We know this is important. Someone will succeed. We should diversify, and we don't have so much hubris as to believe we will always be better than everyone else."

i000 11y ago

I work at an academic institution that generates and analyzes large amounts of genomic data (clinical sequencing of cancer patients). We were really impressed with the Google cloud and executed several large proof-of-concept projects on it. Honestly, being able to spin up 1000 machines to process 10s of thousands of samples within hours (using cheap preemptible machines) is a huge step forward. The biggest challenge we face is data protection. Sequencing data is not considered protected health information so theoretically HIPAA regulations do not apply, but this may change in future so nobody is willing to take responsibility. If sequencing data is considered PHI, HIPAA requires a BAA between the institution and Google. The process of getting these in place was so convoluted and difficult that it was just easier to use another provider more closely associated with our institution.

timeu 11y ago
I am wondering if you did a cost analysis. Last time I checked a dedicated on-premise HPC was still cheaper than using the cloud. Actually the CPU hours are cheaper in the cloud but data-transfer (bandwith) and storage make it more expensive (might not apply to storage anymore but AFAIK data transfer costs are still an issue). What is your experience ?
- i000 11y ago
  Of course we did this, but cost is not the only consideration. Frankly, CPU-hours and TB-years are still within 5-10% of the total cost providing sequencing results to the patients. What is really important is that the cloud allows us to determine exactly our cost-per-analysis without paying for excess capacity or maintenance. What matters also is reliability, redundancy, and peak capacity, where the cloud (both Google and the other provider) win hands down.
- boulos 11y ago
  Just a note, you don't pay for sending data to any provider (ingress) only for pulling data out / serving it (egress). You can put your GCS buckets behind Cloudflare to take advantage of "interconnect" pricing of $.04/GB. Disclaimer: I work on Compute Engine.

dlsx 11y ago

G is for Genetics, or Google, or Genomics. An Alphabet Company (TM) (we don't own our domain, so just go to google.com)

astazangasta 11y ago

At first glance I'm not super into this... this seems like a bunch of opaque, hard-to-use utilities wrapped around already opaque, hard-to-use utilities (like GATK). What am I getting for my effort? I'd rather just use the low-level tool directly. This seems more a way to pipe people into using Google Cloud Services for storing data than it seems a useful way to do bioinformatics. As with most of these cloud-genomics things, it's not useful to have canned analyses that I can't extend (for example, I am so far from giving a shit about transition/transversion ratios and would never think to store that information in my variant table) What's the big plus?

i000 11y ago
I agree the API is at this point is rather limited and it is difficult to use it in a way that would conveniently or efficiently replace any part of a genomic pipeline. But I think google is onto something. If they manage to deconstruct BAM files and store them in "format-free" way that will allow search, aggregation, and computation/reasoning across hundreds of thousands of patients/samples it may accommodate the next big wave of personal genomics data.
- davidglazer 11y ago
  I strongly agree with the idea of deconstructing legacy files (BAM, VCF, etc.) and moving to an API-centric (vs. file-format-centric) world. The Global Alliance for Genomics and Health (GA4GH.org) is working on that, with good early success -- for example, IGV (genome browser) can now work with either files or standard APIs, as can Picard (one of the popular suites of genome analysis tools). See https://www.youtube.com/watch?v=d8EvXtz2uiA https://www.youtube.com/watch?v=d8EvXtz2uiA for details on the Picard support. Disclaimer: I work on Google Genomics, and on the GA4GH APIs.
davidglazer 11y ago
We've found that different users want different levels of wrapping -- some (like you) are experts on the low-level tools, and choose to use our low-level compute, storage, and analysis services directly -- I'm biased, but I believe Google Cloud's low-level services have features, like preemptible VMs and nearline storage, that are a great fit for working with genomic data. Others want gateways into bioinformatics-friendly environments like R Studio and iPython; others want wrapped packaged services; others want point-and-click UIs ,etc. Disclaimer: I work on Google Genomics.

vonklaus 11y ago

Larry Page has been talking about this for a while. Interesting to see what alphabet does

FreedomToCreate 11y ago

So whose first thought was "So this is how google is going to accelerate there own internal genetics research".

awinter-py 11y ago

is there a privacy policy? the only search results are G's standard policy and scarewords like 'G's expertise in privacy'. Are they mining this cross-client?

davidglazer 11y ago
Short answer: it's your data. Period. You decide who can see the data you store, and the default is private. That's spelled out in lots of words in the legal terms somewhere; sorry it wasn't more clear on the website. Disclaimer: I work for Google Genomics.
- awinter-py 11y ago
  if this is subpoena-proof like you imply, I'm going to start storing all my data in this service.
- hxrts 11y ago
  This needs to be front and center. Data protections / PHI are the #1 issue for academic groups in the field. – bioinformatician
username223 11y ago
There seems to be: https://cloud.google.com/terms/ https://cloud.google.com/terms/ . Section 5.2 seems relevant, but I'm not a lawyer, so I don't know whether it means much.

theophrastus 11y ago

The majority of this data is generated via public funding. So over the crusty decades there have arisen public databases of notably varied quality, curation, and stability; e.g. GenBank, NCBI, SwissProt, EMBL, PIR, SBKB, PDB, ...quite a long list. (I've been involved in direct dealings with several of these sites and sometimes the histories and policies get very opaque) Finally, the question: this is public data, (sometimes held in confidence but technically only for a "short" period), google is a private company, why should we place our trust in a for-profit organization to keep our public data publicly available without charging access fees? Admitting freely that having dealt for decades with the old model, i do have an open mind about this approach.

roye 11y ago
I don't think their intent is to replace existing database servers. My impression is that they have copied the data onto their servers to make analysis on their cloud service more attractive. For example, the fact that the 1000 Genomes data is there saves researchers from having to go to the trouble of uploading it themselves.
- kuschku 11y ago
  > I don't think their intent is to replace existing database servers. We thought the same with Freebase and the Knowledge Graph. It will happen again.
- jghn 11y ago
  Having worked with them a bit my impression is that at a high level their intent is to get people to spend money using google cloud services. With that in mind it is not in their interest to annoy half of the field by playing shenanigans with things like data ownership

halite 11y ago

Google is getting lot better at design but I can't help notice little things. Like the "Get Started" section has three boxes with first one slightly bigger, how hard was it to make all of them same size (http://i.imgur.com/eGpjRDv.png http://i.imgur.com/eGpjRDv.png). It is minute detail I know.

jamesblonde 11y ago

Here's some good videos on the latest research on Hadoop and NGS data: https://www.youtube.com/playlist?list=PL5oElY7F-znCU_Ppb7YWJ8jifqbDttxOt https://www.youtube.com/playlist?list=PL5oElY7F-znCU_Ppb7YWJ...

gabeiscoding 11y ago

While I think it's great to have Google putting their weight behind standardization efforts like Global Alliance for Genomic Health (GA4GH), I really don't get the need to replace VCF and BAM files with API calls. Ultimately, the "hard part" about genomics is not big-data requiring Spanner and BigTable to get anything done. I actually wrote a blog post about this this week: http://blog.goldenhelix.com/grudy/genomic-data-is-big-data-but-that-is-not-the-hard-part/ http://blog.goldenhelix.com/grudy/genomic-data-is-big-data-b... Both BAM and VCF files can be hosted through a plain HTTP file-server and be meaningfully queried through their BAI/TBI indexes. Visualization tools like our GenomeBrowse or the Broad's IGV can already read S3 hosted genomic files directly without having an API layer and very efficiently (gzip compressed blocks of binary data). So, I see the translation of the exact same data into API-only accessible storage system, where I can't download the VCF and do quick and iterative analysis on it more of a downside that plus. Disclaimer: I build variant interpretation software for NGS data at Golden Helix. Our customers are often small clinical labs who size of data and volume are not driving them to the cloud.

flux988 11y ago
How do you think this compares against http://basepair.io http://basepair.io ?
- gabeiscoding 11y ago
  Looks great, but I can't comment more as I haven't used it. It looks to be solving the same problems as DNAnexus, Seven Bridges, BaseSpace etc as a way to wrap open source tools in more user-friendly ways. But it's orchestrating the production of smaller set of data that still needs the next step of human interpretation, report writing, family-aware algorithms and most complex annotations (the problem space Golden Helix is in). In other words, the automatable bits that is not the hard part that I mentioned in my blog post.