5 ms·

You know... I'm not so sure I completely agree. I mean, yes, the analysis is the bottleneck. Since about 2012 it takes more time to perform the analysis than it

by mbreese 11y ago

You know... I'm not so sure I completely agree. I mean, yes, the analysis is the bottleneck. Since about 2012 it takes more time to perform the analysis than it does to generate the data. And even though all of the raw data -> BAM -> VCF or counts steps is automated, nothing after that is (or can be, since that's where the science happens). But, the $1k genome really is significant. Costs matter. Researchers will only sequence what they can afford. So, when sequencing a genome was $10k, there weren't too many 30X whole genomes sequenced. Instead we had exome sequencing of individuals or low-depth sequencing of populations. Fast forward a few years, and now the costs have gone down significantly. Because of this, we are now starting to see more whole-genomes. Now, the costs of to sequence a whole-genome at 30X is about the same as targeted exome sequencing (due to the extra capture costs). If it's almost cheaper to just sequence the whole-genome and skip any extra capture probe biases, why not just sequence the whole thing? This is particularly true when looking at complex diseases where the underlying cause(s) may not be in the coding part of a gene. So, in that regard, cost really was a bottleneck. With all of the new whole-genomes available, we can start to ask even tougher questions. And with more data available from more people, we can start to get some real confidence behind our predictions. But, you are right in that it does all lead back to the data-analysis bottleneck. Before we had as more data than we could deal with. Now, we can process data faster, so that's good. But we also have significantly more data to process, so in the end, it's about even. The other issue then becomes data storage. What do you do when you have a few hundred terabytes or a petabyte of sequencing data? S3 can get pretty expensive at that scale. Instead, it's probably just cheaper to store the DNA in a freezer, and if you ever need the data - just resequence it. Edit: reworded a bit

jghn 11y ago
Compute time and storage costs are also bottlenecks, so even though the base calling through VCF is automated large genome centers struggle to keep up with the output of the new sequencers
- gbveiuwlbiu 11y ago
  When you say compute time, do you have in mind purely quantitative processing/analytics, or streaming and data transfer? Is there any way you could describe a sample "experiment" and the associated storage costs and compute time, just a paper-and-pencil estimation perhaps?
  - zitterbewegung 11y ago
    I went to a python talk where the guy did genomics . I asked him how long his program would take and he said I'll tell you when it's done. This means his analysis programs can take days to weeks and he really didn't know how long the current one can take .
    - akuchling 11y ago
      Possibly C. Titus Brown? Here's his PyCon 2014 talk, "Data intensive biology in the cloud": https://www.youtube.com/watch?v=RCRbfKK57X8 https://www.youtube.com/watch?v=RCRbfKK57X8
- mbreese 11y ago
  Right - but we used to struggle with the output of the last generation of sequencers too. Computing capacity and storage have scaled at about the same rate. Or perhaps it's more accurate to say that the costs have come down, so we can throw more iron at the problem - these are embarrassingly parallel problems... As an example, the last sequencing analysis cluster I setup (~2010) ran with 2x bonded gigabit networks between our compute nodes and storage and had 10TB of network storage. The newest one is running 10GigE, has 200+TB of storage, and is cheaper. And we are a small shop in this field.
a_bonobo 11y ago
>With all of the new whole-genomes available, we can start to ask even tougher questions. And with more data available from more people, we can start to get some real confidence behind our predictions. What I'd like to add to this is that we can spend more time and less money on "hobby" projects! These are the projects you don't really have explicit funding for, but you do anyway, using leftover money from various pots [1]. We have two PhD students who work on "hobby" projects. These projects include "outlandish" questions or "basic" research the funding committees didn't see the reason to fund. In our case, these two PhD students work with less studied plant organisms no-one else really looked into so far, but which have some interesting attributes. Some of these hobbies have been fruitful (interesting novel salinity resistance proteins found), some of these less (but it's always good to populate public databases from data of less well-known organisms). Cheaper genomics allow us to do so much more of these projects, exciting times! [1] See http://www.phdcomics.com/comics.php?f=1431 http://www.phdcomics.com/comics.php?f=1431 (Of course, now the cost bottlenecks move somewhere else - it has become cheap to sequence an entire plant population, but it's still expensive to grow & especially phenotype that population - if you're into computer vision and are looking for a new project, automatic plant phenotyping is a field that still has a lot of work to do!)

by mbreese 11y ago

jghn 11y ago

Compute time and storage costs are also bottlenecks, so even though the base calling through VCF is automated large genome centers struggle to keep up with the output of the new sequencers

gbveiuwlbiu 11y ago
When you say compute time, do you have in mind purely quantitative processing/analytics, or streaming and data transfer? Is there any way you could describe a sample "experiment" and the associated storage costs and compute time, just a paper-and-pencil estimation perhaps?
- zitterbewegung 11y ago
  I went to a python talk where the guy did genomics . I asked him how long his program would take and he said I'll tell you when it's done. This means his analysis programs can take days to weeks and he really didn't know how long the current one can take .
  - akuchling 11y ago
    Possibly C. Titus Brown? Here's his PyCon 2014 talk, "Data intensive biology in the cloud": https://www.youtube.com/watch?v=RCRbfKK57X8 https://www.youtube.com/watch?v=RCRbfKK57X8
mbreese 11y ago
Right - but we used to struggle with the output of the last generation of sequencers too. Computing capacity and storage have scaled at about the same rate. Or perhaps it's more accurate to say that the costs have come down, so we can throw more iron at the problem - these are embarrassingly parallel problems... As an example, the last sequencing analysis cluster I setup (~2010) ran with 2x bonded gigabit networks between our compute nodes and storage and had 10TB of network storage. The newest one is running 10GigE, has 200+TB of storage, and is cheaper. And we are a small shop in this field.

a_bonobo 11y ago

>With all of the new whole-genomes available, we can start to ask even tougher questions. And with more data available from more people, we can start to get some real confidence behind our predictions. What I'd like to add to this is that we can spend more time and less money on "hobby" projects! These are the projects you don't really have explicit funding for, but you do anyway, using leftover money from various pots [1]. We have two PhD students who work on "hobby" projects. These projects include "outlandish" questions or "basic" research the funding committees didn't see the reason to fund. In our case, these two PhD students work with less studied plant organisms no-one else really looked into so far, but which have some interesting attributes. Some of these hobbies have been fruitful (interesting novel salinity resistance proteins found), some of these less (but it's always good to populate public databases from data of less well-known organisms). Cheaper genomics allow us to do so much more of these projects, exciting times! [1] See http://www.phdcomics.com/comics.php?f=1431 http://www.phdcomics.com/comics.php?f=1431 (Of course, now the cost bottlenecks move somewhere else - it has become cheap to sequence an entire plant population, but it's still expensive to grow & especially phenotype that population - if you're into computer vision and are looking for a new project, automatic plant phenotyping is a field that still has a lot of work to do!)