7 ms·

People say this all the time, but with some of the most common applications of high throughput sequencing there are very good canned solutions (using open sourc

People say this all the time, but with some of the most common applications of high throughput sequencing there are very good canned solutions (using open source software) that you can pay for. DNAnexus, Seven Bridges, and Illumina BaseSpace all provide cloud storage and analysis. Unless you are doing a custom prep for your sequencing one of these probably has an analysis solution for you.

BKPetkov 11y ago
Is the time required to upload data to the cloud ever a problem with these solutions? Of course, it depends on what you are trying to do, but suppose you were working with thousands of genomes?
- adenadel 11y ago
  Sure, all the time. Network and I/O are the biggest blockers for sequence analysis. For any organization that is working with thousands of genomes they probably have their own compute resources. I know of at least one organization who is currently sending thousands of genomes to the cloud for analysis, so it's certainly feasible to some extent.
  - gbveiuwlbiu 11y ago
    Any chance we could follow up on this? I'm conducting research in this space and would love to have a chat regarding barriers to such large-scale analysis. If you could message me at hngenometemp@forward.cat that would be terrific!
- epistasis 11y ago
  The sequencers can stream to a data analysis center as the data is being generated. It takes a 100mbit stream/$1M of sequencing capital, so network connectivity to transfer to a data center is a tiny tiny cost of the whole ordeal. However, paying for AWS storage is pretty prohibitive, unless you're at a small scale. So big centers will build their own storage facilities. The small data producers like the ones that the thread author talks about can use often use AWS more cost efficiently than building a compute cluster. However, they need to budget for that, which is not always thought of. They may also need to fight their institute's core center so that they can use DNANexus.
  - laurencerowe 11y ago
    S3 storage is pretty cheap, it's the data egress that really costs. For academic centers though there is often an incentive to move things in house due to different treatment for capital expenditures and the opportunity to externalize some of your costs from your grant onto central services.
    - epistasis 11y ago
      Data transfer is less than a single year of Glacier storage, so while it's pricy I wouldn't egress a major portion of the cost. Keeping this data for less than 5-10 years is pretty questionable, since it's so expensive to generate. Eventually it may be cheaper to store the DNA and resequence when if it needs to be looked at again. However, if you're doing petabytes of storage, it's going to me much more economical to have your own storage and compute than to use AWS. Particularly at the rate that academic centers pay for sysadmins.
      - laurencerowe 11y ago
        Running a public data portal our egress is higher than our storage costs. (We now proxy downloads through a direct connect to our university network...) Remember to account for future reductions in storage costs. S3 has come down from $0.1500/GB month in 2010 to $0.0300/GB month today. And the recently introduced infrequent access storage tier is under half that again at $0.0125/GB month. It's now significantly cheaper to use S3/Azure/Google than running the storage ourselves.
jerven 11y ago
Analysis is custom if you are doing genomics right at this time. Either you are doing something pre-clinical, and you better understand what you are doing i.e. analysis in a black box is not ok. Or you are not dealing with human stuff at all and then there is no analysis in a box on the market. There are tools with GUI's but they don't help that much if you don't understand the underlying systems.
- adenadel 11y ago
  I don't agree with you. I would argue that Genomics England is doing genomics right with the 100k genomes project and they certainly aren't doing custom analysis. Their pipeline is standard through variant calling and annotation. The WGS pipelines are pretty well established at this time. What you do with your VCFs is what can be customized. There is no reason for a small lab to write their own aligner or variant caller. The most that many do is write wrappers to create "customized pipelines", which is really just reinventing the wheel. EDIT: I want to clarify my mindset. In the environment that I work in there are three stages of analysis: primary, secondary, and tertiary. We mostly ignore the tertiary stage (I'm not actually doing science). Primary analysis is on instrument (base calling), secondary analysis takes you through variant calls and annotation, and tertiary analysis is where the real science happens. When I'm talking about canned analyses I'm talking about what I refer to as secondary analysis. Tertiary analysis is a very hard problem and will often require custom solutions. In my mind I'm separating bioinformatic analysis (secondary analysis) with interpretation (tertiary analysis).
  - jerven 11y ago
    Getting the variants out is where the analysis starts. What does the variation mean, which ones correlate with a detectable phenotype. Which ones are affecting regulatory systems. Genomics England is brilliant because it has good access to excellent NHS data which it will use to do a lot of analysis. Which its not getting off the shelf but a lot of talented clincal+bio+informaticians are working hard on that. Or in other words getting the variants to the human genome reference is just the start of the analysis not the end. i.e. I am more worried about the end of the pipeline than the contents of them. I suspected as much on the 1,2,3 level. But even 2 is not really fixed yet. There are great off the shelf tools but even the concept of variation calling against a reference linear genome is not perfect yet especially with the alt_loci business. Which driving the variation graph work in the GA4GH projects.
searine 11y ago
>DNAnexus, Seven Bridges, and Illumina BaseSpace all provide cloud storage and analysis. "Analysis". There is a difference between meaningful analysis which answers long standing questions of biology, and running a program which calls SNPs and spits out genome stats. The hard part is knowing what questions to ask, and how to interpret the answers when you get them. That takes years of training to develop a critical mind.
- rcthompson 11y ago
  This is absolutely true. I've done differential expression analysis on many RNA-seq and microarray datasets, and despite the fact that the basic pipeline for such analysis is quite straightforward and easily automated, every analysis is asking different biological questions and has different confounding factors that need to be addressed, so the analysis of each dataset ends up being quite different from all the others.
dikdik 11y ago
All of these will give you some answers. The reality is there is SO MUCH MORE INFORMATION to be gleaned from our genomes that these solutions do not account for.

BKPetkov 11y ago

Is the time required to upload data to the cloud ever a problem with these solutions? Of course, it depends on what you are trying to do, but suppose you were working with thousands of genomes?

adenadel 11y ago
Sure, all the time. Network and I/O are the biggest blockers for sequence analysis. For any organization that is working with thousands of genomes they probably have their own compute resources. I know of at least one organization who is currently sending thousands of genomes to the cloud for analysis, so it's certainly feasible to some extent.
- gbveiuwlbiu 11y ago
  Any chance we could follow up on this? I'm conducting research in this space and would love to have a chat regarding barriers to such large-scale analysis. If you could message me at hngenometemp@forward.cat that would be terrific!
epistasis 11y ago
The sequencers can stream to a data analysis center as the data is being generated. It takes a 100mbit stream/$1M of sequencing capital, so network connectivity to transfer to a data center is a tiny tiny cost of the whole ordeal. However, paying for AWS storage is pretty prohibitive, unless you're at a small scale. So big centers will build their own storage facilities. The small data producers like the ones that the thread author talks about can use often use AWS more cost efficiently than building a compute cluster. However, they need to budget for that, which is not always thought of. They may also need to fight their institute's core center so that they can use DNANexus.
- laurencerowe 11y ago
  S3 storage is pretty cheap, it's the data egress that really costs. For academic centers though there is often an incentive to move things in house due to different treatment for capital expenditures and the opportunity to externalize some of your costs from your grant onto central services.
  - epistasis 11y ago
    Data transfer is less than a single year of Glacier storage, so while it's pricy I wouldn't egress a major portion of the cost. Keeping this data for less than 5-10 years is pretty questionable, since it's so expensive to generate. Eventually it may be cheaper to store the DNA and resequence when if it needs to be looked at again. However, if you're doing petabytes of storage, it's going to me much more economical to have your own storage and compute than to use AWS. Particularly at the rate that academic centers pay for sysadmins.
    - laurencerowe 11y ago
      Running a public data portal our egress is higher than our storage costs. (We now proxy downloads through a direct connect to our university network...) Remember to account for future reductions in storage costs. S3 has come down from $0.1500/GB month in 2010 to $0.0300/GB month today. And the recently introduced infrequent access storage tier is under half that again at $0.0125/GB month. It's now significantly cheaper to use S3/Azure/Google than running the storage ourselves.

jerven 11y ago

Analysis is custom if you are doing genomics right at this time. Either you are doing something pre-clinical, and you better understand what you are doing i.e. analysis in a black box is not ok. Or you are not dealing with human stuff at all and then there is no analysis in a box on the market. There are tools with GUI's but they don't help that much if you don't understand the underlying systems.

adenadel 11y ago
I don't agree with you. I would argue that Genomics England is doing genomics right with the 100k genomes project and they certainly aren't doing custom analysis. Their pipeline is standard through variant calling and annotation. The WGS pipelines are pretty well established at this time. What you do with your VCFs is what can be customized. There is no reason for a small lab to write their own aligner or variant caller. The most that many do is write wrappers to create "customized pipelines", which is really just reinventing the wheel. EDIT: I want to clarify my mindset. In the environment that I work in there are three stages of analysis: primary, secondary, and tertiary. We mostly ignore the tertiary stage (I'm not actually doing science). Primary analysis is on instrument (base calling), secondary analysis takes you through variant calls and annotation, and tertiary analysis is where the real science happens. When I'm talking about canned analyses I'm talking about what I refer to as secondary analysis. Tertiary analysis is a very hard problem and will often require custom solutions. In my mind I'm separating bioinformatic analysis (secondary analysis) with interpretation (tertiary analysis).
- jerven 11y ago
  Getting the variants out is where the analysis starts. What does the variation mean, which ones correlate with a detectable phenotype. Which ones are affecting regulatory systems. Genomics England is brilliant because it has good access to excellent NHS data which it will use to do a lot of analysis. Which its not getting off the shelf but a lot of talented clincal+bio+informaticians are working hard on that. Or in other words getting the variants to the human genome reference is just the start of the analysis not the end. i.e. I am more worried about the end of the pipeline than the contents of them. I suspected as much on the 1,2,3 level. But even 2 is not really fixed yet. There are great off the shelf tools but even the concept of variation calling against a reference linear genome is not perfect yet especially with the alt_loci business. Which driving the variation graph work in the GA4GH projects.

searine 11y ago

>DNAnexus, Seven Bridges, and Illumina BaseSpace all provide cloud storage and analysis. "Analysis". There is a difference between meaningful analysis which answers long standing questions of biology, and running a program which calls SNPs and spits out genome stats. The hard part is knowing what questions to ask, and how to interpret the answers when you get them. That takes years of training to develop a critical mind.

rcthompson 11y ago
This is absolutely true. I've done differential expression analysis on many RNA-seq and microarray datasets, and despite the fact that the basic pipeline for such analysis is quite straightforward and easily automated, every analysis is asking different biological questions and has different confounding factors that need to be addressed, so the analysis of each dataset ends up being quite different from all the others.

dikdik 11y ago

All of these will give you some answers. The reality is there is SO MUCH MORE INFORMATION to be gleaned from our genomes that these solutions do not account for.