8 ms·
200 TB of 1000 genomes data on Amazon S3 as a public dataset
- rickyconnolly 15y agoThe burden has shifted from the bio to the informatics. We need to be able to sort through this much data without needing access to university-scale computer facilities.
- aderaynal 15y agothat's exactly what I'm during at my day job: http://nextbio.com http://nextbio.com
- biopharma_guy 15y agoHow many person's curated genome does the nextbio has? I am just curious.
- sciurus 15y agoAgreed. At EuPathDB, we're trying to provide just that for the genomic data of parasites that cause diseases like sleeping sickness and malaria. And we're hiring! See http://jobs.eupathdb.org/ http://jobs.eupathdb.org/
- link2009 15y agoI wish you guys: A) Have an office in Canada B) Will continue to hire into next year, when I graduate.
- gourneau 15y agoAt Ion Torrent we are working this problem as well. We are always hiring too http://iontorrent.com http://iontorrent.com All of our software is open source http://github.com/iontorrent http://github.com/iontorrent
- biopharma_guy 15y agowhich language the software is written in?
- nmz787 15y agoThe link at: http://www.iontorrent.com/career/ http://www.iontorrent.com/career/ just re-directs me to the homepage.
- bthomas 15y agoI think it's important to give clear examples here - research questions that biologists have posed but are struggling to answer with existing software tools. They certainly exist, but I think some overestimate the bottleneck. For example, I would argue that the bioinformatics challenges at Ion Torrent are less important than, say, chemistry that creates longer reads :)
- breck 15y agoI know very little about genomics and biology and would love it if someone who knows more could answer my silly questions to help me understand this more: - This has genome data from 1,700 people. In theory could a company build a machine that read one of these "files" as input and made a baby as output? - It says the dataset is 200TB. If 1 person has 3 billion dna bases(which I understand can be 1 of 4 values or 2 bits), it seems that even uncompressed the dataset would be 6billion bits (<1GB per person) and so only ~ 1.7TB uncompressed for 1,700 people. What am I missing?
- tomkinstinch 15y agoI haven't looked at the data yet, but I suspect the large data size is due to full sequencer output. Next-gen sequencing methods produce a LOT of noisy overlapping data that can later be reduced to high quality contiguous sequences. When compressed relative to a reference sequence, a ~3.3Gbp human genome can be reduced to ~3.3MiB of data[1]. For more information about the methods currently being used, check out these videos from the WUSTL Genome Center: http://gep.wustl.edu/curriculum/course_materials_WU/introduction_to_genomics/nextgen_video_tour http://gep.wustl.edu/curriculum/course_materials_WU/introduc... 1. http://dhruvbird.com/genome_compression.pdf http://dhruvbird.com/genome_compression.pdf
- deleted 15y ago[deleted]
- skosuri 15y ago1000 genomes includes many datasets including sequencing. That said, I'm sure most of this is sequencing data. The way sequencing is done is using billions of reads (usually b/w 50-100 nucleotides long), that are mapped back to the genome. Average coverage of any individual nucleotide is usually 30-50x. What people usually shared are compressed versions of the mapped reads that show all the sequencing information. The reason why this is done is that due to the limitations of the sequencing technologies, how you call particular mutations, structural variations, and deletions depends on the algorithms you use to reconstruct the genome. Thus the raw data could be in the form of compressed reads, or the compressed mapped reads (or in this case, likely both). Even this though is a vast reduction in size for the raw data, which are usually images that are used to call each base, which are usually 1000x larger than this data set; these usually aren't stored now for obvious space constraints.
- hirenj 15y agoI'm still not 100% sold on these big sequencing projects. Sure, you get the genomes of a bunch of individuals, and then you do a bunch of GWAS on it to see if you can find some mutations that are linked to any phenotypes. It just doesn't seem very exciting. I love the idea that if you're working on any cell line/individual, you can just get it sequenced for decreasing amounts of money, but I'd love to be sold on some big picture stuff as to why 1000 genomes matters (and if GWAS actually has lived up to its promise). That said, I've got a paper coming out real soon now that looked at the relationship between mutations across individual strains of Arabidopsis, and how (surprise surprise) you have fewer mutations in areas of proteins that potentially carry function, so I do think it's somewhat useful!
- lisper 15y ago> why 1000 genomes matters Gotta start somewhere. And 1000 data points is more than enough to get some statistically significant results.
- pedrobeltrao 15y agoUnfortunately, as far as I know there is very little useful phenotypic data along with these genomes to work with. I would rather have fewer genome sequences with better phenotypic information. Even with just 35 genome sequences of "individual" yeast genomes I could do a lot more interesting analysis because they are well studied in turns of phenotypes.
- gpurkins 15y agoThe point is to look at genomic variations across the entirety of the human genome, not so much to match genomic variation to phenotype. What you're saying makes sense, but I think that is more a next step.
- gpurkins 15y agoThe real reason for a lot of this is personalized medicine, or rather more specific medicine. A gene translocation or even a SNP may be able to differentiate cancer types and change the course of treatment.
- bthomas 15y agoThe best dataset for this audience is probably the most recent curated set of variant calls on 1000genomes.org. (Incidentally, the data access links appear down at the moment...) This provides data in VCF format [1], which I would argue is the lowest level you want to go with this data unless you are doing variant calling methods development. One tool you can use to analyze these data is PLINK/SEQ [2] (disclaimer: I work on the project). If any C++ devs are interested in contributing, let me know. (We'll have the source on Github soon...still pushing my group to learn Git :) [1] http://www.1000genomes.org/node/101 http://www.1000genomes.org/node/101 [2] atgu.mgh.harvard.edu/plinkseq/
- apaprocki 15y agoSince you are involved with PLINK/SEQ, you might be able to answer this.. Can you recommend any reading for learning how to perform admixture analysis/visualization? I'm interested in seeing the workflow behind starting with raw genomes and producing visualizations similar to Dodecad.
- stopcodon 15y agoI was just having a conversation with my supervisor yesterday about how amazing the PLINK project is, and how much easier it makes my work. I use it every day and I can't count the number of times it's saved me from writing embarrassingly messy R scripts to accomplish something.
- apawloski 15y agoSimilar to other questions, but can anyone explain what exactly this data is and what analysis is being done on it?
- gpurkins 15y agohttp://en.wikipedia.org/wiki/1000_Genomes_Project http://en.wikipedia.org/wiki/1000_Genomes_Project Here's the project explanation, but the data itself is 1000 complete (hopefully!) genome sequences. The hope is to find the variants, be it single nucleotide differences, copy number difference, etc. Where I work we use it to help figure out SNP (single nucleotide polymorphisms) to see if something we sequenced is a mutation or just a variant with no effect.
- tibbon 15y agoI dig this. However, unless I'm understanding S3 usage improperly, to use it you essentially have to clone that entire S3 bucket and pay for the monthly usage yourself of 200TB- correct? Not the biggest deal in the world if you're seriously crunching on it... but something to consider.
- bbgm 15y agoThe 1000 genomes bucket is open to the public, so you don't need to clone it for your own purposes. Any data you generate will need to be in your own bucket.
- pruman 15y agoHas anyone tried writing a life emulator that could read this gene code and then virtually generate a baby? This would be really cool for injecting one's self into an MMRPG, at least one where you could be a baby.
- deleted 15y ago[deleted]
- ayers 15y agoI would be interested in how much this dataset costs to run every month. Anyone able to shed some light on this?