4 ms·
Genomic data is storage is very redundant, SRA stores the raw data the way it is collected and that is extremely inefficient. For example if scientists were to
by glofish 11y ago
Genomic data is storage is very redundant, SRA stores the raw data the way it is collected and that is extremely inefficient.
For example if scientists were to collect data that matched the human genome perfectly they would still store all the data, tens or even hundreds of gigabytes - yet the information content of all that is actually one bit: the data matches perfectly the human genome, nothing more needed to be stored at all.
In addition the current genomic data collection is a "prisoner" of the technology. It produces lots of small "reads" (DNA fragments) instead of a single genomic measurement. Because the data is so fragmented we need to measure the same thing many times over (hundreds even). Hence even more redundancy at least 100 fold.
Now magnify this over tens or hundreds of thousands of samples and naturally we have a problem.
If we could sequence one human genome it would take 3x10^9 bytes --> 2.3 gigabytes.
But instead if we were to store the only the changes relative to the reference representing the human genome would take 5x10^6 bytes --> 4 megabytes. Hardly a big challenge to store.
But here is another thing, why store that data when you could re sequence it? All we need is a faster sequencing methodology and all of a sudden data does not need to be stored only the sample does.
- tosseraccount 11y agoNote that CRAM format does use the reference genome to compress sequences. http://www.1000genomes.org/faq/what-are-cram-files http://www.1000genomes.org/faq/what-are-cram-files CRAM files are alignment files like BAM files. They represent a compressed version of the alignment. This compression is driven by the reference the sequence data is aligned to. Benchmarks here : http://www.htslib.org/benchmarks/CRAM.html http://www.htslib.org/benchmarks/CRAM.html
- shas3 11y agoClassic compression problem. There is no free lunch in that you'd still need to compute the 'difference', which is a combinatorial search. If you could indeed compute the difference, technically you would throw away the raw data and store only differences. But you'd still need to temporarily have access to the raw data.
- collyw 11y agoLots of stuff genomics researchers do is inefficient. Generally they store sequence in strings and compare those, when 4 bases can be represented by two bytes. Then you can compare them using bitwise operators, which is very close to the way processors work, and orders of magnitude faster than comparing strings.
- jjtheblunt 11y agoI believe you have a typo and meant 2 Bits.
- collyw 11y agoyes you are right.
- mbreese 11y agoSometimes genomics researchers do things inefficiently on purpose. You very rarely want to look to see if two strings are exactly the same. Instead, you usually try to look at how similar the sequences are, taking into account ambiguities or mismatches. In this scenario, storing sequences compressed as 2-bit encoding isn't significantly faster (if it is at all).
- collyw 11y agoIt can be if you represent them correctly. I count mismatches between sequences using bitwise operators by representing the strings as two binary numbers. One number stores a 1 for an A/C, 0 for G/T and one number with 1's A/G as 0 for C/T. And it was at least an order of magnitude faster. I benchmarked it. (It wasn't just comparing one sequence to another, but also other factors, such as having a red or green lazer in a particular position).
- mbreese 11y agoAnd how do you represent an unknown base in that representation? Or, how would you do the comparison with a gap in one of the sequences (which is incredibly common). I've benchmarked it all too, and it is really difficult to represent DNA with just two bits. Once you start to get out from the simplest use-cases, it gets significantly more difficult. The most efficient mechanisms require three bits or at least some mechanism for denoting where the ambiguous calls are (see UCSC's 2-bit format). And now that you're dealing with three bits, the numbers start to get funky quickly - you end up dealing with 24 bits for 6 consecutive bases, and you still have the problem of representing gaps efficiently... have you ever tried to insert 4 bits in the middle of another number?