4 ms·

I agree. The problem in genomics isn't how fast alignment is going, it is where to put the gobs and gobs of data. The bottleneck in speed is how fast you can f

I agree. The problem in genomics isn't how fast alignment is going, it is where to put the gobs and gobs of data. The bottleneck in speed is how fast you can flip a DNA sequencer, which right now with an illumina Novaseq that bottleneck is a few days per run. That gives plenty of time to process each runs data on an HPC on current software for alignment and snp calling. However, when each runfolder takes up 30Tb and you're flipping 20 runfolders a week. The scale of storage get large, fast.

klmr 9y ago
That’s why compression of sequencing data is becoming so important. CRAM is nowhere near the theoretical limit (CRAM achieves ~2x over BAM; commercial products currently achieve up to 6x). (COI disclaimer: I work for such a company.)
- heuermh 9y ago
  No need for the disclaimer, post a link to your benchmarking results. We don't compress as well as CRAM, but distributed reads across an Apache Spark cluster are much more efficient via Parquet http://adam.readthedocs.io/en/latest/benchmarks/storage/ http://adam.readthedocs.io/en/latest/benchmarks/storage/
- dekhn 9y ago
  It's not clear you want to store your sequence data maximally compressed unless you have it archived. If you're doing live work on the data, it's better to have it modestly compressed with a fast decompressor, and a good index so you can minimize your total access.
  - klmr 9y ago
    That’s plausible but it turns out not to be true. Disk read latency is larger than the overhead of our decompression, even on fast storage media. As a consequence, better compression actually leads to (slight) performance improvements. Additionally, genomic data is often accessed via relatively slow network storage on clusters or, worse, via the internet. Increasing throughput trumps all other considerations here. You’re right concerning random access and indexing, of course.
    - dekhn 9y ago
      I counter your argument: I have constructed systems that did this. Disk read latency doesn't matter for streaming reads. The actual math for this for a product at scale is interesting; I built such a product and did the math, and we found that moderate compression gave higher rates. However, this is based on a production-class infrastructure.
      - dekhn 9y ago
        I should also mention: using more CPU to decompress highly compressed data means you have less CPU available (on a multithreaded app in a resource-managed environment) for other work. Since after decompressing you're going to be doing a bunch of other work (and a lot of that work will be going on simultaneousy in other threads), using less CPU to decompress can give higher throughput. But, again, this is a very complex problem.

klmr 9y ago

That’s why compression of sequencing data is becoming so important. CRAM is nowhere near the theoretical limit (CRAM achieves ~2x over BAM; commercial products currently achieve up to 6x). (COI disclaimer: I work for such a company.)

heuermh 9y ago
No need for the disclaimer, post a link to your benchmarking results. We don't compress as well as CRAM, but distributed reads across an Apache Spark cluster are much more efficient via Parquet http://adam.readthedocs.io/en/latest/benchmarks/storage/ http://adam.readthedocs.io/en/latest/benchmarks/storage/
dekhn 9y ago
It's not clear you want to store your sequence data maximally compressed unless you have it archived. If you're doing live work on the data, it's better to have it modestly compressed with a fast decompressor, and a good index so you can minimize your total access.
- klmr 9y ago
  That’s plausible but it turns out not to be true. Disk read latency is larger than the overhead of our decompression, even on fast storage media. As a consequence, better compression actually leads to (slight) performance improvements. Additionally, genomic data is often accessed via relatively slow network storage on clusters or, worse, via the internet. Increasing throughput trumps all other considerations here. You’re right concerning random access and indexing, of course.
  - dekhn 9y ago
    I counter your argument: I have constructed systems that did this. Disk read latency doesn't matter for streaming reads. The actual math for this for a product at scale is interesting; I built such a product and did the math, and we found that moderate compression gave higher rates. However, this is based on a production-class infrastructure.
    - dekhn 9y ago
      I should also mention: using more CPU to decompress highly compressed data means you have less CPU available (on a multithreaded app in a resource-managed environment) for other work. Since after decompressing you're going to be doing a bunch of other work (and a lot of that work will be going on simultaneousy in other threads), using less CPU to decompress can give higher throughput. But, again, this is a very complex problem.