3 ms·

That’s why compression of sequencing data is becoming so important. CRAM is nowhere near the theoretical limit (CRAM achieves ~2x over BAM; commercial products

by klmr 9y ago

That’s why compression of sequencing data is becoming so important. CRAM is nowhere near the theoretical limit (CRAM achieves ~2x over BAM; commercial products currently achieve up to 6x). (COI disclaimer: I work for such a company.)

heuermh 9y ago
No need for the disclaimer, post a link to your benchmarking results. We don't compress as well as CRAM, but distributed reads across an Apache Spark cluster are much more efficient via Parquet http://adam.readthedocs.io/en/latest/benchmarks/storage/ http://adam.readthedocs.io/en/latest/benchmarks/storage/
dekhn 9y ago
It's not clear you want to store your sequence data maximally compressed unless you have it archived. If you're doing live work on the data, it's better to have it modestly compressed with a fast decompressor, and a good index so you can minimize your total access.
- klmr 9y ago
  That’s plausible but it turns out not to be true. Disk read latency is larger than the overhead of our decompression, even on fast storage media. As a consequence, better compression actually leads to (slight) performance improvements. Additionally, genomic data is often accessed via relatively slow network storage on clusters or, worse, via the internet. Increasing throughput trumps all other considerations here. You’re right concerning random access and indexing, of course.
  - dekhn 9y ago
    I counter your argument: I have constructed systems that did this. Disk read latency doesn't matter for streaming reads. The actual math for this for a product at scale is interesting; I built such a product and did the math, and we found that moderate compression gave higher rates. However, this is based on a production-class infrastructure.
    - dekhn 9y ago
      I should also mention: using more CPU to decompress highly compressed data means you have less CPU available (on a multithreaded app in a resource-managed environment) for other work. Since after decompressing you're going to be doing a bunch of other work (and a lot of that work will be going on simultaneousy in other threads), using less CPU to decompress can give higher throughput. But, again, this is a very complex problem.

by klmr 9y ago

heuermh 9y ago

No need for the disclaimer, post a link to your benchmarking results. We don't compress as well as CRAM, but distributed reads across an Apache Spark cluster are much more efficient via Parquet http://adam.readthedocs.io/en/latest/benchmarks/storage/ http://adam.readthedocs.io/en/latest/benchmarks/storage/

dekhn 9y ago

It's not clear you want to store your sequence data maximally compressed unless you have it archived. If you're doing live work on the data, it's better to have it modestly compressed with a fast decompressor, and a good index so you can minimize your total access.

klmr 9y ago
That’s plausible but it turns out not to be true. Disk read latency is larger than the overhead of our decompression, even on fast storage media. As a consequence, better compression actually leads to (slight) performance improvements. Additionally, genomic data is often accessed via relatively slow network storage on clusters or, worse, via the internet. Increasing throughput trumps all other considerations here. You’re right concerning random access and indexing, of course.
- dekhn 9y ago
  I counter your argument: I have constructed systems that did this. Disk read latency doesn't matter for streaming reads. The actual math for this for a product at scale is interesting; I built such a product and did the math, and we found that moderate compression gave higher rates. However, this is based on a production-class infrastructure.
  - dekhn 9y ago
    I should also mention: using more CPU to decompress highly compressed data means you have less CPU available (on a multithreaded app in a resource-managed environment) for other work. Since after decompressing you're going to be doing a bunch of other work (and a lot of that work will be going on simultaneousy in other threads), using less CPU to decompress can give higher throughput. But, again, this is a very complex problem.