4 ms·
That’s plausible but it turns out not to be true. Disk read latency is larger than the overhead of our decompression, even on fast storage media. As a consequen
by klmr 9y ago
That’s plausible but it turns out not to be true. Disk read latency is larger than the overhead of our decompression, even on fast storage media. As a consequence, better compression actually leads to (slight) performance improvements. Additionally, genomic data is often accessed via relatively slow network storage on clusters or, worse, via the internet. Increasing throughput trumps all other considerations here. You’re right concerning random access and indexing, of course.
- dekhn 9y agoI counter your argument: I have constructed systems that did this. Disk read latency doesn't matter for streaming reads. The actual math for this for a product at scale is interesting; I built such a product and did the math, and we found that moderate compression gave higher rates. However, this is based on a production-class infrastructure.
- dekhn 9y agoI should also mention: using more CPU to decompress highly compressed data means you have less CPU available (on a multithreaded app in a resource-managed environment) for other work. Since after decompressing you're going to be doing a bunch of other work (and a lot of that work will be going on simultaneousy in other threads), using less CPU to decompress can give higher throughput. But, again, this is a very complex problem.