4 ms·
There have been a lot of co-processors in the genomics and sequencing world. They have all failed for obvious business reasons. It's not even clear that any ge
by dekhn 9y ago
There have been a lot of co-processors in the genomics and sequencing world. They have all failed for obvious business reasons. It's not even clear that any genomics problem that exists today can't be solved with conventional hardware. Most genomics programs reach a few percent of the capacity of the hardware, so I think more software tuning and design is the right approach.
- searine 9y agoI agree. The problem in genomics isn't how fast alignment is going, it is where to put the gobs and gobs of data. The bottleneck in speed is how fast you can flip a DNA sequencer, which right now with an illumina Novaseq that bottleneck is a few days per run. That gives plenty of time to process each runs data on an HPC on current software for alignment and snp calling. However, when each runfolder takes up 30Tb and you're flipping 20 runfolders a week. The scale of storage get large, fast.
- klmr 9y agoThat’s why compression of sequencing data is becoming so important. CRAM is nowhere near the theoretical limit (CRAM achieves ~2x over BAM; commercial products currently achieve up to 6x). (COI disclaimer: I work for such a company.)
- heuermh 9y agoNo need for the disclaimer, post a link to your benchmarking results. We don't compress as well as CRAM, but distributed reads across an Apache Spark cluster are much more efficient via Parquet http://adam.readthedocs.io/en/latest/benchmarks/storage/ http://adam.readthedocs.io/en/latest/benchmarks/storage/
- dekhn 9y agoIt's not clear you want to store your sequence data maximally compressed unless you have it archived. If you're doing live work on the data, it's better to have it modestly compressed with a fast decompressor, and a good index so you can minimize your total access.
- klmr 9y agoThat’s plausible but it turns out not to be true. Disk read latency is larger than the overhead of our decompression, even on fast storage media. As a consequence, better compression actually leads to (slight) performance improvements. Additionally, genomic data is often accessed via relatively slow network storage on clusters or, worse, via the internet. Increasing throughput trumps all other considerations here. You’re right concerning random access and indexing, of course.
- dekhn 9y agoI counter your argument: I have constructed systems that did this. Disk read latency doesn't matter for streaming reads. The actual math for this for a product at scale is interesting; I built such a product and did the math, and we found that moderate compression gave higher rates. However, this is based on a production-class infrastructure.
- dekhn 9y agoI should also mention: using more CPU to decompress highly compressed data means you have less CPU available (on a multithreaded app in a resource-managed environment) for other work. Since after decompressing you're going to be doing a bunch of other work (and a lot of that work will be going on simultaneousy in other threads), using less CPU to decompress can give higher throughput. But, again, this is a very complex problem.
- neuromantik8086 9y agoOddly enough, I found a similar sentiment echoed last night when I was looking up how widespread the use of ASICs was in various scientific fields. I had discovered an FPGA startup called Edico Genome [1] that seemed to offer substantial advantages in speed of processing, and after searching through reddit came across a post where someone noted that there has been a precedent for FPGA/ASIC related companies to go under in the genomics sphere [2]. The poster there attributed this to an unwillingness to use tools that weren't widely cited, however. [1] http://edicogenome.com/ http://edicogenome.com/ [2] https://www.reddit.com/r/bioinformatics/comments/73wcd3/failed_startups/ https://www.reddit.com/r/bioinformatics/comments/73wcd3/fail...
- dekhn 9y agoTo deal with the citation problem, many FPGAs and ASICs that implemented BLAST guaranteed "bit-identical output". You could easily verify anything found by the accelerator using BLAST, just more slowly.