3 ms·

There are compressors specialized for FASTQ that are faster and denser than zstd. FASTQ is the the most common format for storing DNA sequencing data-- a text f

There are compressors specialized for FASTQ that are faster and denser than zstd. FASTQ is the the most common format for storing DNA sequencing data-- a text file including metadata, sequence, confidence scores, etc. http://kirr.dyndns.org/sequence-compression-benchmark/ http://kirr.dyndns.org/sequence-compression-benchmark/ fastqz (not the best name!) supports reference compression too, for smaller files when they're reads compared to a reference genome. Here's an 2013 paper considering the problem: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0059190 https://journals.plos.org/plosone/article?id=10.1371/journal... Still, zstd is widely available and a simple drop-in replacement for gzip.

_yb2s 4y ago
Reference compression is interesting... technically, this type of compression should be possible without a reference. E.g. in a worst case scenario, you do a genome assembly from the reads, and then compress against that reference. Fascinating to think that in principle, genome assembly and fastq file compression have many parallels, and are essentially the same problem. For example, once you assemble the genome, you can just track the coordinates and 'diff' for each read and the original FASTQ file can be regenerated exactly.
- bnprks 4y ago
  One recent method that does this is SPRING: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6662292/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6662292/ Although I'm not familiar with all the details of the method, my understanding is that the reference used by compression can be extremely low quality for biological purposes while still providing good compression -- making it possible to make a quick-and-dirty reference with much less computational time.
- v8xi 4y ago
  That would be true for FASTA compression but FASTQ files contain other data. With a MiSeq, for example, 1/3 of the data is the DNA sequences, 1/3 is a quality score (0-37) and 1/3 is a header which includes things like flow cell ID but also a lot of info on the physical registration address of the cluster on the flow cell. In some applications this info is not required, but is from a scientific/data integrity perspective

_yb2s 4y ago

Reference compression is interesting... technically, this type of compression should be possible without a reference. E.g. in a worst case scenario, you do a genome assembly from the reads, and then compress against that reference. Fascinating to think that in principle, genome assembly and fastq file compression have many parallels, and are essentially the same problem. For example, once you assemble the genome, you can just track the coordinates and 'diff' for each read and the original FASTQ file can be regenerated exactly.

bnprks 4y ago
One recent method that does this is SPRING: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6662292/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6662292/ Although I'm not familiar with all the details of the method, my understanding is that the reference used by compression can be extremely low quality for biological purposes while still providing good compression -- making it possible to make a quick-and-dirty reference with much less computational time.
v8xi 4y ago
That would be true for FASTA compression but FASTQ files contain other data. With a MiSeq, for example, 1/3 of the data is the DNA sequences, 1/3 is a quality score (0-37) and 1/3 is a header which includes things like flow cell ID but also a lot of info on the physical registration address of the cluster on the flow cell. In some applications this info is not required, but is from a scientific/data integrity perspective