3 ms·
They are FASTQ files, so they are the reads that came off the sequencer- plus metadata and error scores- represented as ASCII A, T, G, C. This isn't a diff fro
by dekhn 4y ago
They are FASTQ files, so they are the reads that came off the sequencer- plus metadata and error scores- represented as ASCII A, T, G, C.
This isn't a diff from a reference genome, it's a ton of reads (files often 30+GB). It's really only useful as an archive format, and even then, it's a pretty bad one for many reasons.
An interesting thing about the file format is that it's three lines in a row each with different coding properties, so if you break the data into three streams and encode each one on its own, the overall compression is somewhat better, because the dictionary and lookback don't have to accomodate the metadata, DNA, and error lines together, giving better dictionary and lookback hit rates.
These formats are terrible for downstream analysis: they are typically not indexed or sorted in a useful way, so it's common to convert them to other formats (BAM, CRAM), which are optimized for compressing DNA data (and likely exceed zstd in some circumstances).
- otherme123 4y ago> These formats are terrible for downstream analysis: they are typically not indexed or sorted in a useful way, so it's common to convert them to other formats (BAM, CRAM), which are optimized for compressing DNA data (and likely exceed zstd in some circumstances). FASTQ are raw data reads. BAM and CRAM are reads mapped against a genome.
- biotinker 4y agoBAM/CRAM reads are often mapped against a genome, but they needn't be. It's not uncommon in industry to convert FASTQ to unmapped BAM in order to only have a single file format, improve compression, etc.