4 ms·
For compressing similar genomes, specialized tools will do better. AGC [1] can compress 620k SARS-CoV-2 genomes (19 GB uncompressed) into tens of MB. More impor
by attractivechaos 4y ago
For compressing similar genomes, specialized tools will do better. AGC [1] can compress 620k SARS-CoV-2 genomes (19 GB uncompressed) into tens of MB. More importantly, AGC allows you to extract individual sequences without decompressing the entire file. In addition, zstd/xz wouldn't have a big enough dictionary to compress multiple human genomes efficiently. AGC can compress 300 GB human assemblies to 1.5 GB, a 200-fold compression ratio.
I agree for sequence reads, zstd is better than gzip in almost every aspect. The only problem is that zstd is not as widely available. Many Linux distros don't ship zstd by default but on managed clusters, common users don't have the root permission to install zstd system-wide.
It is worth mentioning that many mainstream bioinformatics tools can read FASTQ over pipes. You can use something like
mytool <(zstd -dc in.fastq.zst) > output.txt # bash; also works on multiple zstd input files
zstd -dc in.fastq.zst | mytool - > output.txt # general; for multiple input files, use mkfifo
Those who prefer zstd can use it now.
[1] https://www.biorxiv.org/content/10.1101/2022.04.07.487441v1 https://www.biorxiv.org/content/10.1101/2022.04.07.487441v1
- michaelbarton 4y agoUnfortunately some bioinformatics tools require multiple passes over the FASTQ. Spades is an example of this.
- attractivechaos 4y agoFair enough. For the relatively few tools that don’t support streaming, you may generate temporary plain files in your pipeline. Spades itself is a pipeline and writes temporary files. It is anyway not realistic to expect wide adoption of zstd in the near future. If you want to save storage cost now, you have to take some trade off.