3 ms·
Bioinformatician here. Consider using bgzf instead. It's fully gzip backwards compliant (it is a subset of gzip), but de/compression can be implemented to be mu
by jakobnissen 4y ago
Bioinformatician here. Consider using bgzf instead. It's fully gzip backwards compliant (it is a subset of gzip), but de/compression can be implemented to be much faster, and it's also much easier to paralellize. Compression rates are slightly lower, i.e it creates larger files.
The bgzf format was invented for bioinformatics, and BAM files are bgzipped by default.
- jiggawatts 4y agoThese days, skill is not about how you use tools in IT, but simply knowing that a tool exists.
- ROTMetro 4y agoHasn't that always been the case? When I lived in the Bay Area and networked, and was exposed to so much of how people did things, I was able to land much more contracts because I could come up with simple existing solutions. Once I moved away and wasn't exposed to cool people doing cool problem solving things, I had to switch into more bog standard consulting. It blew people's minds when we couple implement huge projects that they could never get of the ground for years in just a matter of months because of 'this one cool trick'. Man I miss Santa Cruz ಥ﹏ಥ worst mistake of my life to leave there.
- elmolino89 4y agobgzf has a clear advantage when it comes to sorted BED/VCF/GTF formats, especially if one does index these. But frankly I have no idea if it may improve IO times for the fastq files when read by say bwa or another mapper. Do you have any experience with that? I have seen some mapping times improvement using fastqs with clustered reads (by clumpify).
- jakobnissen 4y agoYes, bgzf can be much faster in practice. Both because the underlying gzip implementation can be simplified (see e.g. libdeflate), and because it can be more effectively parallelised. It doesn't matter if it's IO-bound of course, but compression rarely is.