3 ms·
I'm not in the bioinformatic domain but maybe there are some legacy tools that depend on some specifics of gzip. I was thinking something like the possibility
by GistNoesis 4y ago
I'm not in the bioinformatic domain but maybe there are some legacy tools that depend on some specifics of gzip.
I was thinking something like the possibility to seek through a compressed file (after having built an index). Your DNA file is probably stored on a shared folder somewhere (if it's big you probably don't want to copy it to every workstation) and you point your software to the gz file directly, it create a seek-index on the first use, and then when you need view only a small section of the file you don't have to extract the whole file.
Some more advanced compression like zstd might use some form of dictionary to do the compression/decompression, which may be bigger (up to 32MB? max dictionary size for zstd whereas lz77 used by gzip use a dictionary based on a sliding window of the last 32K token) which you may have to transfer.
- v8xi 4y agoCan confirm that the data is often, though not always, stored in a shared directory and that yes, many tools can/do read gzip files directly which probably hinders the adoption of better compression algorithms.
- asdff 4y agoIf they do then they are just probably running zcat for you for convenience sake. You can always incorporate better compression by adding it to your pipeline, and piping in uncompressed data to the legacy tool then compressing the resulting output with whatever tool you want.