3 ms·
What metadata do you mean? Since both zstd and gzip are lossless compression tools all you need if you have an old tool that only takes in gzipped fasta would
by jefftk 4y ago
What metadata do you mean? Since both zstd and gzip are lossless compression tools all you need if you have an old tool that only takes in gzipped fasta would be "cat foo.fasta.zstd | zstdcat | gzip | oldtool".
- Retr0id 4y agoCompressing gzip is (relatively) slow if you want to actually compress the data. The biggest slowdown comes from searching for the best backreference at any given point. If you could encode some relatively compact data saying "here's where the good backrefs are", you could optimise your throughput/compression-ratio ratio (at the cost of slightly more data on-disk). Furthermore, if you want access to be truly transparent (e.g. something like a FUSE fs that transparently presents a set of gzip files, each backed by a corresponding zstd file on disk), then you (may) need to support seeking to an arbitrary offset. As far as I know, zstd doesn't support seeking into a stream by default, so you'd have to add some metadata to facilitate that. It is possible that you don't actually care about this, in which case yes, you don't need any extra metadata.
- asdff 4y agoProcesses like alignment are alot slower than compression. This is a space where some tooling takes days or weeks to run, and you are running them on a cluster of xeons that can gzip or gunzip fast enough for your use cases anyhow.