3 ms·

I'm unfamiliar with the bioinformatics scene, but I can imagine there's a lot of "legacy" hardware and software lying around that can't easily be updated to sup

I'm unfamiliar with the bioinformatics scene, but I can imagine there's a lot of "legacy" hardware and software lying around that can't easily be updated to support superior formats. Perhaps an interim solution would be something like "zstd+metadata", where the metadata is sufficient to transparently and efficiently reconstruct a gzip-compressed file on-demand. (similarly to how JPEG-XL allows oldschool JPEGs to be recompressed without losing any data) This could have a bit of compute overhead, but since gzip decompression is so single-threaded I think you could do the conversion in parallel without actually slowing down the hot path. So, the performance would be approximately the same as using gzip (assuming decompression is compute-bound, not io-bound), but with all the storage benefits of using zstd.

croemer 4y ago
Yeah that's exactly the issue. A lot of tools are written once and rarely updated, they're considered "done". And they only support gz out of the box because they were written 10 years ago.
- jltsiren 4y ago
  I think the tools people actually use are rarely that old. Almost all new tools support gzip by default, because almost all public data is gzip-compressed. Developers often don't bother adding zstd support, because there is little zstd-compressed data around. Because the labs developing new tools are rarely the ones that have to store petabytes of data, they don't benefit from zstd directly. And once the active development is over, most tools receive only minimal support, because nobody wants to pay for maintaining the long tail of average tools. Also, zstd is the latest reason why gzip is still widely used. Every time someone releases a new general-purpose data compressor that becomes popular, gzip gains a few more years. Because there have been too many gzip replacements over the years, none of them has managed to become popular enough to actually replace gzip before the attention moves to the next state-of-the art compressor. If people agreed this time that zstd is what everyone will be using for the next 20 years, bioinformatics would probably move to it in a few years.
- asdff 4y ago
  But that's hardly an issue, since you can just add the new compression tool to your pipeline before and after the older tool. Many tools don't even support gzip because they assume the user will be piping in uncompressed data from zcat anyway.
jefftk 4y ago
What metadata do you mean? Since both zstd and gzip are lossless compression tools all you need if you have an old tool that only takes in gzipped fasta would be "cat foo.fasta.zstd | zstdcat | gzip | oldtool".
- Retr0id 4y ago
  Compressing gzip is (relatively) slow if you want to actually compress the data. The biggest slowdown comes from searching for the best backreference at any given point. If you could encode some relatively compact data saying "here's where the good backrefs are", you could optimise your throughput/compression-ratio ratio (at the cost of slightly more data on-disk). Furthermore, if you want access to be truly transparent (e.g. something like a FUSE fs that transparently presents a set of gzip files, each backed by a corresponding zstd file on disk), then you (may) need to support seeking to an arbitrary offset. As far as I know, zstd doesn't support seeking into a stream by default, so you'd have to add some metadata to facilitate that. It is possible that you don't actually care about this, in which case yes, you don't need any extra metadata.
  - asdff 4y ago
    Processes like alignment are alot slower than compression. This is a space where some tooling takes days or weeks to run, and you are running them on a cluster of xeons that can gzip or gunzip fast enough for your use cases anyhow.
asdff 4y ago
The bioinformatics scene is generally legacy in the good way, like how bash tooling is legacy: consistent and highly modular. Sure, the tool you are using might not be able to support compressed data or some new compression algorithm, but then so what, you can just add whatever hotnewcompression tool before and after this tool in your pipeline, and throttle the work to maximize available storage and compute. Many pipelines are written simply in bash, or use something like snakemake or nextflow to ensure consistent environments.

croemer 4y ago

Yeah that's exactly the issue. A lot of tools are written once and rarely updated, they're considered "done". And they only support gz out of the box because they were written 10 years ago.

jltsiren 4y ago
I think the tools people actually use are rarely that old. Almost all new tools support gzip by default, because almost all public data is gzip-compressed. Developers often don't bother adding zstd support, because there is little zstd-compressed data around. Because the labs developing new tools are rarely the ones that have to store petabytes of data, they don't benefit from zstd directly. And once the active development is over, most tools receive only minimal support, because nobody wants to pay for maintaining the long tail of average tools. Also, zstd is the latest reason why gzip is still widely used. Every time someone releases a new general-purpose data compressor that becomes popular, gzip gains a few more years. Because there have been too many gzip replacements over the years, none of them has managed to become popular enough to actually replace gzip before the attention moves to the next state-of-the art compressor. If people agreed this time that zstd is what everyone will be using for the next 20 years, bioinformatics would probably move to it in a few years.
asdff 4y ago
But that's hardly an issue, since you can just add the new compression tool to your pipeline before and after the older tool. Many tools don't even support gzip because they assume the user will be piping in uncompressed data from zcat anyway.

jefftk 4y ago

What metadata do you mean? Since both zstd and gzip are lossless compression tools all you need if you have an old tool that only takes in gzipped fasta would be "cat foo.fasta.zstd | zstdcat | gzip | oldtool".

Retr0id 4y ago
Compressing gzip is (relatively) slow if you want to actually compress the data. The biggest slowdown comes from searching for the best backreference at any given point. If you could encode some relatively compact data saying "here's where the good backrefs are", you could optimise your throughput/compression-ratio ratio (at the cost of slightly more data on-disk). Furthermore, if you want access to be truly transparent (e.g. something like a FUSE fs that transparently presents a set of gzip files, each backed by a corresponding zstd file on disk), then you (may) need to support seeking to an arbitrary offset. As far as I know, zstd doesn't support seeking into a stream by default, so you'd have to add some metadata to facilitate that. It is possible that you don't actually care about this, in which case yes, you don't need any extra metadata.
- asdff 4y ago
  Processes like alignment are alot slower than compression. This is a space where some tooling takes days or weeks to run, and you are running them on a cluster of xeons that can gzip or gunzip fast enough for your use cases anyhow.

asdff 4y ago

The bioinformatics scene is generally legacy in the good way, like how bash tooling is legacy: consistent and highly modular. Sure, the tool you are using might not be able to support compressed data or some new compression algorithm, but then so what, you can just add whatever hotnewcompression tool before and after this tool in your pipeline, and throttle the work to maximize available storage and compute. Many pipelines are written simply in bash, or use something like snakemake or nextflow to ensure consistent environments.