4 ms·
igzip (https://github.com/intel/isa-l https://github.com/intel/isa-l) is much faster than gzip or pigz when it comes to decompression, 2-3x in my experience. Th
by optionalsquid 4y ago
igzip (https://github.com/intel/isa-l https://github.com/intel/isa-l) is much faster than gzip or pigz when it comes to decompression, 2-3x in my experience. There is also a Python module (isal) that provides a GzipFile-like wrapper class, for an easy speed-up of Python scripts that read gzipped files.
However, it only supports up to level 3 when compressing data, so it can't be used as a drop-in replacement for gzip. You also need to make sure to use the latest version if you are going to use it in the context of bioinformatics, since older versions choke on concatenated gzip files common in that field.
- lifthrasiir 4y ago> However, it only supports up to level 3 when compressing data, so it can't be used as a drop-in replacement for gzip. [EDIT: I misread a benchmark result [1] and wrongly concluded that igzip -3 is better than gzip -9, in reality it is somewhere in the middle of -1 and -9.] [1] https://encode.su/threads/3398-Sequence-Compression-Benchmark?p=66191&viewfull=1#post66191 https://encode.su/threads/3398-Sequence-Compression-Benchmar...
- optionalsquid 4y agoThat benchmark seems to be comparing igzip with gzip -1, since the reported gzip output size is 42298786 bytes and not 36548933 bytes (from the earlier benchmarks). My own experience based mostly on FASTQ/text data is that igzip -3 gives results very similar to gzip -3.
- lifthrasiir 4y agoOops! Yes, you are correct. I confused it with an other reply with level 9 benchmarks, please disregard the parent comment.