5 ms·
Lizard – efficient compression with fast decompression
- deepsun 4y agoMy take (I compare about ZSTD only, as that what we need to care about): 1. Lizard's decompression speeds are much higher than ZSTD for any level of compression. 2. Lizards's compression speed is significantly (several times) lower than ZSTD for similar compression ratios. So it makes sense if we mostly don't care about CPU usage for compression.
- dataangel 4y agoif it's faster for both compression and decompression it just seems better period? why do you say "if we mostly don't care about CPU usage"? being faster means less CPU usage...
- deepsun 4y agoI said it's significantly _slower_, not faster, for compression.
- metadat 4y agoMinor nit: I too was confused by the wording: "compression speed is lower" To me this reads as though it's faster. Maybe you meant "compression speed is Slower"? Disclaimer: I generally dislike being pedantic, especially on HN. I hope this message doesn't come across that way.
- darrenf 4y agoIf compression speed is lower, then it's slower. If compression time is lower, then it's faster.
- kordlessagain 4y agoFaster/slower aside, it doesn't matter if it is slower to compress, because you only do that once for most use cases (thinking of downloadable repos or document stores). Decompression happens much more frequently. So, one compression algo that takes a lot of work (and time) to compress but uncompresses with less work than anything else is desirable because of the fact you only need compress once. So, less compute is required. There could be some use cases where this doesn't work out, such as a compressed network connection, where everything has to be compressed/decompressed in a 1:1 ratio.
- CoolGuySteve 4y agoThe thing I don't get about zstd is that their own github page shows lz4 is faster at both compression and decompression at the cost of some compression ratio: https://github.com/facebook/zstd https://github.com/facebook/zstd But most people I work with will pick zstd every time even in cases where decompression speed matters the most.
- adgjlsfhk1 4y agoIf you are bottle-necked by reading data (eg over networks or slow storage), zstd will be faster to decompress since you are transferring less data.
- arinlen 4y ago> If you are bottle-necked by reading data (eg over networks or slow storage), zstd will be faster to decompress since you are transferring less data. This should be stressed. Benchmarks show zstd's compression ratio being 75% of lz4's. Pushing 25% less data over a network is enough to win any real-world test, not to mention the impact it has on operational costs.
- woobilicious 4y agoI have this pathological example, I have a 30GB journal dump from bcachefs, and compressed it down with lz4 and zstd to a sata SSD, the compression ratios are 6.8% and 3% respectively (and this with default level 3!), it takes 14s to ripgrep through the lz4 version, and 11s with the zstd version. So even at 500MB/s zstd is faster for decompression.
- adgjlsfhk1 4y agoLZ4 requires about 2gb/s to be competitive on any relatively recent desktop/laptop cpu
- loxias 4y ago> But most people I work with will pick zstd every time even in cases where decompression speed matters the most. Zstd is really good. If you're a math/compression theory nerd it makes a lot of good choices. Also, if you're willing to tune zstd instead of using the defaults, so far anyway, I haven't found any application where the right answer wasn't zstd. At this point I don't think it's unreasonable to expect new compression developments to to explicitly say "What is improved over ZStandard".
- lifthrasiir 4y agoNote that a benchmark in the README refers to zstd 1.1.1 and brotli 0.5.2, which are very old (the current versions are zstd 1.5.2 and brotli 1.0.9). The same author maintains lzbench [1] which is more current than the README. It shows that Lizard heavily trades off compression speed and ratio for decompression speed; zstd at the default setting (-3) is expected to be on par with Lizard -45 in terms of ratio, but Lizard would be >10x slower to compress while only ~3x faster to decompress. [1] https://github.com/inikep/lzbench https://github.com/inikep/lzbench
- p0nce 4y agoWould be interesting to couple it with QOI, as QOI + LZ4 is very interesting as image format.
- genewitch 4y agoas in you run the bitmap through QOI and then compress that output with LZ4? Or am i misremembering how QOI works and it uses LZ4 internally? PNG uses LZ, doesn't it?
- p0nce 4y ago> as in you run the bitmap through QOI and then compress that output with LZ4? Yes, it can be very beneficial.
- adgjlsfhk1 4y agoQOI doesn't use any entropy compression.