4 ms·
My take (I compare about ZSTD only, as that what we need to care about): 1. Lizard's decompression speeds are much higher than ZSTD for any level of compression
by deepsun 4y ago
My take (I compare about ZSTD only, as that what we need to care about):
1. Lizard's decompression speeds are much higher than ZSTD for any level of compression.
2. Lizards's compression speed is significantly (several times) lower than ZSTD for similar compression ratios.
So it makes sense if we mostly don't care about CPU usage for compression.
- dataangel 4y agoif it's faster for both compression and decompression it just seems better period? why do you say "if we mostly don't care about CPU usage"? being faster means less CPU usage...
- deepsun 4y agoI said it's significantly _slower_, not faster, for compression.
- metadat 4y agoMinor nit: I too was confused by the wording: "compression speed is lower" To me this reads as though it's faster. Maybe you meant "compression speed is Slower"? Disclaimer: I generally dislike being pedantic, especially on HN. I hope this message doesn't come across that way.
- darrenf 4y agoIf compression speed is lower, then it's slower. If compression time is lower, then it's faster.
- kordlessagain 4y agoFaster/slower aside, it doesn't matter if it is slower to compress, because you only do that once for most use cases (thinking of downloadable repos or document stores). Decompression happens much more frequently. So, one compression algo that takes a lot of work (and time) to compress but uncompresses with less work than anything else is desirable because of the fact you only need compress once. So, less compute is required. There could be some use cases where this doesn't work out, such as a compressed network connection, where everything has to be compressed/decompressed in a 1:1 ratio.
- CoolGuySteve 4y agoThe thing I don't get about zstd is that their own github page shows lz4 is faster at both compression and decompression at the cost of some compression ratio: https://github.com/facebook/zstd https://github.com/facebook/zstd But most people I work with will pick zstd every time even in cases where decompression speed matters the most.
- adgjlsfhk1 4y agoIf you are bottle-necked by reading data (eg over networks or slow storage), zstd will be faster to decompress since you are transferring less data.
- arinlen 4y ago> If you are bottle-necked by reading data (eg over networks or slow storage), zstd will be faster to decompress since you are transferring less data. This should be stressed. Benchmarks show zstd's compression ratio being 75% of lz4's. Pushing 25% less data over a network is enough to win any real-world test, not to mention the impact it has on operational costs.
- woobilicious 4y agoI have this pathological example, I have a 30GB journal dump from bcachefs, and compressed it down with lz4 and zstd to a sata SSD, the compression ratios are 6.8% and 3% respectively (and this with default level 3!), it takes 14s to ripgrep through the lz4 version, and 11s with the zstd version. So even at 500MB/s zstd is faster for decompression.
- adgjlsfhk1 4y agoLZ4 requires about 2gb/s to be competitive on any relatively recent desktop/laptop cpu
- loxias 4y ago> But most people I work with will pick zstd every time even in cases where decompression speed matters the most. Zstd is really good. If you're a math/compression theory nerd it makes a lot of good choices. Also, if you're willing to tune zstd instead of using the defaults, so far anyway, I haven't found any application where the right answer wasn't zstd. At this point I don't think it's unreasonable to expect new compression developments to to explicitly say "What is improved over ZStandard".