5 ms·
lz4 is performing pretty well. Better than most zstd-fast settings. It will not be obsolete after adding ZSTD and most likely stay the go-to algorithm if you ar
by foepys 6y ago
lz4 is performing pretty well. Better than most zstd-fast settings. It will not be obsolete after adding ZSTD and most likely stay the go-to algorithm if you are unsure what to choose.
- chungy 6y agoI think it's likely that LZ4 remains the default. There are almost no downsides to having LZ4 enabled. zstd is more likely to represent an obsolescence of gzip. It surpasses gzip pretty much always.
- db48x 6y agoJust going by those graphs, I could double my compression ratio by going from lz4 to zstd-1 without going below the speeds the drives in my pool can manage. The usual caveats apply, but it seems to me that this is a pretty good upgrade for the usual case where you're using hard drives instead of fast ssds in a pool.
- snvzz 6y agoThis sounds good for fileservers. But generally speaking, computers usually do more than i/o.
- rini17 6y agoBasically yes, but also depends on if you're I/O bound and have free CPU cycles. If there's limited bandwidth to underlying storage, with zstd it's possible to get more data in/out.
- rakoo 6y agoFYI lz4 and zstd are made by the same guy, so there is no surprise both of them complement each other. lz4 targets the (de)compress-as-fast-as-possible domain, while zstd is for the rest (good (de)compression with slower speed). For something that is write-once read-many-times, like a filesystem, a good compression algorithm might be more interesting in the future. lz4 is more targeted at write-once read-once, like file transfers for example
- jerryX 6y agoZstd uses this new ANS coding: https://en.wikipedia.org/wiki/Asymmetric_numeral_systems https://en.wikipedia.org/wiki/Asymmetric_numeral_systems
- tester756 6y agoGuy who stands behind this has insane CV 2015- Jagiellonian University, Institute of Computer Science, assistant professor, 2013-2014 Purdue University, NSF Center for Science of Information, Postdoctoral researcher (webpage), 2006-2012 Jagiellonian University, Cracow, PhD in Theoretical Physics (thesis) 2004-2010 Jagiellonian University, Cracow, PhD in Theoretical Computer Science (thesis) 2001-2006 Jagiellonian University, Cracow, MSc in Theoretical Physics (thesis) 2000-2005 Jagiellonian University, Cracow, MSc in Theoretical Mathematics (thesis) 1999-2004 Jagiellonian University, Cracow, MSc in Computer Science (thesis)
- shockinglytrue 6y agoIIRC it doesn't use ANS for every compression level, can't remember much more than that, though
- jfkebwjsbx 6y agoIsn't it the opposite? If LZ4 is optimized for decompression speed as you say, then you would want to use it when you read many times, the same file, very fast. From a quick read, ZSTD looks more about saving space while keeping reasonable speeds, both at write and read. And I'd assume there are other algorithms that focus only on size, trading speed for it.
- the8472 6y agoxz would be an example of the latter kind.
- foepys 6y agoDoes xz actually outperform ZSTD anywhere? IIRC even at high compression ratios ZSTD is a lot faster than xz. Edit: Arch Linux switched from xz to ZSTD for their package manager and somebody compared both: https://sysdfree.wordpress.com/2020/01/04/293/ https://sysdfree.wordpress.com/2020/01/04/293/ The Arch Linux developers state that they expect an 0.8% you increase in package size but an 1300% speedup in decompression. Not too shabby. I'm running Arch on my personal system and it's really noticable, especially when I create by own packages, compression doesn't take longer than compiling anymore.