4 ms·

FYI lz4 and zstd are made by the same guy, so there is no surprise both of them complement each other. lz4 targets the (de)compress-as-fast-as-possible domain,

by rakoo 6y ago

FYI lz4 and zstd are made by the same guy, so there is no surprise both of them complement each other. lz4 targets the (de)compress-as-fast-as-possible domain, while zstd is for the rest (good (de)compression with slower speed). For something that is write-once read-many-times, like a filesystem, a good compression algorithm might be more interesting in the future. lz4 is more targeted at write-once read-once, like file transfers for example

jerryX 6y ago
Zstd uses this new ANS coding: https://en.wikipedia.org/wiki/Asymmetric_numeral_systems https://en.wikipedia.org/wiki/Asymmetric_numeral_systems
- tester756 6y ago
  Guy who stands behind this has insane CV 2015- Jagiellonian University, Institute of Computer Science, assistant professor, 2013-2014 Purdue University, NSF Center for Science of Information, Postdoctoral researcher (webpage), 2006-2012 Jagiellonian University, Cracow, PhD in Theoretical Physics (thesis) 2004-2010 Jagiellonian University, Cracow, PhD in Theoretical Computer Science (thesis) 2001-2006 Jagiellonian University, Cracow, MSc in Theoretical Physics (thesis) 2000-2005 Jagiellonian University, Cracow, MSc in Theoretical Mathematics (thesis) 1999-2004 Jagiellonian University, Cracow, MSc in Computer Science (thesis)
- shockinglytrue 6y ago
  IIRC it doesn't use ANS for every compression level, can't remember much more than that, though
jfkebwjsbx 6y ago
Isn't it the opposite? If LZ4 is optimized for decompression speed as you say, then you would want to use it when you read many times, the same file, very fast. From a quick read, ZSTD looks more about saving space while keeping reasonable speeds, both at write and read. And I'd assume there are other algorithms that focus only on size, trading speed for it.
- the8472 6y ago
  xz would be an example of the latter kind.
  - foepys 6y ago
    Does xz actually outperform ZSTD anywhere? IIRC even at high compression ratios ZSTD is a lot faster than xz. Edit: Arch Linux switched from xz to ZSTD for their package manager and somebody compared both: https://sysdfree.wordpress.com/2020/01/04/293/ https://sysdfree.wordpress.com/2020/01/04/293/ The Arch Linux developers state that they expect an 0.8% you increase in package size but an 1300% speedup in decompression. Not too shabby. I'm running Arch on my personal system and it's really noticable, especially when I create by own packages, compression doesn't take longer than compiling anymore.
    - usefulcat 6y ago
      There definitely exist certain kinds of data which xz can compress better or as well but faster than zstd, even taking into account the more extreme compression options offered by zstd. I know because we have many terabytes of such data, and I've done thorough comparisons between xz and zstd at all different compression levels. However in all cases zstd is much faster for decompression. It just happens that getting the best compression ratio in a not insane amount of time is still the better tradeoff for us.
- rakoo 6y ago
  Depends what "read many times" means. - If you mean that the same archive will be distributed to many peers, such as is the case in package distribution, then in practice archives will be read only once by each process, so one "slow" compression will translate into significant gains in added decompression speed. That's the reason Archlinux switched to zstd for its packages (https://www.archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/ https://www.archlinux.org/news/now-using-zstandard-instead-o...) - If you mean that the same archive will be read multiple times by the same machine, I don't really know what kind of scenario that is; I'd deflate the archive into its initial representation once and then let processes access that folder directly. Note that zstd claims that it isn't that much slower in decompression than competitors, even if you always use compressed archives the difference will be minimal zstd was built more or less to "replace" all formats that favor compression over speed. From their benchmarks (which means what it means) whatever the compression/speed ratio you want, zstd is going to be better than all of them, with a hard exception on extremely fast speed that is still the kingdom of lz4.

by rakoo 6y ago

jerryX 6y ago

Zstd uses this new ANS coding: https://en.wikipedia.org/wiki/Asymmetric_numeral_systems https://en.wikipedia.org/wiki/Asymmetric_numeral_systems

tester756 6y ago
Guy who stands behind this has insane CV 2015- Jagiellonian University, Institute of Computer Science, assistant professor, 2013-2014 Purdue University, NSF Center for Science of Information, Postdoctoral researcher (webpage), 2006-2012 Jagiellonian University, Cracow, PhD in Theoretical Physics (thesis) 2004-2010 Jagiellonian University, Cracow, PhD in Theoretical Computer Science (thesis) 2001-2006 Jagiellonian University, Cracow, MSc in Theoretical Physics (thesis) 2000-2005 Jagiellonian University, Cracow, MSc in Theoretical Mathematics (thesis) 1999-2004 Jagiellonian University, Cracow, MSc in Computer Science (thesis)
shockinglytrue 6y ago
IIRC it doesn't use ANS for every compression level, can't remember much more than that, though

jfkebwjsbx 6y ago

Isn't it the opposite? If LZ4 is optimized for decompression speed as you say, then you would want to use it when you read many times, the same file, very fast. From a quick read, ZSTD looks more about saving space while keeping reasonable speeds, both at write and read. And I'd assume there are other algorithms that focus only on size, trading speed for it.

the8472 6y ago
xz would be an example of the latter kind.
- foepys 6y ago
  Does xz actually outperform ZSTD anywhere? IIRC even at high compression ratios ZSTD is a lot faster than xz. Edit: Arch Linux switched from xz to ZSTD for their package manager and somebody compared both: https://sysdfree.wordpress.com/2020/01/04/293/ https://sysdfree.wordpress.com/2020/01/04/293/ The Arch Linux developers state that they expect an 0.8% you increase in package size but an 1300% speedup in decompression. Not too shabby. I'm running Arch on my personal system and it's really noticable, especially when I create by own packages, compression doesn't take longer than compiling anymore.
  - usefulcat 6y ago
    There definitely exist certain kinds of data which xz can compress better or as well but faster than zstd, even taking into account the more extreme compression options offered by zstd. I know because we have many terabytes of such data, and I've done thorough comparisons between xz and zstd at all different compression levels. However in all cases zstd is much faster for decompression. It just happens that getting the best compression ratio in a not insane amount of time is still the better tradeoff for us.
rakoo 6y ago
Depends what "read many times" means. - If you mean that the same archive will be distributed to many peers, such as is the case in package distribution, then in practice archives will be read only once by each process, so one "slow" compression will translate into significant gains in added decompression speed. That's the reason Archlinux switched to zstd for its packages (https://www.archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/ https://www.archlinux.org/news/now-using-zstandard-instead-o...) - If you mean that the same archive will be read multiple times by the same machine, I don't really know what kind of scenario that is; I'd deflate the archive into its initial representation once and then let processes access that folder directly. Note that zstd claims that it isn't that much slower in decompression than competitors, even if you always use compressed archives the difference will be minimal zstd was built more or less to "replace" all formats that favor compression over speed. From their benchmarks (which means what it means) whatever the compression/speed ratio you want, zstd is going to be better than all of them, with a hard exception on extremely fast speed that is still the kingdom of lz4.