9 ms·

LZ4 – Extremely fast compression

307 points by marcobambini 6y ago

AnthonBerg 6y ago
Here’s a fork of the Windows compression tool 7-Zip which has LZ4 support baked in along with some other useful algorithms – the repo has a good comparison of them: https://github.com/mcmilk/7-Zip-zstd/ https://github.com/mcmilk/7-Zip-zstd/ (Linking to this more for the overview than the Windows tool in itself.)
- Jonnax 6y ago
  The frustrating thing about 7zip is that whilst it's opensource, a single author just does code drops. So there's a bunch of forks with useful features that'll never be adopted because there's no collaboration. At least that's what I could tell when I looked into it
  - rav 6y ago
    It sounds like it needs a fork to combine all the forks, similar to what Neovim did as a fork of Vim.
    - 0-_-0 6y ago
      Would that be like... a reverse fork? A defork? A korf?
      - diggernet 6y ago
        Oh heck, let's just call it a spoon.
        OldHand2018 6y ago
        Nice. Really, anything that had solid support would be nice. My company delivers data to customers and our customers use all sorts of operating systems. Plain Zip seems to be our lowest common denominator because we can’t count on our customers being tech-savvy or having an IT department. I really, really, really wish there were more compression standards that were supported on a wide variety of platforms. “Obscure” software package A just doesn’t cut it.
  - nyanpasu64 6y ago
    I liked http://www.e7z.org/ http://www.e7z.org/ for its UI improvements, but it was last updated in 2016 and misses out on several security bug patches.
roncohen 6y ago
another contender is zstd: https://github.com/facebook/zstd https://github.com/facebook/zstd. It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed. Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data, particularly useful for compression of small pieces of data.
- moonchild 6y ago
  > It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed Per the table at [0], zstd provides only a slight improvement in compression ratio, and in exchange is about half the speed of lz4. They both have their place. 0. https://facebook.github.io/zstd/ https://facebook.github.io/zstd/
  - baybal2 6y ago
    LZ4 has branchless decompression, and lower cache footprint, thus it can work on low end, and non-desktop CPUs equally well. zstd, brotly, snappy were seemingly all made with high end x86 capabilities in mind.
    - jorangreef 6y ago
      I also appreciate LZ4's simplicity and tiny code footprint. zstd is brilliant as well, but in terms of code base it's a whole other beast.
      - makapuf 6y ago
        Yes decompression on baremetal cortex m4 is a mere hundreds of bytes, you can decompress it from flash directly to its output buffer.
        Teknoman117 6y ago
        I've used it in bootloaders that have slow transfer mechanisms (uart, i2c) to get whatever speedup I can for a few hundred bytes of binary.
bombcar 6y ago
LZ4 is so fast there’s almost no reason to NOT have it on for zfs volumes.
- tpetry 6y ago
  Zfs is so fast it should be the default for everything where a slight compression may benefit the system: disk io, network transfers, ...
  - xianwen 6y ago
    I'm curious. I use btrfs daily. Although I have been interested in using zfs, I haven't yet gotten the time. In your experience, is zfs faster than btrfs?
    - kbumsik 6y ago
      Yes. Much faster. Especially for HDDs. But at a cost of a lot of RAM. Also lz4 compression can speed up your HDDs up to 10x (!) to read and 3x to write. [1, see "To compress, or not to compress, that is the question" section.] But it's going to have a considerably higher CPU usage as well. [1]: https://calomel.org/zfs_raid_speed_capacity.html https://calomel.org/zfs_raid_speed_capacity.html
      - ksec 6y ago
        >But it's going to have a considerably higher CPU usage as well. I am going to assume in three to four years time this wouldn't be a problem? I mean a 16nm Quad Core ARM Cortex SoC are only $15. Unfortunately no consumer NAS are implementing ZFS. ( TrueNAS offering isn't really consumer NAS )
        kbumsik 6y ago
        I am not sure about cheap ARM devices but I am using an old Haswell i5-4670 and it is more than enough. So it won't be issue later. Also, when you are talking about consumer NAS, the real problem is that any low-end systems can saturate the gigabit network (100MB/s) very easily so investing on extra resources for ZFS doesn't make difference. At least a 10Gbe network (which is beyond the average consumer) is required to actually make it useful.
viktorcode 6y ago
Weird that the page doesn't list macOS or other Apple's OS in the list of operating systems with LZ4 support.
- 90minuteAPI 6y ago
  It’s even in the recent-ish high-ish level Compression framework: https://developer.apple.com/documentation/compression/compression_lz4 https://developer.apple.com/documentation/compression/compre...
jakozaur 6y ago
LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data over network zstd is quite profitable. Even network attached disk AWS EBS or AWS S3 it can be a hugely profitable.
- deleted 6y ago
  [deleted]
- contravariant 6y ago
  If it's fast enough you wouldn't only be able to feed more data you'd be able to access it faster as well since you need less bandwidth. That said it can be quite tricky to rewrite something to efficiently work on compressed data.
  - mrec 6y ago
    GPU texture compression formats would be the obvious example. With the additional constraint that they need to support efficient random access.
- zX41ZdbW 6y ago
  Using LZ4 can easily improve performance even if all data reside in memory. This is the case in ClickHouse: if data is compressed, we decompress it in blocks that fit in CPU cache and then perform data processing inside cache; if data is uncompressed, larger amount of data is read from memory. Strictly speaking, LZ4 data decompression (typically 3 GB/sec) is slower than memcpy (typically 12 GB/sec). But when using e.g. 128 CPU cores, LZ4 decompression will scale up to memory bandwidth (typically 150 GB/sec) as well as memcpy. And memcpy is wasting more memory bandwidth by reading uncompressed data while LZ4 decompression reads compressed data.
  - pdimitar 6y ago
maeln 6y ago
If I remember correctly, it is very popular in video games because it is faster to load compressed assets from disk and decompress them in memory than loading the uncompressed assets from disk, even on an SSD.
- Jonnax 6y ago
  I did a search and found this patch note for a game which has a comparison graph comparing cold load, hot load and size of some algorithms: https://store.steampowered.com/news/app/991270/view/1806446683762590791 https://store.steampowered.com/news/app/991270/view/18064466...
- gens 6y ago
  It is used, and most every other compression technique was/is used in video games. I was thinking of using LZ4, but it doesn't really work that great on floating point, and images are already compressed (png, jpg, and even BCn, can't be compressed much further). So idk. Good thing about lz4 is that it's very simple[0] and probably faster then memcpy(). http://ticki.github.io/blog/how-lz4-works/ http://ticki.github.io/blog/how-lz4-works/
  - garaetjjte 6y ago
    >and even BCn, can't be compressed much further S3TC is block compression, so if there is repeating data in images it will compress quite well.
    - gens 6y ago
      I tried compressing BC7 (from AMD-s Compressonator) using lz4c and it wasn't much. Just re-ran it (with -hc, version 1.9.3(latest now)) and: "Compressed 5592544 bytes into 5413783 bytes ==> 96.80%". 7z with "7z a -mx=9 -t7z asd.7z albedo_PNG_BC7_1.KTX" does 5592544 bytes to 4844707 bytes (down to ~87%). Original file is 10227047 bytes (PNG, RGBA), i can't remember if the ktx has mipmaps. EDIT: Note that the image is fairly noisy (gravel). Could/should be better with more "artificial" textures. I don't know if ktx does some extra compression, but, looking at it, i doubt it. PS I think that BC could be massaged at compression to be better compressible, and i think i read something about that. Don't remember.
contravariant 6y ago
If they're including transfer time it'd be fun to see how raw data performs.
ant6n 6y ago
How does it work?
- yotamoron 6y ago
  It compresses stuff. Amazing.
- cupofcoffee 6y ago
  Most interesting question on this thread and gets downvoted. :)
- vnxli 6y ago
  I think it’s middle-out compression
- akanet 6y ago
  I found this blog post describing some of the mechanics, but I am still interested in reading how this may be implemented branchlessly. http://ticki.github.io/blog/how-lz4-works/ http://ticki.github.io/blog/how-lz4-works/
- artifact_44 6y ago
  Ring buffer and huffman codes?
yotamoron 6y ago
LZ4 rocks. Used it in the past with great results (much less CPU intensive then the GZIP we were using, still getting good compression).
jansan 6y ago
The wikipedia article about LZ4 says that compression is slightly worse, compression speed is comparable, and decompression can be significantly slower compared to LZO. Can anyone enlighten me why I should not used LZO? I do not know anything about either algorithms, but if I switch from deflate I want to mak the right decision.
- PSeitz 6y ago
  As far as I know LZ4 is much faster that most compression algorithms, with decompression speeds of over 4GB/s
- JanecekPetr 6y ago
  You misread that. It says that decompression is significantly faster than LZO. In fact, decompression speeds of up to 5 GB/s is one of the reasons LZ4 is so popular.
- adzm 6y ago
  LZO is also GPL which can be a conflict, compared with the BSD license of LZ4.
st_goliath 6y ago
A while ago I did some simplistic SquashFS pack/unpack benchmarks[1][2]. I was primarily interested in looking at the behavior of my thread-pool based packer, but as a side effect I got a comparison of compressor speed & ratios over the various available compressors for my Debian test image. I must say that LZ4 definitely stands out for both compression and uncompression speed, while still being able to cut the data size in half, making it probably quite suitable for life filesystems and network protocols. Particularly interesting was also comparing Zstd and LZ4[3], the former being substantially slower, but at the same time achieving a compression ratio somewhere between zlib and xz, while beating both in time (in my benchmark at least). [1] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/benchmark.txt https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/... [2] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/benchmark.ods https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/... [3] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/benchmark.txt#L233 https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/...
- older 6y ago
  F2FS supports compression with LZ4 since Linux 5.6.
- 1996 6y ago
  > lz4 (...) probably quite suitable for life filesystems and network protocols Actually, no. lz4 is less suitable than zstd for filesystems. BTW, lz4 is present in many mozilla tools like thunderbird: it's represented by its bastard child lz4json, which is diverging by just the headers don't work with regular lz4 tools > achieving a compression ratio somewhere between zlib and xz, while beating both in time (in my Your observation is correct: zstd is now standard and the default on openzfs 2.0, replacing lz4. The 19 compression variants offer more flexibility than just lz4- another strength is the decode time is not a function of the compression factor, which is something good on coldish storage that's rarely updated.
  - yjftsjthsd-h 6y ago
    > Actually, no. lz4 is less suitable than zstd for filesystems. Why's that? What benefit would I get from switching? Is it workload-dependent? EDIT: To be clear, I'm not disagreeing; if zstd will work better, I want to know about it so that I can switch my pools to use it.
PSeitz 6y ago
I ported the block format to Rust matching the C implementation in performance and ratio. https://github.com/pseitz/lz4_flex https://github.com/pseitz/lz4_flex
- pmarreck 6y ago
  NICE! Well done!
- dgacmu 6y ago
  Has anyone written the appropriate Reader wrappers to use this with file io? (Asking b/c a quick search didn't turn anything up.)
  - PSeitz 6y ago
    file io should come with the frame format, which is not yet implemented. For the block format it's not really suited.
mschuetz 6y ago
Not sure how LZ4 compares but what convinced me to use brotli over some other compression libraries was how trivialy easy it was to integrate in C++ as well as in javascript. No effort at all. I wish more lib providers would make their things so approachable.
zX41ZdbW 6y ago
It is possible to make LZ4 decompression even faster, here's how: https://habr.com/en/company/yandex/blog/457612/ https://habr.com/en/company/yandex/blog/457612/
zX41ZdbW 6y ago
It is very interesting that compression libraries from Yann Collet outperform their Google counterparts by all means: lz4 >> snappy zstd >> brotli
- specialist 6y ago
  Maybe Adjacent: Corecursive episode w/ Daniel Lemire was just terrific. https://corecursive.com/frontiers-of-performance-with-daniel-lemire/ https://corecursive.com/frontiers-of-performance-with-daniel... Some people are just really good at performance sensitive stuff.
mkhnews 6y ago
LZ4 is awesome, thank you for it. Another interesting and fast compressor I saw a while back was Density.
izackp 6y ago
Some interesting and related projects: https://github.com/strigeus/ipzip https://github.com/strigeus/ipzip - TCP/IP Packet Compressor with LZ4 support https://github.com/centaurean/density https://github.com/centaurean/density - Extremely fast de/compression
maxpert 6y ago
I've personally used LZ4 on production a scale that really proves how using a compression like LZ4 is more efficient than uncompressed data https://doordash.engineering/2019/01/02/speeding-up-redis-with-compression/ https://doordash.engineering/2019/01/02/speeding-up-redis-wi...
- pottertheotter 6y ago
  I've been using it reliably for the past several years for government documents needed for academic research. ~20mm documents that range from very small to 10s of MB. Each one is compressed using LZ4 and, overall, it's ~10% of the uncompressed size. Compressed it's about 2TB of data. It's unbelievable how fast they decompress.
TwoBit 6y ago
OK but how does it compare to Kraken?
minitoar 6y ago
Interana uses lz4 for compressing certain data types, namely list-type data. We found it offers the best decompression speed for reasonably good compression ratio. We actually use the high compression variant.
beefok 6y ago
I've been hunting for a good decompressor to use in a low ram microcontroller, for instance, an ARM Cortex M0. I've read an article [1] on LZ4 decompression on the Cortex, but I couldn't understand what kind of memory requirements are needed. I've yet to really understand what kind of footprint LZ4 uses, and if it's dependent on dictionary size used to compress. What if I have, say, 4KB that I could use to store in-place decompression. Is that related to the compression ratio? [1] https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/lz4-decompression-routine-for-cortex-m0-and-later https://community.arm.com/developer/ip-products/processors/b...
InfiniteRand 6y ago
I'm not a fan of the stacked bar charts, I like the table of data for "Benchmarks" on the github source page: https://github.com/lz4/lz4 https://github.com/lz4/lz4 It makes it very clear where LZ4 fits into comparisons with compression speed, decompression speed and compression ratio
mintyc 6y ago
https://blog.logrocket.com/rust-compression-libraries/ https://blog.logrocket.com/rust-compression-libraries/ Although implementations arein rust, I assume the provided benchmarks are representative of any optimised implementation... Many compressor algorithms are compared on several data sets. The results tables show compressed size, compression and decompression times for a number of normal and pathological cases. Get a good feel about strengths and weaknesses. Some algs really go downhill in pathological cases, such as with random data. Do consider encryption too though you probably want to do that on the compressed data set where possible. Sometimes external encryption means you will be stuck with something close to pathological...

307 points by marcobambini 6y ago

Here’s a fork of the Windows compression tool 7-Zip which has LZ4 support baked in along with some other useful algorithms – the repo has a good comparison of them: https://github.com/mcmilk/7-Zip-zstd/ https://github.com/mcmilk/7-Zip-zstd/ (Linking to this more for the overview than the Windows tool in itself.)

Jonnax 6y ago
The frustrating thing about 7zip is that whilst it's opensource, a single author just does code drops. So there's a bunch of forks with useful features that'll never be adopted because there's no collaboration. At least that's what I could tell when I looked into it
- rav 6y ago
  It sounds like it needs a fork to combine all the forks, similar to what Neovim did as a fork of Vim.
  - 0-_-0 6y ago
    Would that be like... a reverse fork? A defork? A korf?
    - diggernet 6y ago
      Oh heck, let's just call it a spoon.
      - OldHand2018 6y ago
        Nice. Really, anything that had solid support would be nice. My company delivers data to customers and our customers use all sorts of operating systems. Plain Zip seems to be our lowest common denominator because we can’t count on our customers being tech-savvy or having an IT department. I really, really, really wish there were more compression standards that were supported on a wide variety of platforms. “Obscure” software package A just doesn’t cut it.
- nyanpasu64 6y ago
  I liked http://www.e7z.org/ http://www.e7z.org/ for its UI improvements, but it was last updated in 2016 and misses out on several security bug patches.

roncohen 6y ago

another contender is zstd: https://github.com/facebook/zstd https://github.com/facebook/zstd. It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed. Additionally it offers a training mode to tune the algorithm to increase compression ratio on specific types of data, particularly useful for compression of small pieces of data.

moonchild 6y ago
> It typically offers better compression ratios than LZ4 at a slight (depending on your data) cost in speed Per the table at [0], zstd provides only a slight improvement in compression ratio, and in exchange is about half the speed of lz4. They both have their place. 0. https://facebook.github.io/zstd/ https://facebook.github.io/zstd/
- baybal2 6y ago
  LZ4 has branchless decompression, and lower cache footprint, thus it can work on low end, and non-desktop CPUs equally well. zstd, brotly, snappy were seemingly all made with high end x86 capabilities in mind.
  - jorangreef 6y ago
    I also appreciate LZ4's simplicity and tiny code footprint. zstd is brilliant as well, but in terms of code base it's a whole other beast.
    - makapuf 6y ago
      Yes decompression on baremetal cortex m4 is a mere hundreds of bytes, you can decompress it from flash directly to its output buffer.
      - Teknoman117 6y ago
        I've used it in bootloaders that have slow transfer mechanisms (uart, i2c) to get whatever speedup I can for a few hundred bytes of binary.

bombcar 6y ago

LZ4 is so fast there’s almost no reason to NOT have it on for zfs volumes.

tpetry 6y ago
Zfs is so fast it should be the default for everything where a slight compression may benefit the system: disk io, network transfers, ...
- xianwen 6y ago
  I'm curious. I use btrfs daily. Although I have been interested in using zfs, I haven't yet gotten the time. In your experience, is zfs faster than btrfs?
  - kbumsik 6y ago
    Yes. Much faster. Especially for HDDs. But at a cost of a lot of RAM. Also lz4 compression can speed up your HDDs up to 10x (!) to read and 3x to write. [1, see "To compress, or not to compress, that is the question" section.] But it's going to have a considerably higher CPU usage as well. [1]: https://calomel.org/zfs_raid_speed_capacity.html https://calomel.org/zfs_raid_speed_capacity.html
    - ksec 6y ago
      >But it's going to have a considerably higher CPU usage as well. I am going to assume in three to four years time this wouldn't be a problem? I mean a 16nm Quad Core ARM Cortex SoC are only $15. Unfortunately no consumer NAS are implementing ZFS. ( TrueNAS offering isn't really consumer NAS )
      - kbumsik 6y ago
        I am not sure about cheap ARM devices but I am using an old Haswell i5-4670 and it is more than enough. So it won't be issue later. Also, when you are talking about consumer NAS, the real problem is that any low-end systems can saturate the gigabit network (100MB/s) very easily so investing on extra resources for ZFS doesn't make difference. At least a 10Gbe network (which is beyond the average consumer) is required to actually make it useful.

viktorcode 6y ago

Weird that the page doesn't list macOS or other Apple's OS in the list of operating systems with LZ4 support.

90minuteAPI 6y ago
It’s even in the recent-ish high-ish level Compression framework: https://developer.apple.com/documentation/compression/compression_lz4 https://developer.apple.com/documentation/compression/compre...

jakozaur 6y ago

LZ4 is so fast, that in make sense to use it everywhere over uncompressed data. Even storing items in-memory compressed sometimes is profitable as you can fit more items in memory. Still zstd offers way better compression and got variable difficulty factor: https://github.com/facebook/zstd https://github.com/facebook/zstd Decompression is always fast, but you can trade off compression vs. ratio factor. In general if send data over network zstd is quite profitable. Even network attached disk AWS EBS or AWS S3 it can be a hugely profitable.

deleted 6y ago
[deleted]
contravariant 6y ago
If it's fast enough you wouldn't only be able to feed more data you'd be able to access it faster as well since you need less bandwidth. That said it can be quite tricky to rewrite something to efficiently work on compressed data.
- mrec 6y ago
  GPU texture compression formats would be the obvious example. With the additional constraint that they need to support efficient random access.
zX41ZdbW 6y ago
Using LZ4 can easily improve performance even if all data reside in memory. This is the case in ClickHouse: if data is compressed, we decompress it in blocks that fit in CPU cache and then perform data processing inside cache; if data is uncompressed, larger amount of data is read from memory. Strictly speaking, LZ4 data decompression (typically 3 GB/sec) is slower than memcpy (typically 12 GB/sec). But when using e.g. 128 CPU cores, LZ4 decompression will scale up to memory bandwidth (typically 150 GB/sec) as well as memcpy. And memcpy is wasting more memory bandwidth by reading uncompressed data while LZ4 decompression reads compressed data.
- pdimitar 6y ago

maeln 6y ago

If I remember correctly, it is very popular in video games because it is faster to load compressed assets from disk and decompress them in memory than loading the uncompressed assets from disk, even on an SSD.

Jonnax 6y ago
I did a search and found this patch note for a game which has a comparison graph comparing cold load, hot load and size of some algorithms: https://store.steampowered.com/news/app/991270/view/1806446683762590791 https://store.steampowered.com/news/app/991270/view/18064466...
gens 6y ago
It is used, and most every other compression technique was/is used in video games. I was thinking of using LZ4, but it doesn't really work that great on floating point, and images are already compressed (png, jpg, and even BCn, can't be compressed much further). So idk. Good thing about lz4 is that it's very simple[0] and probably faster then memcpy(). http://ticki.github.io/blog/how-lz4-works/ http://ticki.github.io/blog/how-lz4-works/
- garaetjjte 6y ago
  >and even BCn, can't be compressed much further S3TC is block compression, so if there is repeating data in images it will compress quite well.
  - gens 6y ago
    I tried compressing BC7 (from AMD-s Compressonator) using lz4c and it wasn't much. Just re-ran it (with -hc, version 1.9.3(latest now)) and: "Compressed 5592544 bytes into 5413783 bytes ==> 96.80%". 7z with "7z a -mx=9 -t7z asd.7z albedo_PNG_BC7_1.KTX" does 5592544 bytes to 4844707 bytes (down to ~87%). Original file is 10227047 bytes (PNG, RGBA), i can't remember if the ktx has mipmaps. EDIT: Note that the image is fairly noisy (gravel). Could/should be better with more "artificial" textures. I don't know if ktx does some extra compression, but, looking at it, i doubt it. PS I think that BC could be massaged at compression to be better compressible, and i think i read something about that. Don't remember.

contravariant 6y ago

If they're including transfer time it'd be fun to see how raw data performs.

ant6n 6y ago

How does it work?

yotamoron 6y ago
It compresses stuff. Amazing.
cupofcoffee 6y ago
Most interesting question on this thread and gets downvoted. :)
vnxli 6y ago
I think it’s middle-out compression
akanet 6y ago
I found this blog post describing some of the mechanics, but I am still interested in reading how this may be implemented branchlessly. http://ticki.github.io/blog/how-lz4-works/ http://ticki.github.io/blog/how-lz4-works/
artifact_44 6y ago
Ring buffer and huffman codes?

yotamoron 6y ago

LZ4 rocks. Used it in the past with great results (much less CPU intensive then the GZIP we were using, still getting good compression).

jansan 6y ago

The wikipedia article about LZ4 says that compression is slightly worse, compression speed is comparable, and decompression can be significantly slower compared to LZO. Can anyone enlighten me why I should not used LZO? I do not know anything about either algorithms, but if I switch from deflate I want to mak the right decision.

PSeitz 6y ago
As far as I know LZ4 is much faster that most compression algorithms, with decompression speeds of over 4GB/s
JanecekPetr 6y ago
You misread that. It says that decompression is significantly faster than LZO. In fact, decompression speeds of up to 5 GB/s is one of the reasons LZ4 is so popular.
adzm 6y ago
LZO is also GPL which can be a conflict, compared with the BSD license of LZ4.

st_goliath 6y ago

A while ago I did some simplistic SquashFS pack/unpack benchmarks[1][2]. I was primarily interested in looking at the behavior of my thread-pool based packer, but as a side effect I got a comparison of compressor speed & ratios over the various available compressors for my Debian test image. I must say that LZ4 definitely stands out for both compression and uncompression speed, while still being able to cut the data size in half, making it probably quite suitable for life filesystems and network protocols. Particularly interesting was also comparing Zstd and LZ4[3], the former being substantially slower, but at the same time achieving a compression ratio somewhere between zlib and xz, while beating both in time (in my benchmark at least). [1] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/benchmark.txt https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/... [2] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/benchmark.ods https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/... [3] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/benchmark.txt#L233 https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/...

older 6y ago
F2FS supports compression with LZ4 since Linux 5.6.
1996 6y ago
> lz4 (...) probably quite suitable for life filesystems and network protocols Actually, no. lz4 is less suitable than zstd for filesystems. BTW, lz4 is present in many mozilla tools like thunderbird: it's represented by its bastard child lz4json, which is diverging by just the headers don't work with regular lz4 tools > achieving a compression ratio somewhere between zlib and xz, while beating both in time (in my Your observation is correct: zstd is now standard and the default on openzfs 2.0, replacing lz4. The 19 compression variants offer more flexibility than just lz4- another strength is the decode time is not a function of the compression factor, which is something good on coldish storage that's rarely updated.
- yjftsjthsd-h 6y ago
  > Actually, no. lz4 is less suitable than zstd for filesystems. Why's that? What benefit would I get from switching? Is it workload-dependent? EDIT: To be clear, I'm not disagreeing; if zstd will work better, I want to know about it so that I can switch my pools to use it.

PSeitz 6y ago

I ported the block format to Rust matching the C implementation in performance and ratio. https://github.com/pseitz/lz4_flex https://github.com/pseitz/lz4_flex

pmarreck 6y ago
NICE! Well done!
dgacmu 6y ago
Has anyone written the appropriate Reader wrappers to use this with file io? (Asking b/c a quick search didn't turn anything up.)
- PSeitz 6y ago
  file io should come with the frame format, which is not yet implemented. For the block format it's not really suited.

mschuetz 6y ago

Not sure how LZ4 compares but what convinced me to use brotli over some other compression libraries was how trivialy easy it was to integrate in C++ as well as in javascript. No effort at all. I wish more lib providers would make their things so approachable.

zX41ZdbW 6y ago

It is possible to make LZ4 decompression even faster, here's how: https://habr.com/en/company/yandex/blog/457612/ https://habr.com/en/company/yandex/blog/457612/

zX41ZdbW 6y ago

It is very interesting that compression libraries from Yann Collet outperform their Google counterparts by all means: lz4 >> snappy zstd >> brotli

specialist 6y ago
Maybe Adjacent: Corecursive episode w/ Daniel Lemire was just terrific. https://corecursive.com/frontiers-of-performance-with-daniel-lemire/ https://corecursive.com/frontiers-of-performance-with-daniel... Some people are just really good at performance sensitive stuff.

mkhnews 6y ago

LZ4 is awesome, thank you for it. Another interesting and fast compressor I saw a while back was Density.

izackp 6y ago

Some interesting and related projects: https://github.com/strigeus/ipzip https://github.com/strigeus/ipzip - TCP/IP Packet Compressor with LZ4 support https://github.com/centaurean/density https://github.com/centaurean/density - Extremely fast de/compression

maxpert 6y ago

I've personally used LZ4 on production a scale that really proves how using a compression like LZ4 is more efficient than uncompressed data https://doordash.engineering/2019/01/02/speeding-up-redis-with-compression/ https://doordash.engineering/2019/01/02/speeding-up-redis-wi...

pottertheotter 6y ago
I've been using it reliably for the past several years for government documents needed for academic research. ~20mm documents that range from very small to 10s of MB. Each one is compressed using LZ4 and, overall, it's ~10% of the uncompressed size. Compressed it's about 2TB of data. It's unbelievable how fast they decompress.

TwoBit 6y ago

OK but how does it compare to Kraken?

minitoar 6y ago

Interana uses lz4 for compressing certain data types, namely list-type data. We found it offers the best decompression speed for reasonably good compression ratio. We actually use the high compression variant.

beefok 6y ago

I've been hunting for a good decompressor to use in a low ram microcontroller, for instance, an ARM Cortex M0. I've read an article [1] on LZ4 decompression on the Cortex, but I couldn't understand what kind of memory requirements are needed. I've yet to really understand what kind of footprint LZ4 uses, and if it's dependent on dictionary size used to compress. What if I have, say, 4KB that I could use to store in-place decompression. Is that related to the compression ratio? [1] https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/lz4-decompression-routine-for-cortex-m0-and-later https://community.arm.com/developer/ip-products/processors/b...

InfiniteRand 6y ago

I'm not a fan of the stacked bar charts, I like the table of data for "Benchmarks" on the github source page: https://github.com/lz4/lz4 https://github.com/lz4/lz4 It makes it very clear where LZ4 fits into comparisons with compression speed, decompression speed and compression ratio

mintyc 6y ago

https://blog.logrocket.com/rust-compression-libraries/ https://blog.logrocket.com/rust-compression-libraries/ Although implementations arein rust, I assume the provided benchmarks are representative of any optimised implementation... Many compressor algorithms are compared on several data sets. The results tables show compressed size, compression and decompression times for a number of normal and pathological cases. Get a good feel about strengths and weaknesses. Some algs really go downhill in pathological cases, such as with random data. Do consider encryption too though you probably want to do that on the compressed data set where possible. Sometimes external encryption means you will be stuck with something close to pathological...