10 ms·
The LZ4 introduced in PostgreSQL 14 provides faster compression
- AtNightWeCode 5y agoThe LZ77 family of compression is really nice. So many use cases. ZSTD has saved me many times. But it is perhaps not the right tool in this case. Too bad so many tools are ZIP exclusive.
- AtNightWeCode 5y agoAnother one of these magical down votes on this garbage forum. Seriously, what is there to down vote. LZ77 is not nice? ZSTD is the the right tool for Postgres compression even though it does not perform as well for smaller amounts of data? It is good to waste so much time in zipping things like the horrors of Docker?
- klodolph 5y agoLZ4 is just such a nice little algorithm for compression. It's downright simple, and fast, and still works. It's so simple seems like it should have been invented long ago, but it's only 10 years old.
- rosetremiere 5y agoI'd like to implement it as an exercise. Any recommendation on a good resource for that purpose?
- danbst 5y agoThis is explanation from author http://fastcompression.blogspot.com/2011/05/lz4-explained.html http://fastcompression.blogspot.com/2011/05/lz4-explained.ht...
- mjevans 5y agoInitial release (according to Wikipedia) 24 April 2011. It is derived from the LZ77 family of compression algorithms, which have papers published in 1977 and 1978 and thus US patent expiration around the year 2000. It is understandable that it would take time for discourse, inclusion in educational curriculum, and programmers feeling free to tinker with the concepts to produce useful new art. It's almost like patents are a bad thing for the progress of science and skilled arts.
- speedgoose 5y agoI'm very thankful we don't have software patents in Europe. And you should be thankful too if you use vlc or ffmpeg for example.
- klodolph 5y agoWas LZ77 patented? As far as I can tell, patents related to various techniques in LZ77 were filed by various parties in the mid 80s and 90s, but that wouldn't matter, since LZ77 was already published at that point. PNG was explicitly designed to be unencumbered by patents, and it was released in 1996.
- maskros 5y agoI take it you don't remember the Unisys GIF patent licensing mess that prompted the development of the PNG format? https://en.wikipedia.org/wiki/GIF#Unisys_and_LZW_patent_enforcement https://en.wikipedia.org/wiki/GIF#Unisys_and_LZW_patent_enfo...
- deleted 5y ago[deleted]
- st_goliath 5y agoYes, but GIF is based on LZW, as the article you linked to says. LZW was derived from LZ78. They both IIRC use a dictionary coding approach with an explicit dictionary that is constructed during processing. Indices into the dictionary are used to encode repetitions. PNG uses deflate, which is based on a combination of Huffman coding and LZSS, which is derived from LZ77. LZ77 encodes repetitions using back references into a sliding window. To my understanding, LZSS also uses a dictionary based approach, but only for finding such repetitions during encoding. So this does not answer the question if LZ77 was patented or not. The Wikipedia article about LZ77 does cite a patent (https://en.wikipedia.org/wiki/LZ77_and_LZ78#cite_note-3 https://en.wikipedia.org/wiki/LZ77_and_LZ78#cite_note-3) and somebody in the talk page mentions that there's been a law suite (https://en.wikipedia.org/wiki/Talk:LZ77_and_LZ78#Patent_contravercies https://en.wikipedia.org/wiki/Talk:LZ77_and_LZ78#Patent_cont...). However, quickly scanning over the patent, it seems to discuss an improved algorithm based on LZ77, perhaps it relates to LZS? (Note: not LZSS from above; this is again is a different algorithm) The LZS page mentions a law suite sounding similar to the one from the talk page, but contains no direct reference to the patent: https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Stac#Patents https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Sta... It also states that the LZS patent was filed in 1993 and expired in 2007 (due to not paying their fees).
- miohtama 5y agoLZ4 explained for those who got interested http://fastcompression.blogspot.com/2011/05/lz4-explained.html http://fastcompression.blogspot.com/2011/05/lz4-explained.ht...
- fwip 5y agoLike sibling comment said, it's basically just LZ77 with some different parameters, tuned for speed.
- pella 5y agoin the dev PG15 LZ4 WAL compression is enabled https://github.com/postgres/postgres/commit/4035cd5d4eee4dae797bfc77ab07f8dcd8781b41 https://github.com/postgres/postgres/commit/4035cd5d4eee4dae... "Add support for LZ4 with compression of full-page writes in WAL The logic is implemented so as there can be a choice in the compression used when building a WAL record, and an extra per-record bit is used to track down if a block is compressed with PGLZ, LZ4 or nothing. wal_compression, the existing parameter, is changed to an enum with support for the following backward-compatible values: - "off", the default, to not use compression. - "pglz" or "on", to compress FPWs with PGLZ. - "lz4", the new mode, to compress FPWs with LZ4. Benchmarking has showed that LZ4 outclasses easily PGLZ. ZSTD would be also an interesting choice, but going just with LZ4 for now makes the patch minimalistic as toast compression is already able to use LZ4, so there is no need to worry about any build-related needs for this implementation. Author: Andrey Borodin, Justin Pryzby Reviewed-by: Dilip Kumar, Michael Paquier"
- indeyets 5y agozstd would be great, indeed. it's much more performant and should be a great case for database
- ahartmetz 5y ago> zstd [...] much more performant Actually, that's not true. LZ4 is faster at compression and decompression, zstd (even at the minimum setting) is slower and has higher compression ratio.
- zaarn 5y agozstd-fast is almost as fast as LZ4 and gets a tiny bit better compression, though not a lot of software supports that mode. ZFS does, hence I have some numbers to back that up.
- strainer 5y agoI just had a little look into these somewhat new zstd options --fast 1 to fast 5. fast 5 does seem almost equivalent to lz4 performance in compression ratio and compression 'speed' (or cpu load) except decompression 'speed' is still almost half of lz4, so at that extreme of zstds options it is squarely beaten by lz4. But in terms of compression ratio - which in most cases should be the more crucial characteristic for any compressor than cpu load - this setting seems to be well under zstd's knee of diminishing returns. --fast 1 has about 15% more cpu load for about 112% of compression performance than --fast5. And then zstds lightest non-fast setting (zstd-1) for 50% more load in compression and decompression, compresses 32% better than --fast5 (and lzo and lz4). The decompression load is an eye catching feature of lz4, but its compression ratio is not just marginally compromised, it is a much less powerful compressor in that prime characteristic than zstds weakest non-fast setting. I can appreciate why Btrfs is in no hurry to include lz4 while it supports zstd:1 to 15. Although it would be interesting to test if lz4 could have a niche on fast storage combined with slow cpus.
- drej 5y agoI remember benchmarking fast compression algorithms a while back and had lz4 as one of the top contenders, but I remember going with Snappy over lz4, because I ran into some compatibility issues in lz4 land... I can't recall any specifics, but I think it was some Hadoop libraries not being able to read files lz4-compressed by some other versions of the library. Has anyone run into the same issue? I'm considering reopening this investigation (even though I'm very happy with Snappy).
- nonesuchluck 5y agoSimilar problem I've encountered is AWS Athena not supporting Parquet files compressed with Zstd. I re-wrote files with Snappy to get around it.
- olavgg 5y agoIf you are using ZFS, I strongly recommend using LZ4 or ZSTD compression with PostgreSQL. Performance is still awesome. On average I get 2x compressionratio with LZ4 and 4x with ZSTD. With this, you are compressing everything, not just columns. And ZFS has dynamic block sizes which works really great together with compression. For example a 8kb PostgreSQL page, may be stored as a 1kb compressed block on disk.
- RedShift1 5y agoSo disable compression in Postgres and do all compression in ZFS?
- mattashii 5y agoPostgres' own compression helps with memory usage in buffers, saving IOps, so I'd keep my compression in PostgreSQL enabled
- throwdbaaway 5y agoCompressed ARC has been a thing in ZFS for a long time. So let's say you allow ZFS cache to take 50% of total memory. With 4x compress ratio from ZSTD, you will be able to cache 2x the total memory. It is amazing.
- deleted 5y ago[deleted]
- nix23 5y agoYou should not use ARC for pg data, since postgres has it's own cache, otherwise you just have two caches with the same data in it.
- throwdbaaway 5y agoUnfortunately that's just how postgres works? I thought the rule of thumb is still to allocate 25% of memory for postgres 's shared buffer, and 50% of memory for OS buffer cache (i.e. ARC when using ZFS)?
- francoisdevlin 5y agoHonest question here, slightly of topic. How do you handle indexing a compressed column? Or is this considered an anti-pattern?
- roenxi 5y agoAn index is typically a mapping of values to database pages. Nothing about that cares whether the data has been compressed or not. It may or may not be compressed, but it is still on the page somewhere.
- herehere1x 5y agoTo be clear, an index is compression agnostic, it's the uncompressed value that will appear in the index, or the value selected e.g. bigJsonbDoc->>'email' is that right?
- GrayShade 5y agoIndexes are generally a mapping from the key to a row location (or sometimes to the primary key of the row). You can compress the index itself, but it's usually accessed randomly, so it might be detrimental. Depending on the database, you might have compressed pages (sets of consecutive rows), rows, or columns in a row. Postgres does the latter (it's called TOAST) where the large columns are compressed and/or stored in a hidden table. So you first search the index for the key, figure out where your row is, read the row from the table, then, if needed, you read the columns you need from the TOAST table and uncompress them.
- iamflimflam1 5y agoI implemented LZ77 when I was at university as I was trying to load gifs for an image processing project. I was absolutely blown away by the simplicity and elegance algorithm. Need to have a look at LZ4.
- dunham 5y agoI believe GIFs use LZW which is based on LZ78 - the one that builds a dictionary as it goes. (It's been decades since I looked at this one.) LZ77 emits verbatim text and instructions to go back x and copy y characters. A lot of compressors are based on this, including PNG, gzip, lz4, snappy. I think this is partially due to earlier patent issues with LZ78. (LZ4 or Snappy would be a good intro, gzip has extra stuff like huffman encoding tacked on.) You might also be interested in the Burrows-Wheeler transform, which bzip2 is based on. It's completely different and kinda magical. The original paper is here: https://www.hpl.hp.com/techreports/Compaq-DEC/SRC-RR-124.pdf
- minitoar 5y agoInterana uses lz4 in its column store for certain data types as well. It uses the “hc” variant.
- polskibus 5y agoNot sure about postgresql, but in MS SQL turning on the page or row compression usually improves the performance AND storage, esp. on spinning discs, because it's faster to transfer compressed + decompress on the fly that transfer uncompressed - which is a win-win situation.
- anonu 5y agoWhat about compression for numeric-type columns? When will PostgreSQL have full table compression for row-oriented tables? When will PostgreSQL support column-oriented tables? PostgreSQL is amazing - but these 2 missing pieces create quite a large functionality gap IMO.
- pella 5y ago> When will PostgreSQL support column-oriented tables? Citus or Timescale extension? - Timescale: https://news.ycombinator.com/item?id=21412596 https://news.ycombinator.com/item?id=21412596 - Citus: https://news.ycombinator.com/item?id=26369305 https://news.ycombinator.com/item?id=26369305 ----------- "Citus 10 brings columnar compression to Postgres" https://www.citusdata.com/blog/2021/03/06/citus-10-columnar-compression-for-postgres/ https://www.citusdata.com/blog/2021/03/06/citus-10-columnar-... "The following options are available: compression: [none|pglz|zstd|lz4|lz4hc] - set the compression type for newly-inserted data. Existing data will not be recompressed/decompressed. The default value is zstd (if support has been compiled in). compression_level: <integer> - Sets compression level. Valid settings are from 1 through 19. If the compression method does not support the level chosen, the closest level will be selected instead. stripe_row_limit: <integer> - the maximum number of rows per stripe for newly-inserted data. Existing stripes of data will not be changed and may have more rows than this maximum value. The default value is 150000. chunk_group_row_limit: <integer> - the maximum number of rows per chunk for newly-inserted data. Existing chunks of data will not be changed and may have more rows than this maximum value. The default value is 10000." read more: https://github.com/citusdata/citus/tree/master/src/backend/columnar https://github.com/citusdata/citus/tree/master/src/backend/c...
- derefr 5y agoThe columnar implementations are cool and all, but they don't really do what you'd expect columnar storage to do—i.e. make each column independently retrievable / make each column effectively its own "single data column plus shared CTID" table for the dataset. Instead, when you fetch one column of a row from e.g. a Citus "USING columnar" table, that still bakes down to the query engine fetching the entire row-tuple for that row (and then discarding most of it), which in turn means that the columnar storage engine must read in + decompress the equivalent compressed blocks for every column participating in that row. So, unlike in a columnar database, you still have to worry about splitting up your narrow+hot "fact" columns from your wide+cold "dimension" columns, and joining them back together later only when you need them, in order to not be polluting your page cache with the wide+cold column data.
- booleanbetrayal 5y agoIt would be nice if AWS built their PG14 offering for RDS with this supported. PG14 is available on their beta channel now, but I haven't had a chance to play. Has anyone had a chance to explore this yet?
- IronWolve 5y agoWith limited cores or cpu speed, I've been using lz4 with bzip, gzip, pv, and to copy files over ssh. Now pbzip2, I just use defaults with multiple cores for compression.
- unsigner 5y agoLZ4 is amazing - it's basically the same idea as LZ77, only implemented carefully, with all the dozens parameters and tradeoffs made in a certain way. Couple this with modern amounts of brute force thrown in an optimal parser (LZ4HC) and you get something quite extraordinary, on the "Pareto frontier" of compression efficiency. There's a tendency to focus on algorithms and scoff at implementation details, but LZ4 was possible as an algorithm 30 years ago, yet the implementation details crystallized to perfection only, say, five years ago.
- kevin_thibedeau 5y agoWe also have voluminous amounts of cache and multicore processors that can decompress in cache faster than reading uncompressed data. The tech did need to catch up.
- unsigner 5y agoTrue. If you're on your way down this rabbit hole, there's a bunch of old-machine-specific compression algorithms, developed by the emulator community, e.g. LZSA: https://github.com/emmanuel-marty/lzsa https://github.com/emmanuel-marty/lzsa
- cma 5y agoLZ4's choices may have also made less sense then due to memory latency, cache sizes, deep pipelining changing missed branch penalty, and other changes since the 70s.