5 ms·

in the dev PG15 LZ4 WAL compression is enabled https://github.com/postgres/postgres/commit/4035cd5d4eee4dae797bfc77ab07f8dcd8781b41 https://github.com/postgres

by pella 5y ago

in the dev PG15 LZ4 WAL compression is enabled https://github.com/postgres/postgres/commit/4035cd5d4eee4dae797bfc77ab07f8dcd8781b41 https://github.com/postgres/postgres/commit/4035cd5d4eee4dae... "Add support for LZ4 with compression of full-page writes in WAL The logic is implemented so as there can be a choice in the compression used when building a WAL record, and an extra per-record bit is used to track down if a block is compressed with PGLZ, LZ4 or nothing. wal_compression, the existing parameter, is changed to an enum with support for the following backward-compatible values: - "off", the default, to not use compression. - "pglz" or "on", to compress FPWs with PGLZ. - "lz4", the new mode, to compress FPWs with LZ4. Benchmarking has showed that LZ4 outclasses easily PGLZ. ZSTD would be also an interesting choice, but going just with LZ4 for now makes the patch minimalistic as toast compression is already able to use LZ4, so there is no need to worry about any build-related needs for this implementation. Author: Andrey Borodin, Justin Pryzby Reviewed-by: Dilip Kumar, Michael Paquier"

indeyets 5y ago
zstd would be great, indeed. it's much more performant and should be a great case for database
- ahartmetz 5y ago
  > zstd [...] much more performant Actually, that's not true. LZ4 is faster at compression and decompression, zstd (even at the minimum setting) is slower and has higher compression ratio.
  - zaarn 5y ago
    zstd-fast is almost as fast as LZ4 and gets a tiny bit better compression, though not a lot of software supports that mode. ZFS does, hence I have some numbers to back that up.
    - strainer 5y ago
      I just had a little look into these somewhat new zstd options --fast 1 to fast 5. fast 5 does seem almost equivalent to lz4 performance in compression ratio and compression 'speed' (or cpu load) except decompression 'speed' is still almost half of lz4, so at that extreme of zstds options it is squarely beaten by lz4. But in terms of compression ratio - which in most cases should be the more crucial characteristic for any compressor than cpu load - this setting seems to be well under zstd's knee of diminishing returns. --fast 1 has about 15% more cpu load for about 112% of compression performance than --fast5. And then zstds lightest non-fast setting (zstd-1) for 50% more load in compression and decompression, compresses 32% better than --fast5 (and lzo and lz4). The decompression load is an eye catching feature of lz4, but its compression ratio is not just marginally compromised, it is a much less powerful compressor in that prime characteristic than zstds weakest non-fast setting. I can appreciate why Btrfs is in no hurry to include lz4 while it supports zstd:1 to 15. Although it would be interesting to test if lz4 could have a niche on fast storage combined with slow cpus.
      - ahartmetz 5y ago
        There are use cases for lz4, I've had one in an embedded project. Slowish SoC (i.MX6 DualLite IIRC), relatively fast storage (eMMC - these tend to perform better than regular MMC). I measured boot time for the available kernel compression options and lz4 won by 100-200 ms or so compared to the next best option, which may have been gzip. It didn't make a big dent in overall boot time, but not bad for just setting an option. zstd, if available, which it wasn't, would have gotten a marginally worse result probably. IIRC, lzma was slower than uncompressed.
        strainer 5y ago
        gzip would seem to be a surprising runner up as its decompression cpu load is reportedly around 3-4 times as heavy as zstd:1 with a similar compression ratio. I would not rule zstd out, it seems to better every alternative given the right level setting, except lz4 which is the outlier but fixed at a relatively low compression level. As the high performing outlier it tantalisingly suggests room for improvements :]
  - RandyRanderson 5y ago
    Post that uses specious word: "performant" and a reply starting with "Actually, " - can we not create a prize for this? Maybe hn bingo cards? Dang, that would be good.
- adgjlsfhk1 5y ago
  Would it be a good idea to use zstd's dictionary compression for databases? It seems like it could be a big win since you are compressing relatively small blocks with known structure.
  - felixhandte 5y ago
    Yes. For example, RocksDB uses zstd dictionaries: https://rocksdb.org/blog/2021/05/31/dictionary-compression.html https://rocksdb.org/blog/2021/05/31/dictionary-compression.h...
  - infogulch 5y ago
    I've wondered the same. Maybe a per-table dictionary created as part of CREATE STATISTICS? I guess the difficulty will be updating the dictionary, you might need some kind of page epoch to look up what the dict was at the time the page was written.
gigatexal 5y ago
Ooh very nice! Non-statement level replication should get a speed up here then

by pella 5y ago

indeyets 5y ago

zstd would be great, indeed. it's much more performant and should be a great case for database

ahartmetz 5y ago
> zstd [...] much more performant Actually, that's not true. LZ4 is faster at compression and decompression, zstd (even at the minimum setting) is slower and has higher compression ratio.
- zaarn 5y ago
  zstd-fast is almost as fast as LZ4 and gets a tiny bit better compression, though not a lot of software supports that mode. ZFS does, hence I have some numbers to back that up.
  - strainer 5y ago
    I just had a little look into these somewhat new zstd options --fast 1 to fast 5. fast 5 does seem almost equivalent to lz4 performance in compression ratio and compression 'speed' (or cpu load) except decompression 'speed' is still almost half of lz4, so at that extreme of zstds options it is squarely beaten by lz4. But in terms of compression ratio - which in most cases should be the more crucial characteristic for any compressor than cpu load - this setting seems to be well under zstd's knee of diminishing returns. --fast 1 has about 15% more cpu load for about 112% of compression performance than --fast5. And then zstds lightest non-fast setting (zstd-1) for 50% more load in compression and decompression, compresses 32% better than --fast5 (and lzo and lz4). The decompression load is an eye catching feature of lz4, but its compression ratio is not just marginally compromised, it is a much less powerful compressor in that prime characteristic than zstds weakest non-fast setting. I can appreciate why Btrfs is in no hurry to include lz4 while it supports zstd:1 to 15. Although it would be interesting to test if lz4 could have a niche on fast storage combined with slow cpus.
    - ahartmetz 5y ago
      There are use cases for lz4, I've had one in an embedded project. Slowish SoC (i.MX6 DualLite IIRC), relatively fast storage (eMMC - these tend to perform better than regular MMC). I measured boot time for the available kernel compression options and lz4 won by 100-200 ms or so compared to the next best option, which may have been gzip. It didn't make a big dent in overall boot time, but not bad for just setting an option. zstd, if available, which it wasn't, would have gotten a marginally worse result probably. IIRC, lzma was slower than uncompressed.
      - strainer 5y ago
        gzip would seem to be a surprising runner up as its decompression cpu load is reportedly around 3-4 times as heavy as zstd:1 with a similar compression ratio. I would not rule zstd out, it seems to better every alternative given the right level setting, except lz4 which is the outlier but fixed at a relatively low compression level. As the high performing outlier it tantalisingly suggests room for improvements :]
- RandyRanderson 5y ago
  Post that uses specious word: "performant" and a reply starting with "Actually, " - can we not create a prize for this? Maybe hn bingo cards? Dang, that would be good.
adgjlsfhk1 5y ago
Would it be a good idea to use zstd's dictionary compression for databases? It seems like it could be a big win since you are compressing relatively small blocks with known structure.
- felixhandte 5y ago
  Yes. For example, RocksDB uses zstd dictionaries: https://rocksdb.org/blog/2021/05/31/dictionary-compression.html https://rocksdb.org/blog/2021/05/31/dictionary-compression.h...
- infogulch 5y ago
  I've wondered the same. Maybe a per-table dictionary created as part of CREATE STATISTICS? I guess the difficulty will be updating the dictionary, you might need some kind of page epoch to look up what the dict was at the time the page was written.

gigatexal 5y ago

Ooh very nice! Non-statement level replication should get a speed up here then