8 ms·

So disable compression in Postgres and do all compression in ZFS?

mattashii 5y ago
Postgres' own compression helps with memory usage in buffers, saving IOps, so I'd keep my compression in PostgreSQL enabled
- throwdbaaway 5y ago
  Compressed ARC has been a thing in ZFS for a long time. So let's say you allow ZFS cache to take 50% of total memory. With 4x compress ratio from ZSTD, you will be able to cache 2x the total memory. It is amazing.
  - deleted 5y ago
    [deleted]
  - nix23 5y ago
    You should not use ARC for pg data, since postgres has it's own cache, otherwise you just have two caches with the same data in it.
    - throwdbaaway 5y ago
      Unfortunately that's just how postgres works? I thought the rule of thumb is still to allocate 25% of memory for postgres 's shared buffer, and 50% of memory for OS buffer cache (i.e. ARC when using ZFS)?
      - ComputerGuru 5y ago
        That’s how postgres works, but ZFS lets you work around that by disabling arc caching on a dataset by dataset basis.
      - nix23 5y ago
        >Unfortunately No, postgres know better what to cache then arc.
        radiowave 5y ago
        For a transaction processing workload (where hitting indexes is the all-important thing), that's exactly right. But if you have more of an analytic workload involving sequential scans, Postgres definitely expects the underlying storage system to be handling at least some of the caching.
  - mattashii 5y ago
    Yes, but on the other hand, my toast relation now needs (e.g.) 4x the amount of blocks it would have had when compression was enabled. So we have a net 0 storage savings, except that we now need to request 4x as many pages from the OS to get the full toast data, those pages taking up 4x as much space in Postgres' shared buffers, also evicting 4x as many pages from the buffers as it would have otherwise. Unless your read operations to the file system are somehow very cheap, I don't think disabling toast compression is a good idea.
    - throwdbaaway 5y ago
      I see your point. Perhaps the best way is to use TOAST compression for wide field values, and then rely on ZFS filesystem compression for others? For some context, in MySQL, both the table-level compression and page-level compression are problematic, while ZFS has served me well, thus I have a lot more faith in it.
      - ComputerGuru 5y ago
        > a Perhaps the best way is to use TOAST compression for wide field values You don’t have to do anything. Toast compression is enabled for compatible types by default but never kicks in until a cell hits 2K bytes in size (by default).
    - magicalhippo 5y ago
      LZ4 is quite fast to bail on data it can't compress, so why not just have LZ4 enabled on both?
nix23 5y ago
I would recommend that (ATM?): For high-speed pgsql (on ZFS) enable zfs-compression (lz4) just on the filesystem, and disable zfs-caching (the arc, just leave metadata on) because postgres has it's own cache. >ZFS-Dataset: atime=off compression=lz4 recordsize=8k logbias=throughput xattr=sa redundant_metadata=most <===That's a maybe primarycache=metadata >In postgres: full_page_writes=off <===That's a maybe >Plus maybe some pgtune: https://pgtune.leopard.in.ua/#/ https://pgtune.leopard.in.ua/#/
- RedShift1 5y ago
  What about VDEV organization in a virtual machines? I have no physical machines left, everything is running on VMware. Just create one big VDEV?
  - nix23 5y ago
    Yes in a virtual machine just one vdev/dataset (if it's about speed, if redundancy...well that depends on you host-filesystem, then maybe two virtual disks?), maybe some tuning with the write-through etc on the Host to that virtual-disk? And raw would be a good thing too.
- ahachete 5y ago
  >Plus maybe some pgtune: Alternative tuning guide: https://postgresqlco.nf/tuning-guide https://postgresqlco.nf/tuning-guide (Disclosure: part of the team behind it)
- paulmd 5y ago
  Have not used this in anger but - from what I've read best practice is also to use separate datasets for the WAL and the rest of the data (including committed data) - this helps ZFS understand that WAL and committed are both separate datasets that both need separate QOS handling within ARC rather than just letting the WAL eat the whole cache. It'll try its best to balance them regardless, of course. You can still snapshot them atomically by making them both children of a parent dataset and performing a recursive snapshot on the parent. so you have dataset: myservice/pgdata myservice/pgwal or myservice/pgdata myservice/pgdata/pgwal or whatever. So question, given that this article is about LZ4 memory compression for postgres, would you want it enabled for both/either/none of those datasets? Obviously compression-on-compression doesn't generally help at all, but lz4 performance obviously doesn't hurt much at all either, so if there's anything that postgres doesn't compress, maybe it would be faster in a few niche situations.

Postgres' own compression helps with memory usage in buffers, saving IOps, so I'd keep my compression in PostgreSQL enabled

throwdbaaway 5y ago
Compressed ARC has been a thing in ZFS for a long time. So let's say you allow ZFS cache to take 50% of total memory. With 4x compress ratio from ZSTD, you will be able to cache 2x the total memory. It is amazing.
- deleted 5y ago
  [deleted]
- nix23 5y ago
  You should not use ARC for pg data, since postgres has it's own cache, otherwise you just have two caches with the same data in it.
  - throwdbaaway 5y ago
    Unfortunately that's just how postgres works? I thought the rule of thumb is still to allocate 25% of memory for postgres 's shared buffer, and 50% of memory for OS buffer cache (i.e. ARC when using ZFS)?
    - ComputerGuru 5y ago
      That’s how postgres works, but ZFS lets you work around that by disabling arc caching on a dataset by dataset basis.
    - nix23 5y ago
      >Unfortunately No, postgres know better what to cache then arc.
      - radiowave 5y ago
        For a transaction processing workload (where hitting indexes is the all-important thing), that's exactly right. But if you have more of an analytic workload involving sequential scans, Postgres definitely expects the underlying storage system to be handling at least some of the caching.
- mattashii 5y ago
  Yes, but on the other hand, my toast relation now needs (e.g.) 4x the amount of blocks it would have had when compression was enabled. So we have a net 0 storage savings, except that we now need to request 4x as many pages from the OS to get the full toast data, those pages taking up 4x as much space in Postgres' shared buffers, also evicting 4x as many pages from the buffers as it would have otherwise. Unless your read operations to the file system are somehow very cheap, I don't think disabling toast compression is a good idea.
  - throwdbaaway 5y ago
    I see your point. Perhaps the best way is to use TOAST compression for wide field values, and then rely on ZFS filesystem compression for others? For some context, in MySQL, both the table-level compression and page-level compression are problematic, while ZFS has served me well, thus I have a lot more faith in it.
    - ComputerGuru 5y ago
      > a Perhaps the best way is to use TOAST compression for wide field values You don’t have to do anything. Toast compression is enabled for compatible types by default but never kicks in until a cell hits 2K bytes in size (by default).
  - magicalhippo 5y ago
    LZ4 is quite fast to bail on data it can't compress, so why not just have LZ4 enabled on both?

nix23 5y ago

I would recommend that (ATM?): For high-speed pgsql (on ZFS) enable zfs-compression (lz4) just on the filesystem, and disable zfs-caching (the arc, just leave metadata on) because postgres has it's own cache. >ZFS-Dataset: atime=off compression=lz4 recordsize=8k logbias=throughput xattr=sa redundant_metadata=most <===That's a maybe primarycache=metadata >In postgres: full_page_writes=off <===That's a maybe >Plus maybe some pgtune: https://pgtune.leopard.in.ua/#/ https://pgtune.leopard.in.ua/#/

RedShift1 5y ago
What about VDEV organization in a virtual machines? I have no physical machines left, everything is running on VMware. Just create one big VDEV?
- nix23 5y ago
  Yes in a virtual machine just one vdev/dataset (if it's about speed, if redundancy...well that depends on you host-filesystem, then maybe two virtual disks?), maybe some tuning with the write-through etc on the Host to that virtual-disk? And raw would be a good thing too.
ahachete 5y ago
>Plus maybe some pgtune: Alternative tuning guide: https://postgresqlco.nf/tuning-guide https://postgresqlco.nf/tuning-guide (Disclosure: part of the team behind it)
paulmd 5y ago
Have not used this in anger but - from what I've read best practice is also to use separate datasets for the WAL and the rest of the data (including committed data) - this helps ZFS understand that WAL and committed are both separate datasets that both need separate QOS handling within ARC rather than just letting the WAL eat the whole cache. It'll try its best to balance them regardless, of course. You can still snapshot them atomically by making them both children of a parent dataset and performing a recursive snapshot on the parent. so you have dataset: myservice/pgdata myservice/pgwal or myservice/pgdata myservice/pgdata/pgwal or whatever. So question, given that this article is about LZ4 memory compression for postgres, would you want it enabled for both/either/none of those datasets? Obviously compression-on-compression doesn't generally help at all, but lz4 performance obviously doesn't hurt much at all either, so if there's anything that postgres doesn't compress, maybe it would be faster in a few niche situations.