6 ms·

On the other hand there are a couple of things in ZFS that can make a big difference. For instance: changing the page size to be the same size as Postgresql use

On the other hand there are a couple of things in ZFS that can make a big difference. For instance: changing the page size to be the same size as Postgresql uses (8kb); disabling the writing of 'last accessed' timestamps; or whether or not the filesystem attempts deduplication. From what I remember (it's been a while) these can be configured on a per-dataset basis, where a dataset is a lightweight filesystem more akin to a directory, and many of them can be changed on a live filesystem. Here, have at it: https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops... So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference. It's a shame a bug crept in to an otherwise pretty much faultless filesystem. I'm sure there are some truths in there about commercial support or not running the bleeding edge or whatever. And making backups, even for ZFS backed filesystems. Sacrilege, I know.

3np 3y ago
> whether or not the filesystem attempts deduplication Unless I'm just parroting truisms here, dedupe is almost never a good idea and compression is a much more common case-by-case. > So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference. 100%. So much depends on the particular workload and resources. > And making backups, even for ZFS backed filesystems. Sacrilege, I know. Logical backups, specifically. We have seen more than once corruption and data-loss in replicated encrypted snapshots due to bugs in ZFS.
- colechristensen 3y ago
  Deduplication requires an absolutely enormous amount of memory, unless you have a dataset that has a huge amount of duplication for some reason (maybe a company or school where a lot of people have their own network storage and put the same large files in?) it is usually never the right choice to enable dedup. It is cheaper to buy more disk than the memory to enable dedup (somebody could run the numbers). ZFS ARC also likes lots of memory and you get good performance gains with more memory allocated to it. In other words, turn on LZ4 compression everywhere and don’t think about it otherwise.
  - 3np 3y ago
    This is my understanding as well. These days we default to zstd, though.
  - erhaetherth 3y ago
    Oh good, I have it off. I was pondering that feature years ago when I set it up. Couldn't remember where I landed.
  - keep_reading 3y ago
    Deduplication in ZFS can be pushed onto dedicated flash storage now instead of requiring gobs of RAM.
    - ungamedplayer 3y ago
      Great channel for side attacks btw.
      - nolist_policy 3y ago
        Doesn't even need SSD's to open a side-channel: https://lore.kernel.org/linux-btrfs/CAKDzk=-HZardsLFH5c9HYre73NYNszUJqpfsh0YJnnaQToB3BA@mail.gmail.com/ https://lore.kernel.org/linux-btrfs/CAKDzk=-HZardsLFH5c9HYre...
      - keep_reading 3y ago
        Without having physical access to my hardware, where every disk is FDE anyway?
        ungamedplayer 3y ago
        Physical access not required. Dedupe side attack required timing measurement.
      - ungamedplayer 3y ago
        Oh got some hate there.. pissed off the zfs kids with a dose of reality. Dedupe can be used to infer secret data. .. more news at 11
- eternityforest 3y ago
  Data loss bugs in ZFS?? Wow. My personal approach to filesystem evaluation is pretty much "How long has it been since I've seen someone talk about a data loss bug". This is my first time hearing one for ZFS. BTRFS seems to have at least one post a year. Ext4 has almost none, but maybe that's because people trust it and blame hardware when unsure what the issue is.
  - 3np 3y ago
    Comparing zfs/btrfs with ext4 is not entirely fair. They're in different domains when it comes to scope and features. To some extent, silent corruption in ext4 is expected but in zfs it's a bug.
chungy 3y ago
> changing the page size to be the same size as Postgresql uses (8kb) Mind, you shouldn't really change the record size without knowing your use-case. Make a PostgreSQL file system, set it's recordsize to 8k. That's a good idea. Not so much a good idea doing that to all your filesystems. Stick to the defaults without a good justification to deviate.
BSDobelix 3y ago
For Postgres just follow the original doc: https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html#postgresql https://openzfs.github.io/openzfs-docs/Performance%20and%20T... compression=lz4 recordsize=32K In postgres -> full_page_writes=off
- auspiv 3y ago
  But also experiment with recordsizes of 8k and 16k. The larger the recordsize, the more potential compression you get, which may be offset by additional delay caused by read/modify/write of the larger record (i.e. reading/writing 32k instead of 8k or 16k).
  - BSDobelix 3y ago
    True, with platter storage i tend more to 32k with nvme more toward 16k. But yes testing is a good thing, but some settings should be just untouched if one is not absolutely sure it's a good thing, aka logbias=throughput.
sgarland 3y ago
Also, compression. CPUs are more than capable of handling it on the fly, and the reduced dataset size helps IO. This [0] is a brief post on MySQL, but most of it applies equally to Postgres. [0]: https://www.percona.com/blog/mysql-zfs-performance-update/ https://www.percona.com/blog/mysql-zfs-performance-update/
- doublepg23 3y ago
  I'd be curious if the addition of zstd compression has any effect on the results. Could be a fun follow-up follow-up :-)
GrumpySloth 3y ago
> disabling the writing of 'last accessed' timestamps atime is the most useless filesystem feature I’ve heard of. But on most Unix-like systems it’s enabled by default and the main thing it does is turn reads into writes, degrading performance. All the while ctime isn’t a standard Unix filesystem feature. And ctime to me always seemed infinitely more useful (and cheaper!) than atime. </rant>
- yencabulator 3y ago
  relatime has been the Linux default since 2009. It makes that be one write per 24 hours.
- mischief6 3y ago
  while it is unfortunate it requires a write, it is useful in some cases. for example, finding old objects in a cache, or today i used it to find out what firmware my kernel is loading by checking atime of /lib/firmware.
  - mike_hock 3y ago
    So it's useful as a debugging feature that you might want to briefly turn on once every decade. Finding old objects in a cache is an issue specific to caches and should be implemented there.
  - cromka 3y ago
    For the latter you could enable it, reboot and get the same information right away, then disable it?
    - mischief6 3y ago
      i did not see a clear list of all firmware files read by the kernel in dmesg, so the find/atime trick worked well.

3np 3y ago

> whether or not the filesystem attempts deduplication Unless I'm just parroting truisms here, dedupe is almost never a good idea and compression is a much more common case-by-case. > So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference. 100%. So much depends on the particular workload and resources. > And making backups, even for ZFS backed filesystems. Sacrilege, I know. Logical backups, specifically. We have seen more than once corruption and data-loss in replicated encrypted snapshots due to bugs in ZFS.

colechristensen 3y ago
Deduplication requires an absolutely enormous amount of memory, unless you have a dataset that has a huge amount of duplication for some reason (maybe a company or school where a lot of people have their own network storage and put the same large files in?) it is usually never the right choice to enable dedup. It is cheaper to buy more disk than the memory to enable dedup (somebody could run the numbers). ZFS ARC also likes lots of memory and you get good performance gains with more memory allocated to it. In other words, turn on LZ4 compression everywhere and don’t think about it otherwise.
- 3np 3y ago
  This is my understanding as well. These days we default to zstd, though.
- erhaetherth 3y ago
  Oh good, I have it off. I was pondering that feature years ago when I set it up. Couldn't remember where I landed.
- keep_reading 3y ago
  Deduplication in ZFS can be pushed onto dedicated flash storage now instead of requiring gobs of RAM.
  - ungamedplayer 3y ago
    Great channel for side attacks btw.
    - nolist_policy 3y ago
      Doesn't even need SSD's to open a side-channel: https://lore.kernel.org/linux-btrfs/CAKDzk=-HZardsLFH5c9HYre73NYNszUJqpfsh0YJnnaQToB3BA@mail.gmail.com/ https://lore.kernel.org/linux-btrfs/CAKDzk=-HZardsLFH5c9HYre...
    - keep_reading 3y ago
      Without having physical access to my hardware, where every disk is FDE anyway?
      - ungamedplayer 3y ago
        Physical access not required. Dedupe side attack required timing measurement.
    - ungamedplayer 3y ago
      Oh got some hate there.. pissed off the zfs kids with a dose of reality. Dedupe can be used to infer secret data. .. more news at 11
eternityforest 3y ago
Data loss bugs in ZFS?? Wow. My personal approach to filesystem evaluation is pretty much "How long has it been since I've seen someone talk about a data loss bug". This is my first time hearing one for ZFS. BTRFS seems to have at least one post a year. Ext4 has almost none, but maybe that's because people trust it and blame hardware when unsure what the issue is.
- 3np 3y ago
  Comparing zfs/btrfs with ext4 is not entirely fair. They're in different domains when it comes to scope and features. To some extent, silent corruption in ext4 is expected but in zfs it's a bug.

chungy 3y ago

> changing the page size to be the same size as Postgresql uses (8kb) Mind, you shouldn't really change the record size without knowing your use-case. Make a PostgreSQL file system, set it's recordsize to 8k. That's a good idea. Not so much a good idea doing that to all your filesystems. Stick to the defaults without a good justification to deviate.

BSDobelix 3y ago

For Postgres just follow the original doc: https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html#postgresql https://openzfs.github.io/openzfs-docs/Performance%20and%20T... compression=lz4 recordsize=32K In postgres -> full_page_writes=off

auspiv 3y ago
But also experiment with recordsizes of 8k and 16k. The larger the recordsize, the more potential compression you get, which may be offset by additional delay caused by read/modify/write of the larger record (i.e. reading/writing 32k instead of 8k or 16k).
- BSDobelix 3y ago
  True, with platter storage i tend more to 32k with nvme more toward 16k. But yes testing is a good thing, but some settings should be just untouched if one is not absolutely sure it's a good thing, aka logbias=throughput.

sgarland 3y ago

Also, compression. CPUs are more than capable of handling it on the fly, and the reduced dataset size helps IO. This [0] is a brief post on MySQL, but most of it applies equally to Postgres. [0]: https://www.percona.com/blog/mysql-zfs-performance-update/ https://www.percona.com/blog/mysql-zfs-performance-update/

doublepg23 3y ago
I'd be curious if the addition of zstd compression has any effect on the results. Could be a fun follow-up follow-up :-)

GrumpySloth 3y ago

> disabling the writing of 'last accessed' timestamps atime is the most useless filesystem feature I’ve heard of. But on most Unix-like systems it’s enabled by default and the main thing it does is turn reads into writes, degrading performance. All the while ctime isn’t a standard Unix filesystem feature. And ctime to me always seemed infinitely more useful (and cheaper!) than atime. </rant>

yencabulator 3y ago
relatime has been the Linux default since 2009. It makes that be one write per 24 hours.
mischief6 3y ago
while it is unfortunate it requires a write, it is useful in some cases. for example, finding old objects in a cache, or today i used it to find out what firmware my kernel is loading by checking atime of /lib/firmware.
- mike_hock 3y ago
  So it's useful as a debugging feature that you might want to briefly turn on once every decade. Finding old objects in a cache is an issue specific to caches and should be implemented there.
- cromka 3y ago
  For the latter you could enable it, reboot and get the same information right away, then disable it?
  - mischief6 3y ago
    i did not see a clear list of all firmware files read by the kernel in dmesg, so the find/atime trick worked well.