9 ms·

Demystifying OpenZFS 2.0

159 points by vermaden 5y ago

xoa 5y ago
Looks like a decent basic rundown of what was another nice jump forward (2020 fwiw, 2.0 wasn't just introduced it launched about 1 year ago, current version is 2.1.1). Worth noting a number of these have significant implications for tuning performance in common use cases, so some resources from even a few years ago are now a bit out of date. I know a lot of people for example have followed JRS' "ZFS tuning cheat sheet" [0], and a lot of that still applies. But given the increasing usage of enormously faster but pricier solid state drives, commonly a relative surfeit of processing power, and the performance of ZStandard, I think most people should now consider it over lz4 by default when using solid state drives unless using a very weak system or every last cycle is needed for other tasks (VMs perhaps). A bunch of detailed performance graphs comparing different compression algorithms were posted in the ZSTD feature introduction [1] which can help give a better idea of the right balance. HN had a long discussion on the kind of improvement it can make at scale just a few days ago [2]. Persistent L2ARC has also potentially changed the math there in some cases, though that one in turn may well be obsoleted entirely for SSD pools. And not new here I think but somewhat more recent, for those who do still have things that make use of atime ZFS now includes the Linux style 'relatime' support which does a nice job of splitting the difference between noatime and full atime. Also, since a reasonable number of people may use ZFS via TrueNAS (formerly FreeNAS), note the GUI there doesn't expose a ton of useful features itself but can still be told to do them since it handles interfacing with ZFS via a plain Python. Currently, GUI pool/fs creation stuff is handled via: /usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py 'options' handles pool flags and 'fsoptions' fs ones. Modifying that (reboot seems needed to make it take effect afterwards though perhaps reloading something else would do it) will then allow all the GUI setup to work and slice things the way TrueNAS wants while still offering finer grained property control. Particularly useful for things that can only be set at creation time, like pool properties such as ashift/feature flags or fs properties like utf8/normalization. ---- 0: https://jrs-s.net/2018/08/17/zfs-tuning-cheat-sheet/ https://jrs-s.net/2018/08/17/zfs-tuning-cheat-sheet/ 1: https://github.com/openzfs/zfs/pull/9735#issuecomment-570082078 https://github.com/openzfs/zfs/pull/9735#issuecomment-570082... 2: https://news.ycombinator.com/item?id=29164727 https://news.ycombinator.com/item?id=29164727
john37386 5y ago
What is the difference between Klara and IXSystems? Are they competitor, partners or something else?
- vermaden 5y ago
  Just two different independent companies. Like Netflix and Apple TV for example.
  - deleted 5y ago
    [deleted]
- magicalhippo 5y ago
  To add a bit more, iXsystems makes enterprise NAS systems, and develops TrueNAS (formerly FreeNAS and TrueOS) which has ZFS as one of it's main features. Klara systems is more of a development and support consultancy firm for FreeBSD and ZFS.
- willis936 5y ago
  Competitors, but they both participate in OpenZFS development. Yet another success story for open standards. https://www.ixsystems.com/blog/openzfs-2-on-truenas/ https://www.ixsystems.com/blog/openzfs-2-on-truenas/
  - thijsvandien 5y ago
    Competing how, exactly?
    - willis936 5y ago
      They both make NAS software and have similar monetization strategies. Business to one is lost to the other.
bhouston 5y ago
EDIT: There is a now an incorrect response to this post of mine stating that I forgot to mention the ZIL SLOG -- but the poster doesn't understand how the ZIL SLOG actually works unfortunately. :( I applaud OpenZFS for these major improvements, but its performance is not competitive with some of the commercial offerings, even if OpenZFS is a more easily administered solution. The main issue with OpenZFS performance is its write speed. While OpenZFS has excellent read caching via ARC and L2ARC, it doesn't enable NVMe write caching nor does it allow for automatic tiered storage pools (which can have NVMe paired with HDDs.) This means that large writes, those which exceed the available RAM, will drop down to the HDD write speed quite quickly instead of achieving wire speed. Which means I get write speeds of 300 MBps versus 10 GBps wirespeed. The solutions competitors have implemented are: - SSD read-write caching, usually using mirrored NVMe drives to ensure redundancy. - SSD tiering, which is including your SSDs in your storage pool. Frequently accessed data is moved onto the SSDs while infrequently accessed data is moved to the HDDs. Without these features, my QNAP (configured with SSD tiering) kicks the pants off of ZFS on writes that do not fit into memory. References: [1] QNAP Tiering: https://www.qnap.com/solution/qtier/en-us/ https://www.qnap.com/solution/qtier/en-us/ [2] QNAP SSD Read-Write Caching: https://www.qnap.com/solution/ssd-cache/en/ https://www.qnap.com/solution/ssd-cache/en/ [3] Synology SSD Read-Write Caching: https://kb.synology.com/en-my/DSM/help/DSM/StorageManager/genericssdcache?version=6 https://kb.synology.com/en-my/DSM/help/DSM/StorageManager/ge...
- walrus01 5y ago
  > Without these features, my QNAP (configured with SSD tiering) kicks the pants off of ZFS on writes that do not fit into memory. > Synology I am perfectly willing to sacrifice some performance in exchange for not being locked into a vendor-specific proprietary NAS/SAN hardware platform or closed source software solution that's proprietary to that one vendor.
  - danudey 5y ago
    While I like the idea of moving to a proper self-managed system, my Synology NAS is the least proprietary of all the proprietary systems I've used. That is to say that yes, it does have a bunch of proprietary stuff on top, but it also: 1. Runs Linux 2. Uses Linux systems, like BTRFS and LVM, to pool data 3. Provides SSH access 4. Has tons of packages that you might want to use (Squid, Node, Plex, etc.) available 5. Supports Docker, among other things, so you can run what you like. So yeah, they have a lot of proprietary offerings on top of a generic Linux storage solution, but if you change your mind you can take that generic Linux storage solution somewhere else without being locked in to anything Synology provides. The only caveat is if you choose to use their all-the-way proprietary systems, like their Google Docs equivalent or their cloud music/drive/etc. stuff, all of which is still stored on the same accessible drives. You might lose your "music library" or "photo library" or "video library", but your music and photo and video files will still be accessible.
todd8 5y ago
This is a great discussion on ZFS. I thoroughly enjoy reading HN! For those who aren't real IT people here is how I set up my first ZFS system at home. My background includes a few years at IBM as an OS architect in charge of, among other things, the file systems and network services for AIX. However, this was a long time ago (~1988). So I understood NFS, inodes, what it meant to mount a device and the classic Unix commands for managing filesystems. The first ZFS problem for me was just the vocabulary. What's a pool? What's a VDev? Fortunately, there are lots of high-level descriptions of ZFS on the internet that cover the high level concepts. Getting the hardware set up was the second hurdle. There are a lot of discussions on forums that cover hardware issues and there you will find answers to many hardware issues, like: is ECC memory required, how fast does the processor need to be, are SATA drives recommended and what drive controllers are compatible with ZFS. Naturally, the answers to these questions depend on how critical is the system you are putting together. I suggest for most people to try setting up a ZFS on some old hardware just to get practice with it. I had a low end Dell tower server model that I wasn't using so it became my first FreeBSD/ZFS server. It had four hard drives. The real difficulty for me was just the lack of a deep understanding of how ZFS works. ZFS has lots of features totally new to me so working with only the man pages for the ZFS related commands was not giving me much fun. I ended up buying two small books that I can't recommend enough, FreeBSD Mastery: ZFS and FreeBSD Mastery: Advanced ZFS by Michael W Lucas and Allan Jude, see [1] and [2]. These books are each around 200 pages long. The chapters of the first book are: 1. Introducing ZFS 2. Virtual Devices 3. Pools 4. ZFS Datasets 5. Repairs and Renovations 6. Disk Space Management 7. Snapshots and Clones 8. Installing ZFS Each of the chapters is quite detailed, and I found these two books were just what I needed. [1] https://read.amazon.com/kp/embed?asin=B00Y32OHNM&preview=newtab&linkCode=kpe&ref_=cm_sw_r_kb_dp_Z60Q58E3RVN4V0C2JA3P https://read.amazon.com/kp/embed?asin=B00Y32OHNM&preview=new... [2] https://read.amazon.com/kp/embed?asin=B01E40YIRM&preview=newtab&linkCode=kpe&ref_=cm_sw_p_kb_dp&reshareId=1GAQXKANM9Z3N2K8MQXE&reshareChannel=system https://read.amazon.com/kp/embed?asin=B01E40YIRM&preview=new...

159 points by vermaden 5y ago

xoa 5y ago

Looks like a decent basic rundown of what was another nice jump forward (2020 fwiw, 2.0 wasn't just introduced it launched about 1 year ago, current version is 2.1.1). Worth noting a number of these have significant implications for tuning performance in common use cases, so some resources from even a few years ago are now a bit out of date. I know a lot of people for example have followed JRS' "ZFS tuning cheat sheet" [0], and a lot of that still applies. But given the increasing usage of enormously faster but pricier solid state drives, commonly a relative surfeit of processing power, and the performance of ZStandard, I think most people should now consider it over lz4 by default when using solid state drives unless using a very weak system or every last cycle is needed for other tasks (VMs perhaps). A bunch of detailed performance graphs comparing different compression algorithms were posted in the ZSTD feature introduction [1] which can help give a better idea of the right balance. HN had a long discussion on the kind of improvement it can make at scale just a few days ago [2]. Persistent L2ARC has also potentially changed the math there in some cases, though that one in turn may well be obsoleted entirely for SSD pools. And not new here I think but somewhat more recent, for those who do still have things that make use of atime ZFS now includes the Linux style 'relatime' support which does a nice job of splitting the difference between noatime and full atime. Also, since a reasonable number of people may use ZFS via TrueNAS (formerly FreeNAS), note the GUI there doesn't expose a ton of useful features itself but can still be told to do them since it handles interfacing with ZFS via a plain Python. Currently, GUI pool/fs creation stuff is handled via: /usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py 'options' handles pool flags and 'fsoptions' fs ones. Modifying that (reboot seems needed to make it take effect afterwards though perhaps reloading something else would do it) will then allow all the GUI setup to work and slice things the way TrueNAS wants while still offering finer grained property control. Particularly useful for things that can only be set at creation time, like pool properties such as ashift/feature flags or fs properties like utf8/normalization. ---- 0: https://jrs-s.net/2018/08/17/zfs-tuning-cheat-sheet/ https://jrs-s.net/2018/08/17/zfs-tuning-cheat-sheet/ 1: https://github.com/openzfs/zfs/pull/9735#issuecomment-570082078 https://github.com/openzfs/zfs/pull/9735#issuecomment-570082... 2: https://news.ycombinator.com/item?id=29164727 https://news.ycombinator.com/item?id=29164727

john37386 5y ago

What is the difference between Klara and IXSystems? Are they competitor, partners or something else?

vermaden 5y ago
Just two different independent companies. Like Netflix and Apple TV for example.
- deleted 5y ago
  [deleted]
magicalhippo 5y ago
To add a bit more, iXsystems makes enterprise NAS systems, and develops TrueNAS (formerly FreeNAS and TrueOS) which has ZFS as one of it's main features. Klara systems is more of a development and support consultancy firm for FreeBSD and ZFS.
willis936 5y ago
Competitors, but they both participate in OpenZFS development. Yet another success story for open standards. https://www.ixsystems.com/blog/openzfs-2-on-truenas/ https://www.ixsystems.com/blog/openzfs-2-on-truenas/
- thijsvandien 5y ago
  Competing how, exactly?
  - willis936 5y ago
    They both make NAS software and have similar monetization strategies. Business to one is lost to the other.

bhouston 5y ago

EDIT: There is a now an incorrect response to this post of mine stating that I forgot to mention the ZIL SLOG -- but the poster doesn't understand how the ZIL SLOG actually works unfortunately. :( I applaud OpenZFS for these major improvements, but its performance is not competitive with some of the commercial offerings, even if OpenZFS is a more easily administered solution. The main issue with OpenZFS performance is its write speed. While OpenZFS has excellent read caching via ARC and L2ARC, it doesn't enable NVMe write caching nor does it allow for automatic tiered storage pools (which can have NVMe paired with HDDs.) This means that large writes, those which exceed the available RAM, will drop down to the HDD write speed quite quickly instead of achieving wire speed. Which means I get write speeds of 300 MBps versus 10 GBps wirespeed. The solutions competitors have implemented are: - SSD read-write caching, usually using mirrored NVMe drives to ensure redundancy. - SSD tiering, which is including your SSDs in your storage pool. Frequently accessed data is moved onto the SSDs while infrequently accessed data is moved to the HDDs. Without these features, my QNAP (configured with SSD tiering) kicks the pants off of ZFS on writes that do not fit into memory. References: [1] QNAP Tiering: https://www.qnap.com/solution/qtier/en-us/ https://www.qnap.com/solution/qtier/en-us/ [2] QNAP SSD Read-Write Caching: https://www.qnap.com/solution/ssd-cache/en/ https://www.qnap.com/solution/ssd-cache/en/ [3] Synology SSD Read-Write Caching: https://kb.synology.com/en-my/DSM/help/DSM/StorageManager/genericssdcache?version=6 https://kb.synology.com/en-my/DSM/help/DSM/StorageManager/ge...

walrus01 5y ago
> Without these features, my QNAP (configured with SSD tiering) kicks the pants off of ZFS on writes that do not fit into memory. > Synology I am perfectly willing to sacrifice some performance in exchange for not being locked into a vendor-specific proprietary NAS/SAN hardware platform or closed source software solution that's proprietary to that one vendor.
- danudey 5y ago
  While I like the idea of moving to a proper self-managed system, my Synology NAS is the least proprietary of all the proprietary systems I've used. That is to say that yes, it does have a bunch of proprietary stuff on top, but it also: 1. Runs Linux 2. Uses Linux systems, like BTRFS and LVM, to pool data 3. Provides SSH access 4. Has tons of packages that you might want to use (Squid, Node, Plex, etc.) available 5. Supports Docker, among other things, so you can run what you like. So yeah, they have a lot of proprietary offerings on top of a generic Linux storage solution, but if you change your mind you can take that generic Linux storage solution somewhere else without being locked in to anything Synology provides. The only caveat is if you choose to use their all-the-way proprietary systems, like their Google Docs equivalent or their cloud music/drive/etc. stuff, all of which is still stored on the same accessible drives. You might lose your "music library" or "photo library" or "video library", but your music and photo and video files will still be accessible.

todd8 5y ago

This is a great discussion on ZFS. I thoroughly enjoy reading HN! For those who aren't real IT people here is how I set up my first ZFS system at home. My background includes a few years at IBM as an OS architect in charge of, among other things, the file systems and network services for AIX. However, this was a long time ago (~1988). So I understood NFS, inodes, what it meant to mount a device and the classic Unix commands for managing filesystems. The first ZFS problem for me was just the vocabulary. What's a pool? What's a VDev? Fortunately, there are lots of high-level descriptions of ZFS on the internet that cover the high level concepts. Getting the hardware set up was the second hurdle. There are a lot of discussions on forums that cover hardware issues and there you will find answers to many hardware issues, like: is ECC memory required, how fast does the processor need to be, are SATA drives recommended and what drive controllers are compatible with ZFS. Naturally, the answers to these questions depend on how critical is the system you are putting together. I suggest for most people to try setting up a ZFS on some old hardware just to get practice with it. I had a low end Dell tower server model that I wasn't using so it became my first FreeBSD/ZFS server. It had four hard drives. The real difficulty for me was just the lack of a deep understanding of how ZFS works. ZFS has lots of features totally new to me so working with only the man pages for the ZFS related commands was not giving me much fun. I ended up buying two small books that I can't recommend enough, FreeBSD Mastery: ZFS and FreeBSD Mastery: Advanced ZFS by Michael W Lucas and Allan Jude, see [1] and [2]. These books are each around 200 pages long. The chapters of the first book are: 1. Introducing ZFS 2. Virtual Devices 3. Pools 4. ZFS Datasets 5. Repairs and Renovations 6. Disk Space Management 7. Snapshots and Clones 8. Installing ZFS Each of the chapters is quite detailed, and I found these two books were just what I needed. [1] https://read.amazon.com/kp/embed?asin=B00Y32OHNM&preview=newtab&linkCode=kpe&ref_=cm_sw_r_kb_dp_Z60Q58E3RVN4V0C2JA3P https://read.amazon.com/kp/embed?asin=B00Y32OHNM&preview=new... [2] https://read.amazon.com/kp/embed?asin=B01E40YIRM&preview=newtab&linkCode=kpe&ref_=cm_sw_p_kb_dp&reshareId=1GAQXKANM9Z3N2K8MQXE&reshareChannel=system https://read.amazon.com/kp/embed?asin=B01E40YIRM&preview=new...