7 ms·

EDIT: There is a now an incorrect response to this post of mine stating that I forgot to mention the ZIL SLOG -- but the poster doesn't understand how the ZIL S

EDIT: There is a now an incorrect response to this post of mine stating that I forgot to mention the ZIL SLOG -- but the poster doesn't understand how the ZIL SLOG actually works unfortunately. :( I applaud OpenZFS for these major improvements, but its performance is not competitive with some of the commercial offerings, even if OpenZFS is a more easily administered solution. The main issue with OpenZFS performance is its write speed. While OpenZFS has excellent read caching via ARC and L2ARC, it doesn't enable NVMe write caching nor does it allow for automatic tiered storage pools (which can have NVMe paired with HDDs.) This means that large writes, those which exceed the available RAM, will drop down to the HDD write speed quite quickly instead of achieving wire speed. Which means I get write speeds of 300 MBps versus 10 GBps wirespeed. The solutions competitors have implemented are: - SSD read-write caching, usually using mirrored NVMe drives to ensure redundancy. - SSD tiering, which is including your SSDs in your storage pool. Frequently accessed data is moved onto the SSDs while infrequently accessed data is moved to the HDDs. Without these features, my QNAP (configured with SSD tiering) kicks the pants off of ZFS on writes that do not fit into memory. References: [1] QNAP Tiering: https://www.qnap.com/solution/qtier/en-us/ https://www.qnap.com/solution/qtier/en-us/ [2] QNAP SSD Read-Write Caching: https://www.qnap.com/solution/ssd-cache/en/ https://www.qnap.com/solution/ssd-cache/en/ [3] Synology SSD Read-Write Caching: https://kb.synology.com/en-my/DSM/help/DSM/StorageManager/genericssdcache?version=6 https://kb.synology.com/en-my/DSM/help/DSM/StorageManager/ge...

walrus01 5y ago
> Without these features, my QNAP (configured with SSD tiering) kicks the pants off of ZFS on writes that do not fit into memory. > Synology I am perfectly willing to sacrifice some performance in exchange for not being locked into a vendor-specific proprietary NAS/SAN hardware platform or closed source software solution that's proprietary to that one vendor.
- danudey 5y ago
  While I like the idea of moving to a proper self-managed system, my Synology NAS is the least proprietary of all the proprietary systems I've used. That is to say that yes, it does have a bunch of proprietary stuff on top, but it also: 1. Runs Linux 2. Uses Linux systems, like BTRFS and LVM, to pool data 3. Provides SSH access 4. Has tons of packages that you might want to use (Squid, Node, Plex, etc.) available 5. Supports Docker, among other things, so you can run what you like. So yeah, they have a lot of proprietary offerings on top of a generic Linux storage solution, but if you change your mind you can take that generic Linux storage solution somewhere else without being locked in to anything Synology provides. The only caveat is if you choose to use their all-the-way proprietary systems, like their Google Docs equivalent or their cloud music/drive/etc. stuff, all of which is still stored on the same accessible drives. You might lose your "music library" or "photo library" or "video library", but your music and photo and video files will still be accessible.
Intermernet 5y ago
Is QNAP doing something outside the ZFS spec here? If not, it's almost certain that solutions like TrueNAS can replicate this behaviour. If they are, I'd be concerned about vendor lock-in and data recovery. I'm assuming that QNAP are using ZFS, which a cursory search supports.
- bhouston 5y ago
  > I'm assuming that QNAP are using ZFS, which a cursory search supports. QNAP has two NAS OSes right now. The newer one is QuTS and it support ZFS. The other older one is QTS which doesn't use ZFS. Only the older QTS OS supports SSD Tiering. > Is QNAP doing something outside the ZFS spec here? If not, it's almost certain that solutions like TrueNAS can replicate this behaviour. If they are, I'd be concerned about vendor lock-in and data recovery. I am using TrueNAS for my ZFS solution and it doesn't offer anything out of the box to address the issues I am bringing up.
  - lizknope 5y ago
    I don't use ZFS or QNAP but Linux has its own native dm-cache and bcache implementations for creating hybrid volumes out of SSD plus hard drives. I would suspect that QNAP is using one of them. https://en.wikipedia.org/wiki/Dm-cache https://en.wikipedia.org/wiki/Dm-cache https://en.wikipedia.org/wiki/Bcache https://en.wikipedia.org/wiki/Bcache
  - tw04 5y ago
    TrueNAS very much supports a write log: https://www.truenas.com/docs/references/slog/ https://www.truenas.com/docs/references/slog/
    - bhouston 5y ago
      It is commonly misunderstood that ZIL SLOGs are write-back caches, they are not. They do not speed up large writes that do not fit into memory. I have tested this myself. You are spreading incorrect information, please stop. You know enough to be dangerous, but not enough to actually be right.
tw04 5y ago
>The main issue with OpenZFS performance is its write speed. >While OpenZFS has excellent read caching via ARC and L2ARC, it doesn't enable NVMe write caching nor does it allow for automatic tiered storage pools (which can have NVMe paired with HDDs.) Huh? What are you talking about? ZFS has had a write cache from day 1: ZIL. ZFS Intent Log, with SLOG which is a dedicated device. Back in the day we'd use RAM based devices, now you can use optane (or any other fast device of your choosing including just a regular old SSD). https://openzfs.org/w/images/c/c8/10-ZIL_performance.pdf https://openzfs.org/w/images/c/c8/10-ZIL_performance.pdf https://www.servethehome.com/exploring-best-zfs-zil-slog-ssd-intel-optane-nand/ https://www.servethehome.com/exploring-best-zfs-zil-slog-ssd...
- magicalhippo 5y ago
  > Huh? What are you talking about? ZFS has had a write cache from day 1: ZIL. The ZIL is specifically not a cache. It can have cache-like behavior in certain cases, particularly when paired with fast SLOG devices, but that's incidental and not its purpose. Its purpose is to ensure integrity and consistency. Specifically, the writes to disk are served from RAM[1], the ZIL is only read from in case of recovery from an unclean shutdown. ZFS won't store more writes in the SLOG than what it can store in RAM, unlike a write-back cache device (which ZFS does not support yet). [1]: https://klarasystems.com/articles/what-makes-a-good-time-to-use-openzfs-slog-and-when-should-you-avoid-it/ https://klarasystems.com/articles/what-makes-a-good-time-to-...
  - bhouston 5y ago
    That you for being right! So many people misunderstand the ZIL SLOG, including the guy you are responding to. I think most posts on the internet about the ZIL SLOG do not explain it correctly and thus a large number of people misunderstand it.
    - magicalhippo 5y ago
      Yes it took me some time to fully grasp how the ZIL and SLOG worked and how it is not a cache. There was some work done[1] on proper write-back cache for ZFS by Nexenta several years ago, but it seems it either stalled or there was a decision to keep it in-house as it hasn't made its way back to OpenZFS. [1]: https://openzfs.org/wiki/Writeback_Cache https://openzfs.org/wiki/Writeback_Cache
gorkish 5y ago
ZLOG/ZIL devices journal writes; they have always been in zfs. SSD/NVMe with power loss protection are recommended. Yes, it requires dedicated devices, but so does L2ARC.
- bhouston 5y ago
  It is commonly misunderstood that ZIL SLOGs are write-back caches, they are not. They do not speed up large writes that do not fit into memory. I have tested this myself. It seems to be an incredibly popular misconception that has spread very widely.
  - reacharavindh 5y ago
    _that_do not fit into memory_ is an important caveat to your abandoning the speedup offered by SLOG for sync writes. Case A : No SLOG. You have sync writes and they hit memory, the transaction groups get flushed to disks and _only then_ the ack goes back completing the write IO operation. Case B : low latency SLOG. You have sync writes and they hit memory, immediately gets written to SLOG, immediately the ack goes back completing the write IO operation while batching in transaction group does the flush to disks in background(at slow disk speeds). Sure, if you are out of memory to commit those write IOs, you're going to hit the disks in hot path and get undesirable perf. Case B surely is always better than case A wouldn't you say?
    - bhouston 5y ago
      I agree with your statement above completely. You understand ZIL SLOG perfectly. I have an NVMe as my ZIL SLOG device but it really doesn’t help much for my workloads - which involve transfer of large files around. In my scenarios were I ran out of memory on large writes and them get slow downs I would love a real write cache on an NVMe or tiered storage. It would make a huge difference.
reacharavindh 5y ago
Isn't that what ZIL(Intent Log)/SLOG is for? I have a huge ZFS server (> 1 PB usable ZFS storage) that runs with 768 GiB of memory and 480 GB of Intel Optane as SLOG. All our writes are sync by virtue of using NFS from our compute cluster. I'm sure we see better write speeds, and not wait on spinning disks for write IOs.
- formerly_proven 5y ago
  Are you using a single ZFS server as the primary storage system for the cluster? If so, how do you handle availability? Do nodes have local scratch space or is it a do-everything-on-the-cluster-FS kind of setup? How is this setup treating you (administration, "oh no something is slow / hanging / broke, whatdoido", availability, performance, especially on those heavy days)? Is this some kind of COTS solution supported by one of the usual vendors or did you roll it yourself? Being ZFS I'm guessing it is far better at dealing with small files, and with directories containing many files than clustered FS which fall over pretty easily in these cases? e: Not doing market research here, I'm just seeing (sometimes, feeling) all the pain involved with cluster filesystems and wondering how your solution stacks up.
  - Datagenerator 5y ago
    >7PB on ZFS in production here. No vendor support, FreeBSD with send/receive pairs to another 7PB location monitoring with Grafana, delayed sync so we always have proper DR available.
    - reacharavindh 5y ago
      Similar setup here. Do you have any pointers on your monitoring setup? I’m working on settling it up right next week, and could use a nice reference.
  - reacharavindh 5y ago
    Yes. Single ZFS server attached to 3 JBODs(60 disks each). Well, single ZFS server per namespace. We have 3 similar configurations for logically separate storage needs. Yes, nodes have NVMe SSD local scratch space for fast compute needs. Large datasets reside in central storage. Smaller chunks are usually brought to local scratch to let compute jobs do their thing. Results go back to central storage again. I built it for our needs based on freely available docs and community support. The server runs FreeBSD + ZFS + NFS and nothing else. Snapshots from this primary storage gets shipped over to a secondary location that has a slow mirror(similar setup but with cheap SATA disks). Our need is only minor geographical data durability and not lower downtime DR. Experience: beats most commercial stuff we have run so far. Minimal setup of FreeBSD, ZFS, and NFS runs buttery smooth with almost no operational needs. I run upgrades on OS after it has been in community for a while, and it has only been smooth. We didn’t find a need to buy specialist support as our setup is minimal, and workload not any crazy, and availability targets not so strict. Performance wise this setup far exceeded our expectations in all our usecases. We do have 768 GiB of memory for this that ZFS happily makes use of. ARC is about 60-70% of mem most of the time. I will happily build the same again should there be a need for storage.
xoa 5y ago
>The main issue with OpenZFS performance is its write speed. At first I thought you were talking about actual raw read/write speed and how things like ARC or write caches can actually become bottlenecks when using NVMe storage, which can easily get to 200 Gbps with a small number of devices. That's being worked on via efforts like adding Direct IO. Instead though I think you've fallen into one of those niche holes that always takes longer to get filled because there isn't much demand. Big ZFS users simply have tons and tons of drives, and with large enough arrays to spread across even rust can do alright. They'll also have more custom direction of hot/cold to different pools entirely. Smaller scale users, depending on size vs budget, may just be using pure solid state at this point, or multiple pools with more manual management. Basic 2TB SATA SSDs are down to <$170 and 2TB NVMe drives are <$190. 10 Gbps isn't much to hit, and there are lots of ways to script things up for management. RAM for buffer is pretty cheap now too. Using a mirrored NVMe-based ZIL and bypassing cache might also get many people where they want to be on sync writes. I can see why it'd be a nice bit of polish to have hybrid pool management built-in, but I can also see how there wouldn't be a ton of demand to implement it given the various incentives in play. Might be worth seeing if there is any open work on such a thing or at least feature request. Also to your lower comment: >I am using TrueNAS for my ZFS solution and it doesn't offer anything out of the box to address the issues I am bringing up. While not directly related to precisely what you're asking for, I recently started using TrueNAS myself for the first time after over a decade of using ZFS full time and one thing that immediately surprised me is how unexpectedly limited the GUI is. TONS and tons of normal ZFS functionality is not exposed there for no reason I can understand. However, it's all still supported, it's still FreeBSD and normal ZFS underneath. You can still pull up a shell and manipulate things via the commandline, or (better for some stuff) modify the GUI framework helpers to customize things like pool creation options. The power is still there at least, though other issues like inexplicably shitty user support in the GUI show it's clearly aimed at home/SoHo, maybe with some SMB roles.
- bhouston 5y ago
  You are correct -- I am in a niche area. I have less than 12 HDDs in a RAIDZ2 configuration. If I had more HDDs in the RAID I would see better write speeds. But I think most home users with TrueNAS ZFS do not have massive arrays, thus my envy of the tiering or true SSD caching. You are correct that prices of small SSD are near HDD prices, but most people would want larger drives like 12 to 16TB drives in their array, and SSDs can not compete with these on price at all. Maybe we are in the twilight of HDDs? And SSDs will compete on price across the whole storage capacity range soon? Maybe...

walrus01 5y ago

> Without these features, my QNAP (configured with SSD tiering) kicks the pants off of ZFS on writes that do not fit into memory. > Synology I am perfectly willing to sacrifice some performance in exchange for not being locked into a vendor-specific proprietary NAS/SAN hardware platform or closed source software solution that's proprietary to that one vendor.

danudey 5y ago
While I like the idea of moving to a proper self-managed system, my Synology NAS is the least proprietary of all the proprietary systems I've used. That is to say that yes, it does have a bunch of proprietary stuff on top, but it also: 1. Runs Linux 2. Uses Linux systems, like BTRFS and LVM, to pool data 3. Provides SSH access 4. Has tons of packages that you might want to use (Squid, Node, Plex, etc.) available 5. Supports Docker, among other things, so you can run what you like. So yeah, they have a lot of proprietary offerings on top of a generic Linux storage solution, but if you change your mind you can take that generic Linux storage solution somewhere else without being locked in to anything Synology provides. The only caveat is if you choose to use their all-the-way proprietary systems, like their Google Docs equivalent or their cloud music/drive/etc. stuff, all of which is still stored on the same accessible drives. You might lose your "music library" or "photo library" or "video library", but your music and photo and video files will still be accessible.

Intermernet 5y ago

Is QNAP doing something outside the ZFS spec here? If not, it's almost certain that solutions like TrueNAS can replicate this behaviour. If they are, I'd be concerned about vendor lock-in and data recovery. I'm assuming that QNAP are using ZFS, which a cursory search supports.

bhouston 5y ago
> I'm assuming that QNAP are using ZFS, which a cursory search supports. QNAP has two NAS OSes right now. The newer one is QuTS and it support ZFS. The other older one is QTS which doesn't use ZFS. Only the older QTS OS supports SSD Tiering. > Is QNAP doing something outside the ZFS spec here? If not, it's almost certain that solutions like TrueNAS can replicate this behaviour. If they are, I'd be concerned about vendor lock-in and data recovery. I am using TrueNAS for my ZFS solution and it doesn't offer anything out of the box to address the issues I am bringing up.
- lizknope 5y ago
  I don't use ZFS or QNAP but Linux has its own native dm-cache and bcache implementations for creating hybrid volumes out of SSD plus hard drives. I would suspect that QNAP is using one of them. https://en.wikipedia.org/wiki/Dm-cache https://en.wikipedia.org/wiki/Dm-cache https://en.wikipedia.org/wiki/Bcache https://en.wikipedia.org/wiki/Bcache
- tw04 5y ago
  TrueNAS very much supports a write log: https://www.truenas.com/docs/references/slog/ https://www.truenas.com/docs/references/slog/
  - bhouston 5y ago
    It is commonly misunderstood that ZIL SLOGs are write-back caches, they are not. They do not speed up large writes that do not fit into memory. I have tested this myself. You are spreading incorrect information, please stop. You know enough to be dangerous, but not enough to actually be right.

tw04 5y ago

>The main issue with OpenZFS performance is its write speed. >While OpenZFS has excellent read caching via ARC and L2ARC, it doesn't enable NVMe write caching nor does it allow for automatic tiered storage pools (which can have NVMe paired with HDDs.) Huh? What are you talking about? ZFS has had a write cache from day 1: ZIL. ZFS Intent Log, with SLOG which is a dedicated device. Back in the day we'd use RAM based devices, now you can use optane (or any other fast device of your choosing including just a regular old SSD). https://openzfs.org/w/images/c/c8/10-ZIL_performance.pdf https://openzfs.org/w/images/c/c8/10-ZIL_performance.pdf https://www.servethehome.com/exploring-best-zfs-zil-slog-ssd-intel-optane-nand/ https://www.servethehome.com/exploring-best-zfs-zil-slog-ssd...

magicalhippo 5y ago
> Huh? What are you talking about? ZFS has had a write cache from day 1: ZIL. The ZIL is specifically not a cache. It can have cache-like behavior in certain cases, particularly when paired with fast SLOG devices, but that's incidental and not its purpose. Its purpose is to ensure integrity and consistency. Specifically, the writes to disk are served from RAM[1], the ZIL is only read from in case of recovery from an unclean shutdown. ZFS won't store more writes in the SLOG than what it can store in RAM, unlike a write-back cache device (which ZFS does not support yet). [1]: https://klarasystems.com/articles/what-makes-a-good-time-to-use-openzfs-slog-and-when-should-you-avoid-it/ https://klarasystems.com/articles/what-makes-a-good-time-to-...
- bhouston 5y ago
  That you for being right! So many people misunderstand the ZIL SLOG, including the guy you are responding to. I think most posts on the internet about the ZIL SLOG do not explain it correctly and thus a large number of people misunderstand it.
  - magicalhippo 5y ago
    Yes it took me some time to fully grasp how the ZIL and SLOG worked and how it is not a cache. There was some work done[1] on proper write-back cache for ZFS by Nexenta several years ago, but it seems it either stalled or there was a decision to keep it in-house as it hasn't made its way back to OpenZFS. [1]: https://openzfs.org/wiki/Writeback_Cache https://openzfs.org/wiki/Writeback_Cache

gorkish 5y ago

ZLOG/ZIL devices journal writes; they have always been in zfs. SSD/NVMe with power loss protection are recommended. Yes, it requires dedicated devices, but so does L2ARC.

bhouston 5y ago
It is commonly misunderstood that ZIL SLOGs are write-back caches, they are not. They do not speed up large writes that do not fit into memory. I have tested this myself. It seems to be an incredibly popular misconception that has spread very widely.
- reacharavindh 5y ago
  _that_do not fit into memory_ is an important caveat to your abandoning the speedup offered by SLOG for sync writes. Case A : No SLOG. You have sync writes and they hit memory, the transaction groups get flushed to disks and _only then_ the ack goes back completing the write IO operation. Case B : low latency SLOG. You have sync writes and they hit memory, immediately gets written to SLOG, immediately the ack goes back completing the write IO operation while batching in transaction group does the flush to disks in background(at slow disk speeds). Sure, if you are out of memory to commit those write IOs, you're going to hit the disks in hot path and get undesirable perf. Case B surely is always better than case A wouldn't you say?
  - bhouston 5y ago
    I agree with your statement above completely. You understand ZIL SLOG perfectly. I have an NVMe as my ZIL SLOG device but it really doesn’t help much for my workloads - which involve transfer of large files around. In my scenarios were I ran out of memory on large writes and them get slow downs I would love a real write cache on an NVMe or tiered storage. It would make a huge difference.

reacharavindh 5y ago

Isn't that what ZIL(Intent Log)/SLOG is for? I have a huge ZFS server (> 1 PB usable ZFS storage) that runs with 768 GiB of memory and 480 GB of Intel Optane as SLOG. All our writes are sync by virtue of using NFS from our compute cluster. I'm sure we see better write speeds, and not wait on spinning disks for write IOs.

formerly_proven 5y ago
Are you using a single ZFS server as the primary storage system for the cluster? If so, how do you handle availability? Do nodes have local scratch space or is it a do-everything-on-the-cluster-FS kind of setup? How is this setup treating you (administration, "oh no something is slow / hanging / broke, whatdoido", availability, performance, especially on those heavy days)? Is this some kind of COTS solution supported by one of the usual vendors or did you roll it yourself? Being ZFS I'm guessing it is far better at dealing with small files, and with directories containing many files than clustered FS which fall over pretty easily in these cases? e: Not doing market research here, I'm just seeing (sometimes, feeling) all the pain involved with cluster filesystems and wondering how your solution stacks up.
- Datagenerator 5y ago
  >7PB on ZFS in production here. No vendor support, FreeBSD with send/receive pairs to another 7PB location monitoring with Grafana, delayed sync so we always have proper DR available.
  - reacharavindh 5y ago
    Similar setup here. Do you have any pointers on your monitoring setup? I’m working on settling it up right next week, and could use a nice reference.
- reacharavindh 5y ago
  Yes. Single ZFS server attached to 3 JBODs(60 disks each). Well, single ZFS server per namespace. We have 3 similar configurations for logically separate storage needs. Yes, nodes have NVMe SSD local scratch space for fast compute needs. Large datasets reside in central storage. Smaller chunks are usually brought to local scratch to let compute jobs do their thing. Results go back to central storage again. I built it for our needs based on freely available docs and community support. The server runs FreeBSD + ZFS + NFS and nothing else. Snapshots from this primary storage gets shipped over to a secondary location that has a slow mirror(similar setup but with cheap SATA disks). Our need is only minor geographical data durability and not lower downtime DR. Experience: beats most commercial stuff we have run so far. Minimal setup of FreeBSD, ZFS, and NFS runs buttery smooth with almost no operational needs. I run upgrades on OS after it has been in community for a while, and it has only been smooth. We didn’t find a need to buy specialist support as our setup is minimal, and workload not any crazy, and availability targets not so strict. Performance wise this setup far exceeded our expectations in all our usecases. We do have 768 GiB of memory for this that ZFS happily makes use of. ARC is about 60-70% of mem most of the time. I will happily build the same again should there be a need for storage.

xoa 5y ago

>The main issue with OpenZFS performance is its write speed. At first I thought you were talking about actual raw read/write speed and how things like ARC or write caches can actually become bottlenecks when using NVMe storage, which can easily get to 200 Gbps with a small number of devices. That's being worked on via efforts like adding Direct IO. Instead though I think you've fallen into one of those niche holes that always takes longer to get filled because there isn't much demand. Big ZFS users simply have tons and tons of drives, and with large enough arrays to spread across even rust can do alright. They'll also have more custom direction of hot/cold to different pools entirely. Smaller scale users, depending on size vs budget, may just be using pure solid state at this point, or multiple pools with more manual management. Basic 2TB SATA SSDs are down to <$170 and 2TB NVMe drives are <$190. 10 Gbps isn't much to hit, and there are lots of ways to script things up for management. RAM for buffer is pretty cheap now too. Using a mirrored NVMe-based ZIL and bypassing cache might also get many people where they want to be on sync writes. I can see why it'd be a nice bit of polish to have hybrid pool management built-in, but I can also see how there wouldn't be a ton of demand to implement it given the various incentives in play. Might be worth seeing if there is any open work on such a thing or at least feature request. Also to your lower comment: >I am using TrueNAS for my ZFS solution and it doesn't offer anything out of the box to address the issues I am bringing up. While not directly related to precisely what you're asking for, I recently started using TrueNAS myself for the first time after over a decade of using ZFS full time and one thing that immediately surprised me is how unexpectedly limited the GUI is. TONS and tons of normal ZFS functionality is not exposed there for no reason I can understand. However, it's all still supported, it's still FreeBSD and normal ZFS underneath. You can still pull up a shell and manipulate things via the commandline, or (better for some stuff) modify the GUI framework helpers to customize things like pool creation options. The power is still there at least, though other issues like inexplicably shitty user support in the GUI show it's clearly aimed at home/SoHo, maybe with some SMB roles.

bhouston 5y ago
You are correct -- I am in a niche area. I have less than 12 HDDs in a RAIDZ2 configuration. If I had more HDDs in the RAID I would see better write speeds. But I think most home users with TrueNAS ZFS do not have massive arrays, thus my envy of the tiering or true SSD caching. You are correct that prices of small SSD are near HDD prices, but most people would want larger drives like 12 to 16TB drives in their array, and SSDs can not compete with these on price at all. Maybe we are in the twilight of HDDs? And SSDs will compete on price across the whole storage capacity range soon? Maybe...