3 ms·

Isn't that what ZIL(Intent Log)/SLOG is for? I have a huge ZFS server (> 1 PB usable ZFS storage) that runs with 768 GiB of memory and 480 GB of Intel Optane as

Isn't that what ZIL(Intent Log)/SLOG is for? I have a huge ZFS server (> 1 PB usable ZFS storage) that runs with 768 GiB of memory and 480 GB of Intel Optane as SLOG. All our writes are sync by virtue of using NFS from our compute cluster. I'm sure we see better write speeds, and not wait on spinning disks for write IOs.

formerly_proven 5y ago
Are you using a single ZFS server as the primary storage system for the cluster? If so, how do you handle availability? Do nodes have local scratch space or is it a do-everything-on-the-cluster-FS kind of setup? How is this setup treating you (administration, "oh no something is slow / hanging / broke, whatdoido", availability, performance, especially on those heavy days)? Is this some kind of COTS solution supported by one of the usual vendors or did you roll it yourself? Being ZFS I'm guessing it is far better at dealing with small files, and with directories containing many files than clustered FS which fall over pretty easily in these cases? e: Not doing market research here, I'm just seeing (sometimes, feeling) all the pain involved with cluster filesystems and wondering how your solution stacks up.
- Datagenerator 5y ago
  >7PB on ZFS in production here. No vendor support, FreeBSD with send/receive pairs to another 7PB location monitoring with Grafana, delayed sync so we always have proper DR available.
  - reacharavindh 5y ago
    Similar setup here. Do you have any pointers on your monitoring setup? I’m working on settling it up right next week, and could use a nice reference.
- reacharavindh 5y ago
  Yes. Single ZFS server attached to 3 JBODs(60 disks each). Well, single ZFS server per namespace. We have 3 similar configurations for logically separate storage needs. Yes, nodes have NVMe SSD local scratch space for fast compute needs. Large datasets reside in central storage. Smaller chunks are usually brought to local scratch to let compute jobs do their thing. Results go back to central storage again. I built it for our needs based on freely available docs and community support. The server runs FreeBSD + ZFS + NFS and nothing else. Snapshots from this primary storage gets shipped over to a secondary location that has a slow mirror(similar setup but with cheap SATA disks). Our need is only minor geographical data durability and not lower downtime DR. Experience: beats most commercial stuff we have run so far. Minimal setup of FreeBSD, ZFS, and NFS runs buttery smooth with almost no operational needs. I run upgrades on OS after it has been in community for a while, and it has only been smooth. We didn’t find a need to buy specialist support as our setup is minimal, and workload not any crazy, and availability targets not so strict. Performance wise this setup far exceeded our expectations in all our usecases. We do have 768 GiB of memory for this that ZFS happily makes use of. ARC is about 60-70% of mem most of the time. I will happily build the same again should there be a need for storage.
  - yjftsjthsd-h 5y ago
    But how are you handling availability? If something happens to the hardware (ex. motherboard) or even just for installing OS updates (ex. FreeBSD security patches), do you lose storage for the whole cluster?
    - reacharavindh 5y ago
      Yes. Our availability target is not high. Our priorities while designing this system were 1. Data durability - need to make sure scientific data is never lost, 2. cost - our budget is small, and we build what we can afford, 3. performance - extract most value out of the hardware we use, 4. Availability - maximise availability by virtue of having fewer things that can fail. However, fancy failover systems and distributed filesystems may be out of reach(at least for now). So, we make do with what we have. We have three similar storage configs(and three mirrors as slow backups). We carry spare memory sticks, Optane drives, disks, and power supply for minimising down time due to hardware failures. If something were to go wrong for which we don't have spares, yes, the storage will be offline until we can fix it. Planned updates? - we have two scheduled downtimes in a year. All patches to these machines happen at that time. These machines are strictly internal and don't have internet access except for those planned maintenance days.
magicalhippo 5y ago
> Isn't that what ZIL(Intent Log)/SLOG is for? No, the ZIL is just for integrity[1]. [1]: https://news.ycombinator.com/item?id=29266628 https://news.ycombinator.com/item?id=29266628
- tw04 5y ago
  That's not true. When a write is committed to the SLOG, it acknowledges back to the host immediately. If you've got disk contention on the back-end, the host will absolutely see faster write response until the point your memory is 3/8th full. If you do not have an SLOG, the system will not ack to the host until it has committed the TXG to disk on a sync write, meaning any disk contention will present itself as massive latency. With 768GB of memory and a 480GB SLOG in the system that is a LOT of outstanding transactions that could be buffered. Sure if your workload is primarily long, sustained transactions the SLOG will do little. But for any workload that is even a little bursty in nature (like hosting virtual machines) it will absolutely make a difference. All of the above is assuming you have enabled sync=always, which I hope people are doing for production workloads unless they happen to have an app that can tolerate lost writes/partial writes.
  - magicalhippo 5y ago
    The ZIL is not a write-back cache. The ZIL is for maintaining integrity. The SLOG is for speeding up the ZIL. As such it can behave a bit like a write-back cache as I mentioned in the linked post. A crucial difference is that the ZIL is limited by the available memory regardless (and tunables, by default 5 seconds worth of data), which would not be the case for a write-back cache device.
    - tw04 5y ago
      I don't know why you keep responding by saying it's not a write-back cache. 1. I didn't ever call it a write-back cache and never even hinted at it operating like one 2. Write-back caching is only one of many options for write caches Regardless of both of those points, the SLOG ABSOLUTELY will speed up writes if you have a bursty workload (like the virtual machines he's running) and sync=always enabled. >A crucial difference is that the ZIL is limited by the available memory regardless (and tunables, by default 5 seconds worth of data), which would not be the case for a write-back cache device. And yet that point is irrelevant to OPs question of whether or not SLOG was speeding up his writes, which it almost assuredly is doing with the amount of memory and the size of his SLOG device when he is doing sync writes.
myrandomcomment 5y ago
https://jrs-s.net/2019/05/02/zfs-sync-async-zil-slog/ https://jrs-s.net/2019/05/02/zfs-sync-async-zil-slog/ ZIL and SLOG are NOT a write cache. They are 100% about not loosing data. A ZIL is part of a zpool where data is written to when the file protocol requires sync. When it is written the sync is cleared. In this case you are sharing the zpool with all its others needs to write and read, so more load on the pool. The data is also in RAM given to the write aggregator process. The RAM is only written latter. If the system crashes before the aggregator writes the data, on restart the data written to the ZIL is read into the aggregator and then is written out to the zpool. By default the ZIL is never read. An SLOG is just a ZIL that is on a separate device from the zpool. So a sync write comes in and is written to the SLOG and not part of the spinning rust zpool so no shared resources with the pool. The sync is marked as cleared when the data is written to the SLOG. The SLOG should be really fast as the goal is to ack the sync quickly. Remember at this point the data is NOT written to the zpool, it’s RAM and the aggregator will write the data out later. Same thing applies on crash. Reboot, SLOG is reviewed for uncleared items, sent to RAM, written to pool. TLDR - zil/slog are write devices where the data is written when sync is required and almost never read back. Data is alway in RAM cache and written out to the spool from RAM. ZIL/SLOG is not a buffer. Speed of ack sync will be as fast as data can be written to ZIL/SLOG.

formerly_proven 5y ago

Are you using a single ZFS server as the primary storage system for the cluster? If so, how do you handle availability? Do nodes have local scratch space or is it a do-everything-on-the-cluster-FS kind of setup? How is this setup treating you (administration, "oh no something is slow / hanging / broke, whatdoido", availability, performance, especially on those heavy days)? Is this some kind of COTS solution supported by one of the usual vendors or did you roll it yourself? Being ZFS I'm guessing it is far better at dealing with small files, and with directories containing many files than clustered FS which fall over pretty easily in these cases? e: Not doing market research here, I'm just seeing (sometimes, feeling) all the pain involved with cluster filesystems and wondering how your solution stacks up.

Datagenerator 5y ago
>7PB on ZFS in production here. No vendor support, FreeBSD with send/receive pairs to another 7PB location monitoring with Grafana, delayed sync so we always have proper DR available.
- reacharavindh 5y ago
  Similar setup here. Do you have any pointers on your monitoring setup? I’m working on settling it up right next week, and could use a nice reference.
reacharavindh 5y ago
Yes. Single ZFS server attached to 3 JBODs(60 disks each). Well, single ZFS server per namespace. We have 3 similar configurations for logically separate storage needs. Yes, nodes have NVMe SSD local scratch space for fast compute needs. Large datasets reside in central storage. Smaller chunks are usually brought to local scratch to let compute jobs do their thing. Results go back to central storage again. I built it for our needs based on freely available docs and community support. The server runs FreeBSD + ZFS + NFS and nothing else. Snapshots from this primary storage gets shipped over to a secondary location that has a slow mirror(similar setup but with cheap SATA disks). Our need is only minor geographical data durability and not lower downtime DR. Experience: beats most commercial stuff we have run so far. Minimal setup of FreeBSD, ZFS, and NFS runs buttery smooth with almost no operational needs. I run upgrades on OS after it has been in community for a while, and it has only been smooth. We didn’t find a need to buy specialist support as our setup is minimal, and workload not any crazy, and availability targets not so strict. Performance wise this setup far exceeded our expectations in all our usecases. We do have 768 GiB of memory for this that ZFS happily makes use of. ARC is about 60-70% of mem most of the time. I will happily build the same again should there be a need for storage.
- yjftsjthsd-h 5y ago
  But how are you handling availability? If something happens to the hardware (ex. motherboard) or even just for installing OS updates (ex. FreeBSD security patches), do you lose storage for the whole cluster?
  - reacharavindh 5y ago
    Yes. Our availability target is not high. Our priorities while designing this system were 1. Data durability - need to make sure scientific data is never lost, 2. cost - our budget is small, and we build what we can afford, 3. performance - extract most value out of the hardware we use, 4. Availability - maximise availability by virtue of having fewer things that can fail. However, fancy failover systems and distributed filesystems may be out of reach(at least for now). So, we make do with what we have. We have three similar storage configs(and three mirrors as slow backups). We carry spare memory sticks, Optane drives, disks, and power supply for minimising down time due to hardware failures. If something were to go wrong for which we don't have spares, yes, the storage will be offline until we can fix it. Planned updates? - we have two scheduled downtimes in a year. All patches to these machines happen at that time. These machines are strictly internal and don't have internet access except for those planned maintenance days.

magicalhippo 5y ago

> Isn't that what ZIL(Intent Log)/SLOG is for? No, the ZIL is just for integrity[1]. [1]: https://news.ycombinator.com/item?id=29266628 https://news.ycombinator.com/item?id=29266628

tw04 5y ago
That's not true. When a write is committed to the SLOG, it acknowledges back to the host immediately. If you've got disk contention on the back-end, the host will absolutely see faster write response until the point your memory is 3/8th full. If you do not have an SLOG, the system will not ack to the host until it has committed the TXG to disk on a sync write, meaning any disk contention will present itself as massive latency. With 768GB of memory and a 480GB SLOG in the system that is a LOT of outstanding transactions that could be buffered. Sure if your workload is primarily long, sustained transactions the SLOG will do little. But for any workload that is even a little bursty in nature (like hosting virtual machines) it will absolutely make a difference. All of the above is assuming you have enabled sync=always, which I hope people are doing for production workloads unless they happen to have an app that can tolerate lost writes/partial writes.
- magicalhippo 5y ago
  The ZIL is not a write-back cache. The ZIL is for maintaining integrity. The SLOG is for speeding up the ZIL. As such it can behave a bit like a write-back cache as I mentioned in the linked post. A crucial difference is that the ZIL is limited by the available memory regardless (and tunables, by default 5 seconds worth of data), which would not be the case for a write-back cache device.
  - tw04 5y ago
    I don't know why you keep responding by saying it's not a write-back cache. 1. I didn't ever call it a write-back cache and never even hinted at it operating like one 2. Write-back caching is only one of many options for write caches Regardless of both of those points, the SLOG ABSOLUTELY will speed up writes if you have a bursty workload (like the virtual machines he's running) and sync=always enabled. >A crucial difference is that the ZIL is limited by the available memory regardless (and tunables, by default 5 seconds worth of data), which would not be the case for a write-back cache device. And yet that point is irrelevant to OPs question of whether or not SLOG was speeding up his writes, which it almost assuredly is doing with the amount of memory and the size of his SLOG device when he is doing sync writes.

myrandomcomment 5y ago

https://jrs-s.net/2019/05/02/zfs-sync-async-zil-slog/ https://jrs-s.net/2019/05/02/zfs-sync-async-zil-slog/ ZIL and SLOG are NOT a write cache. They are 100% about not loosing data. A ZIL is part of a zpool where data is written to when the file protocol requires sync. When it is written the sync is cleared. In this case you are sharing the zpool with all its others needs to write and read, so more load on the pool. The data is also in RAM given to the write aggregator process. The RAM is only written latter. If the system crashes before the aggregator writes the data, on restart the data written to the ZIL is read into the aggregator and then is written out to the zpool. By default the ZIL is never read. An SLOG is just a ZIL that is on a separate device from the zpool. So a sync write comes in and is written to the SLOG and not part of the spinning rust zpool so no shared resources with the pool. The sync is marked as cleared when the data is written to the SLOG. The SLOG should be really fast as the goal is to ack the sync quickly. Remember at this point the data is NOT written to the zpool, it’s RAM and the aggregator will write the data out later. Same thing applies on crash. Reboot, SLOG is reviewed for uncleared items, sent to RAM, written to pool. TLDR - zil/slog are write devices where the data is written when sync is required and almost never read back. Data is alway in RAM cache and written out to the spool from RAM. ZIL/SLOG is not a buffer. Speed of ack sync will be as fast as data can be written to ZIL/SLOG.