4 ms·

ZFS Resilver on SMR Drives

6 points by vermaden 2y ago

ggm 2y ago
Nice write up. My home ZFS is off pending two replacement drives because the hot spare failed during resilver. Oy vey.
- vermaden 2y ago
  Thank You. ... and yes - this is the problem with 'dedicated' SPARE drives - they just sit there unused and You never know if they will work when the time comes. ZFS resolved that with DRAID where there are not 'dedicated' SPARE disks but 'spare block' spread across the disks: - https://github.com/openzfs/zfs/wiki/dRAID-HOWTO/001b44728e1ac3329a4d91be97bcd565a8f351b7 https://github.com/openzfs/zfs/wiki/dRAID-HOWTO/001b44728e1a...
elmerfud 2y ago
SMR on a z-mirror is quite a bit different than a raidz/2/3 setup due to how SMR works. A mirror will do much better at optimizing the writes for SMR rebuild. When using raidz, your milage may vary, as they say. It absolutely will work. Before I knew the difference between SMR & CMR, and when WD obscured this on their reds, I bought SMR to use in a raid2 setup. I first noticed something was wrong when sequential write performance would begin to tank sharply on the drives and it wasn't due to the ZiL. Replacement of a failed drive was slow, very slow but I would work. After 5 years in service I replaced them with CMR drives one at a time. The replacement speed from the first to the last increase about 3x. So don't agree with the doom sayers about SMR but for the few $ more over the course of 5 years, I'll go with the speed.
- vermaden 2y ago
  Yes, it will 'hurt' more to use SMR drives with raidz1/raidz2/raidz3. ... but I believe ZFS DRAID mode will not hurt that much on SMR drives. In ZFS DRAID there is no dedicated SPARE drives - but 'spare blocks' are spread across the disks along with 'data blocks' and 'parity blocks' - this means that all drives 'read' for the resilver and all drives 'write' for the resilver. More here: - https://github.com/openzfs/zfs/wiki/dRAID-HOWTO/001b44728e1ac3329a4d91be97bcd565a8f351b7 https://github.com/openzfs/zfs/wiki/dRAID-HOWTO/001b44728e1a... ... also the Sequential Resilver helps even more: - https://docs.google.com/presentation/d/1vLsgQ1MaHlifw40C9R2sPsSiHiQpxglxMbK2SMthu0Q/edit#slide=id.g995720a6cf_1_19 https://docs.google.com/presentation/d/1vLsgQ1MaHlifw40C9R2s...
bell-cot 2y ago
Worth noting: During a zfs resilver (as with any sort of RAID rebuild) your drives are continuously busy. So the vastly-longer CoW/SMR rebuild can be rather problematic on production systems. Best to know your workload, and make sure that you have an ironclad paper trail - showing how PHB ignored your repeated objections and warnings when he decided to save money by buying SMR disks.
- vermaden 2y ago
  To be honest I would not expect using SMR drives in some important production systems. I would even prefer to use CMR ones ... but the 2.5 drives with 4-5 TB capacity are only available in SMR - maybe in the future when 2.5 drives will ofeer 8-10 TB of space - there would be a 4-5 TB CMD 2.5 drive.

6 points by vermaden 2y ago

ggm 2y ago

Nice write up. My home ZFS is off pending two replacement drives because the hot spare failed during resilver. Oy vey.

vermaden 2y ago
Thank You. ... and yes - this is the problem with 'dedicated' SPARE drives - they just sit there unused and You never know if they will work when the time comes. ZFS resolved that with DRAID where there are not 'dedicated' SPARE disks but 'spare block' spread across the disks: - https://github.com/openzfs/zfs/wiki/dRAID-HOWTO/001b44728e1ac3329a4d91be97bcd565a8f351b7 https://github.com/openzfs/zfs/wiki/dRAID-HOWTO/001b44728e1a...

elmerfud 2y ago

SMR on a z-mirror is quite a bit different than a raidz/2/3 setup due to how SMR works. A mirror will do much better at optimizing the writes for SMR rebuild. When using raidz, your milage may vary, as they say. It absolutely will work. Before I knew the difference between SMR & CMR, and when WD obscured this on their reds, I bought SMR to use in a raid2 setup. I first noticed something was wrong when sequential write performance would begin to tank sharply on the drives and it wasn't due to the ZiL. Replacement of a failed drive was slow, very slow but I would work. After 5 years in service I replaced them with CMR drives one at a time. The replacement speed from the first to the last increase about 3x. So don't agree with the doom sayers about SMR but for the few $ more over the course of 5 years, I'll go with the speed.

vermaden 2y ago
Yes, it will 'hurt' more to use SMR drives with raidz1/raidz2/raidz3. ... but I believe ZFS DRAID mode will not hurt that much on SMR drives. In ZFS DRAID there is no dedicated SPARE drives - but 'spare blocks' are spread across the disks along with 'data blocks' and 'parity blocks' - this means that all drives 'read' for the resilver and all drives 'write' for the resilver. More here: - https://github.com/openzfs/zfs/wiki/dRAID-HOWTO/001b44728e1ac3329a4d91be97bcd565a8f351b7 https://github.com/openzfs/zfs/wiki/dRAID-HOWTO/001b44728e1a... ... also the Sequential Resilver helps even more: - https://docs.google.com/presentation/d/1vLsgQ1MaHlifw40C9R2sPsSiHiQpxglxMbK2SMthu0Q/edit#slide=id.g995720a6cf_1_19 https://docs.google.com/presentation/d/1vLsgQ1MaHlifw40C9R2s...

bell-cot 2y ago

Worth noting: During a zfs resilver (as with any sort of RAID rebuild) your drives are continuously busy. So the vastly-longer CoW/SMR rebuild can be rather problematic on production systems. Best to know your workload, and make sure that you have an ironclad paper trail - showing how PHB ignored your repeated objections and warnings when he decided to save money by buying SMR disks.

vermaden 2y ago
To be honest I would not expect using SMR drives in some important production systems. I would even prefer to use CMR ones ... but the 2.5 drives with 4-5 TB capacity are only available in SMR - maybe in the future when 2.5 drives will ofeer 8-10 TB of space - there would be a 4-5 TB CMD 2.5 drive.