7 ms·

We (rsync.net) have several PB of raidz3 deployed all over the world. We use conservatively sized (12-15 drive) vdevs and typically join 3 or 4 of those togeth

by rsync 6y ago

We (rsync.net) have several PB of raidz3 deployed all over the world. We use conservatively sized (12-15 drive) vdevs and typically join 3 or 4 of those together to make a pool. I can see getting nervous about raidz2 (sort of analogous to "raid6") after a drive failure ... but losing 4 drives out of 12 in a single raidz3 failure cascade is extremely improbable. We all sleep quite well with this arrangement and have since we first migrated from UFS2 to ZFS in 2012.

tutfbhuf 6y ago
May I ask what your worst incident with ZFS was ever since 2012?
- rsync 6y ago
  Some background ... We have a fairly robust drive-proofing procedure prior to deployment - I think we still use 'badblocks' and really beat them up for 5-7 days. It's not enough to merely not fail - we have to see perfect returns on the SMART diagnostics and zero complaints from FreeBSD. So we're starting with a known-good population of drives. Then we monitor them very closely - again with both SMART and FreeBSD/ZFS system data and we are fairly aggressive about failing them out early if we see them start to misbehave. So we don't see things like straight up drive failures - the bad ones have already been rejected and drives that misbehave are culled out. The last time I got pulled in to make a judgement call (and got a little nervous) was (IIRC) we had a 15 drive vdev that had a drive failure and a candidate for early removal that had started to misbehave and the choice was made to yank them both because why do two big long resilvers ... and then during the subsequent resilver a third drive, while not failing, started to spit out errors. So the resilver completed and all was well but we had a potential third failure that could have died in the resilver and then we would be running with no protection. But that's why I love raidz3 - even if that had happened, we would have needed to lose yet another drive to lose the vdev (and the entire pool). I think the actionable recommendations here are to burn in your drives when you get them because we do, indeed, find rejects in most batches. Also, pick an error threshold that is low and be disciplined about sticking to it - don't let drives spin out SMART errors sporadically for months ... EDIT: Here is another thought and this goes back to pre-ZFS days and old-style RAID, etc. In normal operations we're aggressively failing out drives that misbehave BUT if you're in a marginal situation you need to flip that logic around - especially if the pool is in-use during resilver/rebuild. If you've lost 2 drives in a raidz3 (or, say, 1 drive in a RAID6) that remaining protection drive even if it is failing is like gold. If you fail it out, it's gone forever and has no relevance to the pool/array - but if you keep it, even as it's dying, you can either limp along OR you can even offline the pool/array and send that last drive to recovery or clone it or whatever ... the point is, when an array goes sideways every single bit of parity, no matter how poorly behaved, should be treated like gold.
  - seized 6y ago
    Agreed on the limping along. I survived a RAIDZ1 double drive failure exactly like that, one drive completely died and a second started acting up during the resilver. Resilver slowed way down but made it eventually. All those drives got replaced with a RAIDZ2 pool.
  - dnautics 6y ago
    thank you for this. can you speak to the retention rate post culling using your proofing? And do you have preferred manufacturers? Also, do you bother to do "the trick" where you zfs across heterogeneous SKUs to reduce the risk of use-case-coupled failures?
    - rsync 6y ago
      I don't have real numbers for this but I feel like post-burn-in retention is very high - like 99%. We do not have preferred manufacturers. Data suggests that Hitachis are better than Seagates but I reject this kind of Ford vs. Chevy preference in hard drives - the "good drives" can flip immediately - with new model or even new revision of existing model. I don't think it makes sense to try to pursue a particular manufacturer. FWIW, we have bought a ton of seagates over the last four years and they have been fine.
      - dnautics 6y ago
        Thanks for the good info!
aduitsis 6y ago
UFS2? So you are using FreeBSD?! Yet another reason to support you folks, seriously.
- rsync 6y ago
  All rsync.net infrastructure is, and has been, built solely on FreeBSD. Since 2001 ... :)
aidenn0 6y ago
I was running 3x 12 drive vdevs in raidz2 and write performance was terrible. We moved some data to a different machine and rebuilt as 18 mirrors. This was a long time ago (as in running on Solaris long time ago), so maybe things are better now.
- segfaultbuserr 6y ago
  Slow performance is always my impression with RAIDZ. There's an obvious performance hit if you are coming from conventional RAID setups (like mdadm). I've even seen some unbelievable slow speed like 1 MiB/s in the middle of copying a large git repo (which have a lot of small files, repeated git pull also creates a lot of fragmentation). But my experience was based on early ZFSOnLinux, perhaps a real BSD will perform better. Anyway, I knew ZFS's first goal is data safety, not performance, and it's why I could tolerate the performance...
- myrond 6y ago
  Each 12 drive array would have an interesting physical block layout. What was your ASHIFT and what was your physical disk raw sector size because if you had a ASHIFT=9 you would have 5120k blocks, if it was a ASHIFT=12 you would have physical 40960 blocks. None of those line up well with zfs. Additionally if this was a long time ago the pool would have defaulted to ASHIFT=9 and if you added any 4K drives you could have been having write amplification occurring. Looks as if you had your drives doing a lot of unnecessary extra wasted work.
  - aidenn0 6y ago
    Ashift=9 native 512b sector drives
  - fomine3 6y ago
    This is huge thing (called "Slop Space") that I noticed after I bought all ZFS NAS parts! It wastes much capacity by default if drive count is not ideal. IIRC use larger recordsize (1MB or above) and enable compression almost solves issue. I use default 128K recordsize for small files zvol for performance, and use larger recordsize for the rest of zvols. https://wintelguy.com/zfs-calc.pl https://wintelguy.com/zfs-calc.pl
aDfbrtVt 6y ago
I love the backup solution you guys provide with Borg, the pricing is amazing and the product has been rock solid. Any chance of getting similar "expert level" pricing for accounts using ZFS send | receive ?
- rsync 6y ago
  zfs send enabled accounts are (relatively) expensive because we need to give you a full-blown VPN with a routable ipv4, etc. If you can make do with an ipv6 address and if you have a reasonable quantity, email us ... perhaps we can work something out ...
  - aDfbrtVt 6y ago
    I’m relatively new to ZFS, but I thought you can send/receive over ssh (much like rsync). What’s the purpose of the VPN layer? My use is relatively small, just a home user with less than a TB worth of data. Definitely not worthwhile for your team to set up as a one off.
    - cahuk 6y ago
      It is always worth to write an email to info@rsync.net to work out an acceptable offer. I did it and got a relatively good offer. I have now 2 TB at rsync.net. rsync.net is not cheap but you have to know what you get: ZFS with great redundancy and a great support. All you have to do is using the service. You don't have to worry about maintenance.
deleted 6y ago
[deleted]
silenteh 6y ago
May I ask if you run a distributed filesystem on top of ZFS and if so which one ?
- rsync 6y ago
  No. We keep things as simple as possible.

by rsync 6y ago

tutfbhuf 6y ago

May I ask what your worst incident with ZFS was ever since 2012?

rsync 6y ago
Some background ... We have a fairly robust drive-proofing procedure prior to deployment - I think we still use 'badblocks' and really beat them up for 5-7 days. It's not enough to merely not fail - we have to see perfect returns on the SMART diagnostics and zero complaints from FreeBSD. So we're starting with a known-good population of drives. Then we monitor them very closely - again with both SMART and FreeBSD/ZFS system data and we are fairly aggressive about failing them out early if we see them start to misbehave. So we don't see things like straight up drive failures - the bad ones have already been rejected and drives that misbehave are culled out. The last time I got pulled in to make a judgement call (and got a little nervous) was (IIRC) we had a 15 drive vdev that had a drive failure and a candidate for early removal that had started to misbehave and the choice was made to yank them both because why do two big long resilvers ... and then during the subsequent resilver a third drive, while not failing, started to spit out errors. So the resilver completed and all was well but we had a potential third failure that could have died in the resilver and then we would be running with no protection. But that's why I love raidz3 - even if that had happened, we would have needed to lose yet another drive to lose the vdev (and the entire pool). I think the actionable recommendations here are to burn in your drives when you get them because we do, indeed, find rejects in most batches. Also, pick an error threshold that is low and be disciplined about sticking to it - don't let drives spin out SMART errors sporadically for months ... EDIT: Here is another thought and this goes back to pre-ZFS days and old-style RAID, etc. In normal operations we're aggressively failing out drives that misbehave BUT if you're in a marginal situation you need to flip that logic around - especially if the pool is in-use during resilver/rebuild. If you've lost 2 drives in a raidz3 (or, say, 1 drive in a RAID6) that remaining protection drive even if it is failing is like gold. If you fail it out, it's gone forever and has no relevance to the pool/array - but if you keep it, even as it's dying, you can either limp along OR you can even offline the pool/array and send that last drive to recovery or clone it or whatever ... the point is, when an array goes sideways every single bit of parity, no matter how poorly behaved, should be treated like gold.
- seized 6y ago
  Agreed on the limping along. I survived a RAIDZ1 double drive failure exactly like that, one drive completely died and a second started acting up during the resilver. Resilver slowed way down but made it eventually. All those drives got replaced with a RAIDZ2 pool.
- dnautics 6y ago
  thank you for this. can you speak to the retention rate post culling using your proofing? And do you have preferred manufacturers? Also, do you bother to do "the trick" where you zfs across heterogeneous SKUs to reduce the risk of use-case-coupled failures?
  - rsync 6y ago
    I don't have real numbers for this but I feel like post-burn-in retention is very high - like 99%. We do not have preferred manufacturers. Data suggests that Hitachis are better than Seagates but I reject this kind of Ford vs. Chevy preference in hard drives - the "good drives" can flip immediately - with new model or even new revision of existing model. I don't think it makes sense to try to pursue a particular manufacturer. FWIW, we have bought a ton of seagates over the last four years and they have been fine.
    - dnautics 6y ago
      Thanks for the good info!

aduitsis 6y ago

UFS2? So you are using FreeBSD?! Yet another reason to support you folks, seriously.

rsync 6y ago
All rsync.net infrastructure is, and has been, built solely on FreeBSD. Since 2001 ... :)

aidenn0 6y ago

I was running 3x 12 drive vdevs in raidz2 and write performance was terrible. We moved some data to a different machine and rebuilt as 18 mirrors. This was a long time ago (as in running on Solaris long time ago), so maybe things are better now.

segfaultbuserr 6y ago
Slow performance is always my impression with RAIDZ. There's an obvious performance hit if you are coming from conventional RAID setups (like mdadm). I've even seen some unbelievable slow speed like 1 MiB/s in the middle of copying a large git repo (which have a lot of small files, repeated git pull also creates a lot of fragmentation). But my experience was based on early ZFSOnLinux, perhaps a real BSD will perform better. Anyway, I knew ZFS's first goal is data safety, not performance, and it's why I could tolerate the performance...
myrond 6y ago
Each 12 drive array would have an interesting physical block layout. What was your ASHIFT and what was your physical disk raw sector size because if you had a ASHIFT=9 you would have 5120k blocks, if it was a ASHIFT=12 you would have physical 40960 blocks. None of those line up well with zfs. Additionally if this was a long time ago the pool would have defaulted to ASHIFT=9 and if you added any 4K drives you could have been having write amplification occurring. Looks as if you had your drives doing a lot of unnecessary extra wasted work.
- aidenn0 6y ago
  Ashift=9 native 512b sector drives
- fomine3 6y ago
  This is huge thing (called "Slop Space") that I noticed after I bought all ZFS NAS parts! It wastes much capacity by default if drive count is not ideal. IIRC use larger recordsize (1MB or above) and enable compression almost solves issue. I use default 128K recordsize for small files zvol for performance, and use larger recordsize for the rest of zvols. https://wintelguy.com/zfs-calc.pl https://wintelguy.com/zfs-calc.pl

aDfbrtVt 6y ago

I love the backup solution you guys provide with Borg, the pricing is amazing and the product has been rock solid. Any chance of getting similar "expert level" pricing for accounts using ZFS send | receive ?

rsync 6y ago
zfs send enabled accounts are (relatively) expensive because we need to give you a full-blown VPN with a routable ipv4, etc. If you can make do with an ipv6 address and if you have a reasonable quantity, email us ... perhaps we can work something out ...
- aDfbrtVt 6y ago
  I’m relatively new to ZFS, but I thought you can send/receive over ssh (much like rsync). What’s the purpose of the VPN layer? My use is relatively small, just a home user with less than a TB worth of data. Definitely not worthwhile for your team to set up as a one off.
  - cahuk 6y ago
    It is always worth to write an email to info@rsync.net to work out an acceptable offer. I did it and got a relatively good offer. I have now 2 TB at rsync.net. rsync.net is not cheap but you have to know what you get: ZFS with great redundancy and a great support. All you have to do is using the service. You don't have to worry about maintenance.

deleted 6y ago

[deleted]

silenteh 6y ago

May I ask if you run a distributed filesystem on top of ZFS and if so which one ?

rsync 6y ago
No. We keep things as simple as possible.