12 ms·

ZFS: Use mirror vdevs, not RAIDZ

115 points by segfaultbuserr 6y ago

naniwaduni 6y ago
"Use RAID10" is basically the storage equivalent of "use paper ballots". Sure, it feels suboptimal, but critically your intuition on how it can fail (mostly) works. That's a really nice property to have for storage.
chokeartist 6y ago
Parity pools are for poors. Mirror pools are for chads.
nix23 6y ago
For big installations i do striped (sometimes mirrors) vdevs (2 disks) and on top of them Raidz2/3. Works fast as hell and reliable.
npteljes 6y ago
I had my pitchforks out because I really like my raidz1 setup, but the arguments are very good for the mirror. When I set my pool up, I chose raidz1 because compared to mirroring, I still have 37.5% more space, which is huge if the budget is constrained. But if you feel paranoid and the raidz3 begin to sound good, that 12.5% extra space doesn't seem to worth the extra complexity over a much straightforward mirror.
gbrown_ 6y ago
> don’t be greedy. 50% storage efficiency is plenty. LOL.
- deleted 6y ago
  [deleted]
minimaul 6y ago
As always with RAID-style setups, there’s an inevitable trade off of cost vs capacity vs performance. There’s still a place for RAIDZ/RAIDZ2, and in my opinion that place is storing bulk data that isn’t too heavily accessed or that needs to be stored with an eye towards keeping £/GB down. Yes, mirrors are faster. Yes, mirrors are easier to expand. But across 12 4TB disks that is 24TB instead of 40TB with RAIDZ2 - and that’s a lot of capacity to lose if you’re on a budget. The rebuild times in this post seem high to me, though. I replaced 7x 2TB disks (nearly full) in a raidz1 in a backup pool with larger drives in about 30 hours.
- minimaul 6y ago
  The one thing in the article I agree with without reservation though is that you should always have a backup of your pool! ZFS makes doing good backups easy, with zfs send | zfs receive.
  - gh02t 6y ago
    I mean, is this controversial? The problem for a budget conscious home user is: where do you back up to? Doing a complete backup to a remote storage provider isn't a practical option for many users, which leaves the affordable options being to only back up critical data and/or use something like a big USB external drive. Neither of these options are great, so in reality many people end up relying on RAID or mirroring to provide some measure of redundancy regardless of whether or not it's a good idea.
    - benlivengood 6y ago
      I just started playing around with s3blkdevd. S3-compatible storage at backblaze B2 is $0.005/GB-month and I'll see if I can manage a vdev made out of a huge nbd disk. Topical to the article, backblaze has piles of RAID6 and appears to be happy with it. Another thought has been to find a ZFS-friend in another part of the world and exchange snapshots. Inline encryption makes that a little more viable now. Right now I'm sending incremental snapshots to AWS Glacier and have another local box mirroring the data (mostly to fully verify the snapshots before sending them to Glacier).
    - unethical_ban 6y ago
      I have a raidz2 that I keep everything on: (A) Personal, (B) core media, and (C) all other media. I backup A and B on a workstation drive, and I further backup A to S3 Glacier for $1/month/100GB, syncing every few months. It isn't all or nothing, 3-copies vs. striped pool. Mirrors are fine if you have quick access to fix dead drives, but I like the idea of being able to lose any two drives vs. only one drive in a given mirror. And while I currently have a 4 drive RAIDz2 (same disk utilization as a mirror) I wish I had a 4+2 array, which I'll probably build next year.
geerlingguy 6y ago
Every time I make this argument (whether regular RAID or ZFS), everyone pulls out their pitchforks and tells me about how they have run RAID 5 or 6 for years and never had a problem, plus "I can have two drives fail! You would lose your array if the wrong two drives fail!" But drive failure is much more likely during an operation like a resilver/resync, and I wonder if the majority of people espousing more risky RAID a setups (especially single parity) are those who just want hundreds of TB for a growing media collection. I know for my critical data, I don't trust parity. Plus I can't afford the days or weeks long resilver operation, I need a working storage array that doesn't suffer drastically in performance if one drive goes bad.
- Cyph0n 6y ago
  Critical data for the average user typically does not require much space, so the cost of storage overhead in a mirrored ZFS pool is not too bad. RAIDZ is much more cost efficient when you need to reliably store large content that isn’t really critical. As you’ve noted, media is one good example of this. In my case, I’m using a RAIDZ1 vdev of 3 disks, with the understanding that things could go south during a resilver.
  - nix23 6y ago
    Yeah true, in my Home-server i have one mirror (2 disks for OS/Swap etc) and one RaidZ1 (5 Disks for data), for me it's just important that i have no bit-rot etc. Daily backups (restic) are made so no problem whatsoever.
- nix23 6y ago
  I then tell a real horror-story about my past, not so long ago there was a EVA (A SAN from HP) with a freaking out hardware Raid out of nowhere (uptime about 200 days no Firmware update before nothing) it just started to freak out, the story goes on but the lesson...No more HW-Raid/proprietary stuff for me EVER again.
  - labawi 6y ago
    While none for me either, I used to look down on HW RAID etc. Lately I've realized, such solutions are genuinely useful and more likely to survive/work properly, if the people managing it do not understand the technology, don't want to understand or just don't have the time for it. HW RAID with red blinking lights seems more likely to survive than a SW RAID that works until it doesn't, because you lost all the redundancy and noone there knows how either work.
jryb 6y ago
My gripe with this is that the author assumes everyone has the same workload and prefers the same set of tradeoffs. I use raidz1 on my home desktop precisely because I would absolutely prefer to have to wait for a resilver than lose data. “So backup your data!” - of course, but that’s just an implicitly larger pool.
- aidenn0 6y ago
  TFA suggests using mirrors instead of RAIDZ. There is no case in which RAIDZ1 is more durable against data loss than a mirror is.
  - Naac 6y ago
    Doesn't that ignore the fact that RAIDZ1 gives you more storage space than a mirror?
    - aidenn0 6y ago
      The comment I was replying to had no mention whatsoever of storage space requirements.
- caillou 6y ago
  I use 10x8TB in RaidZ 2 in my home server. TimeMachine Backup for 6 people, docker volumes and an excessively huge media collection. The TimeMachine datasets are backed up offsite. Losing this pool would be a PITA, but not critical. My primary goal with ZFS is some data redundancy. At a good cost. And quick remote backup for a fraction of the pool. Not performance. At one point, 2 disks died within 2 days. While there was some panic involved, the data on the server could be reproduced with some time. There isn’t a best solution, that fits all needs. If there was, ZFS wouldn‘t offer all the options it does.
joekrill 6y ago
I've been using ZFS for a few years now and it's just been amazing. Storage is so cheap these days that it's just much simpler and straight-forward to use mirrored vdevs. If a single drive fails, the pool is still completely usable, and all I have to do is swap out the bad drive (when I get around to it, no hurry usually) and resilver. Resilvering can take a while, but everything is still completely usable while it's happening so it just runs in the background and I don't even notice. I just upgraded a pool of 4 drives from 4TB each to 10TB - I'd never done anything like that before and was rather nervous, but it was just so simple with the mirrored set up - just swap out each drive and resilver one- by-one.
- jacob019 6y ago
  That's risky. Better to add the new drive and re-silver before removing the old drive.
blueflame7 6y ago
Mirror pools are dangerous if you have a failing power-supply. All it takes is to lose the wrong two drives together (which are often in the same rack) and you lose a portion of your dataset. You should use mirrored raidz2 sets
js2 6y ago
This article is pretty hand-wavy. It doesn't give any empirical numbers at all. I've had a small FreeNAS server using 4 x 3TB SATA drives in a mirrored config for years now and it's out of space. I'm about to build a new server using used 10 x 3TB SAS drives and intend to put all ten disks into a RAIDZ2 vdev. I care more about space than performance or rebuild times. Before I load it with data, I'll do some testing of read/write performance and rebuilding times. If they're unacceptable, I'll try two smaller RAIDZ vdevs and if that still doesn't work, I'll go back to mirrors.
- _jal 6y ago
  The article is another entry in a long series of bad ZFS articles. For some reason a lot of people get to a point where they're comfortable with it and suddenly their use case is everyone's, they've become an expert, and you should Just Do What They Say. I highly recommend people ignore articles like this. ZFS is very flexible, and can serve a variety of workloads. It also assumes you know what you're doing, and the tradeoffs are not always apparent up-front. If you want to become comfortable-enough with ZFS to make your own choices, I recommend standing up your ZFS box well before you need it, and play with it. Set up configs you'd never use in production, just to see what happens. Yank a disk, figure out how to recover. If you have time, fill it up with garbage and look for yourself how fragmentation effects resilver times. If you're a serious user, join the mailing lists - it is high-signal. And value random articles on the interwebs telling you the Real Way at what they cost you. I'm convinced articles like this are a big part of what gives ZFS a bad name. People follow authoritative-sounding bad advice and blame their results on the file system.
  - myrond 6y ago
    bang-on. This article should be thrown in the trash. The EXAMPLES for a raidz and raidz2 and raidz3 are mis-aligned in the article. They will obviously have terrible performance, and waste space due to blocks not fitting nicely due to non-standard physical block sizes. mirror's are easy because you don't have to worry about ASHIFT values, and physical block sizes of devices and making sure all of your blocks can be divided nicely into 128K block sizes. I get the feeling that the author of the article never actually has ran a properly configured raidz(x) with SLOG and cache devices.
Aaargh20318 6y ago
One problem with mirror vdevs vs. RAIDZ2 : In the RAIDZ2 case you can lose any two drives and still have your data, in the mirror vdev case, 2 drives failing in the same vdev means all your data is gone. You could potentially be okay with losing half your drives, but only if all the failed drives happen to be part of different vdevs.
- heavenlyblue 6y ago
  That's why you can RAIDZ3.
tjoff 6y ago
I don't agree. And if one of your disks failed, and age was a factor… you’re going to be sweating bullets wondering if another will fail before your resilver completes. So every every single time you loose any drive in a mirror setup you risk all the data on all drives in the entire pool. I sure do hope you aren't on vacation and/or have to order a drive online. It all depends on your use case. For me, raidz3 wins easily. Performance during resilvering is not something most users would suffer that much from anyway either. And that 8-drive recommendation comes from many factors and don't apply to most home-users anyway. Not that you should go overboard with it but it will easily pay for fast cache or whatever else you might want. If performance was the goal you would not be using spinning rust anyway, and if you are still limited by a single gbit then don't even think about it (but don't over-utilize your pool). But there are still lots of potential ways for your data to die, and you still need to back up your pool. Period. PERIOD! Of course! Yet for home users there is not a single reasonable way to do it if you have decently sized pool. There is bound to be sacrifices on what you choose to backup. Except for maybe another pool. Which is going to hurt since ZFS with buying everything up front is VASTLY more expensive than RAID where you can grow the array as needed - under the assumption that the storage needs grow slowly, which they typically do for home users. As a ZFS user and fanboy the temptation of a dual raid6 setup is tempting. But I can't compromise on the filesystem so in the end I've compromised on backups instead. Likely not the smartest move considering how rare bitrot is, I am very well aware. For many drives I'd go for raidz3. And for a 4-drive NAS I'd go with raidz2 rather than a mirror setup for the ease of mind. For fast SSD pools I'd go with mirrors. Much easier to backup the entire pool as well.
- necheffa 6y ago
  Repeat after me: RAID is not a backup solution, its an uptime solution.
  - tjoff 6y ago
    RAID is not a backup solution, its an uptime solution. I know. Doesn't change the fact there exist no sensible way to backup a decently sized pool in a home-user scenario. Now we might disagree on what is sensible. And some things might not need a backup.
tw04 6y ago
There's no planet on which I would trust a mirror with a 10TB + SATA drive. There's a reason the major storage vendors already have or are working on 3-disk parity for large NL-SAS/SATAS drives. Give me RAID-Z2 or RAID-Z3 with dRAID all day long (although I wouldn't deploy dRAID quite yet on production workloads).
- magicalhippo 6y ago
  You could always go with 3-way or more mirror vdevs in ZFS if you're paranoid (and rich) enough.
angry_octet 6y ago
I would like to see an update for this for SSDs. The devices are black boxes of mystery, possibly likely to fail nearly simultaneously (at least in terms of being unable to write). Early on we left 20% if capacity unformatted to allow more wear leveling, but running them in RAID1 seemed both necessary and likely to cause synchronous errors. RAIDZ1 over three disks might be okay. Obviously huge R/W bandwidth so resilvering doesn't take long. Also, it isn't mentioned, but having the ZIL on battery backed flash can give huge improvements in IOPS to anything, and is far more valuable than extra TB or spindles.
- Rafuino 6y ago
  Battery backed flash sounds like a huge PitA to deal with though. Where do you even get that?
  - angry_octet 6y ago
    The battery is on the HBA. If you lose power it will replay all uncommitted writes. Specifically, to the ZIL but for everything else too. (Originally these were batteries but they are mainly supercapacitors now.) Just saw this interesting SNIA presentation, which indicates that having NVDIMMs might be even better than NVMe for ZIL. https://www.snia.org/sites/default/files/SDC/2018/presentations/PM/Principe_Nick_Real-World_Performance_Advantages_of_NVDIMM_and_NVMe_A_Case_Study_with_OpenZFS.pdf https://www.snia.org/sites/default/files/SDC/2018/presentati... NVDIMM as RAM+FLASH+capacitor: https://www.vikingtechnology.com/products/nvdimm/ https://www.vikingtechnology.com/products/nvdimm/
xoa 6y ago
I looked for, and didn't see, "SSD" in this article (let alone "NVMe"). Maybe because it's from 2015? But at any rate, I'm not sure the logic applies there. High performance SSDs remain much more expensive, so losing major capacity is a much costlier issue, and simultaneously they rebuild vastly faster. I thought about this when making a pool out of U.2 NVMe drives, and with rebuild times measured in minutes and given the cost/GB I think RAIDZ2 (or even Z1) vdevs are plenty sufficient for most use cases. By the same token, what does the backup system and unique pool data lifetime look like? If someone is using a very fast/smaller/expensive pool as a local working space, but it's constantly being replicated to a much more heavily redundant pool of spinning rust in turn backing up sufficiently fast to remote, it may be perfectly acceptable to have minimal redundancy (I still like being able to heal from corruption) in the working pool. If the whole thing going kaput only means losing a few minutes of data it's totally reasonable to consider how much money that's actually worth. I guess a lot of the blanket advice for ZFS rubs me the wrong way. It's offers a very powerful toolbox full of options that are genuinely great in different circumstances, and there aren't many footguns (dedup being the biggest one that immediately comes to mind) that are hard to reason about. It's a shame if users aren't considering there own budgets, needs, hardware, and so on and taking advantage of it to get the most out of them.
jiveturkey 6y ago
needs 2015 tag
rsync 6y ago
We (rsync.net) have several PB of raidz3 deployed all over the world. We use conservatively sized (12-15 drive) vdevs and typically join 3 or 4 of those together to make a pool. I can see getting nervous about raidz2 (sort of analogous to "raid6") after a drive failure ... but losing 4 drives out of 12 in a single raidz3 failure cascade is extremely improbable. We all sleep quite well with this arrangement and have since we first migrated from UFS2 to ZFS in 2012.
- tutfbhuf 6y ago
  May I ask what your worst incident with ZFS was ever since 2012?
  - rsync 6y ago
    Some background ... We have a fairly robust drive-proofing procedure prior to deployment - I think we still use 'badblocks' and really beat them up for 5-7 days. It's not enough to merely not fail - we have to see perfect returns on the SMART diagnostics and zero complaints from FreeBSD. So we're starting with a known-good population of drives. Then we monitor them very closely - again with both SMART and FreeBSD/ZFS system data and we are fairly aggressive about failing them out early if we see them start to misbehave. So we don't see things like straight up drive failures - the bad ones have already been rejected and drives that misbehave are culled out. The last time I got pulled in to make a judgement call (and got a little nervous) was (IIRC) we had a 15 drive vdev that had a drive failure and a candidate for early removal that had started to misbehave and the choice was made to yank them both because why do two big long resilvers ... and then during the subsequent resilver a third drive, while not failing, started to spit out errors. So the resilver completed and all was well but we had a potential third failure that could have died in the resilver and then we would be running with no protection. But that's why I love raidz3 - even if that had happened, we would have needed to lose yet another drive to lose the vdev (and the entire pool). I think the actionable recommendations here are to burn in your drives when you get them because we do, indeed, find rejects in most batches. Also, pick an error threshold that is low and be disciplined about sticking to it - don't let drives spin out SMART errors sporadically for months ... EDIT: Here is another thought and this goes back to pre-ZFS days and old-style RAID, etc. In normal operations we're aggressively failing out drives that misbehave BUT if you're in a marginal situation you need to flip that logic around - especially if the pool is in-use during resilver/rebuild. If you've lost 2 drives in a raidz3 (or, say, 1 drive in a RAID6) that remaining protection drive even if it is failing is like gold. If you fail it out, it's gone forever and has no relevance to the pool/array - but if you keep it, even as it's dying, you can either limp along OR you can even offline the pool/array and send that last drive to recovery or clone it or whatever ... the point is, when an array goes sideways every single bit of parity, no matter how poorly behaved, should be treated like gold.
somehnguy 6y ago
No thanks. I use raidz2. With mirror vdevs if you lose the wrong 2 drives you lose everything. I can lose any 2 drives and be totally fine. The probability of losing the wrong 2 drives at once is small, sure. But I would rather just not care about that probability. And I don't lose half my capacity, which for a home user (I don't have an unlimited budget!) matters a whole lot more than having the absolute best iops.
- segfaultbuserr 6y ago
  The original article already included a reply to this question. > But wait, why would I want to trade guaranteed two disk failure in RAIDZ2 with only 85.7% survival of two disk failure in a pool of mirrors? Because of the drastically shorter time to resilver, and drastically lower load placed on the pool while doing so. The only disk more heavily loaded than usual during a mirror vdev resilvering is the other disk in the vdev – which might sound bad, but remember that it’s no more heavily loaded than it would’ve been as a RAIDZ member. Each block resilvered on a RAIDZ vdev requires a block to be read from each surviving RAIDZ member; each block written to a resilvering mirror only requires one block to be read from a surviving vdev member. For a six-disk RAIDZ1 vs a six disk pool of mirrors, that’s five times the extra I/O demands required of the surviving disks. I think it's perfectly okay to disagree. But any comment must contain a counterargument to be useful, for example, one could've argued that load is not an issue in a small array and so on, and I may happily accept the other side of the argument. However, I respectfully criticize that your comment doesn't include any counterargument, it's not useful, please read the article more carefully next time.
  - somehnguy 6y ago
    It’s not a question, it’s a statement and my opinion on the matter. I respectfully criticize that your comment isn’t helpful at all nor a counter argument and is therefore useless as well. Please read my comment more carefully next time.
- 67868018 6y ago
  You don’t lose everything, only the files that were on that vdev. I’ve been through this. It’s not fun, though, and your pool is irreversibly damaged but your data is not all lost
louwrentius 6y ago
I strongly disagree with this old blogpost. I feel that this advice is a somewhat dishonest attempt to plaster over the fact that you can't expand a VDEV. https://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html https://louwrentius.com/the-hidden-cost-of-using-zfs-for-you... So you try to burry that fact by promoting mirrors. But mirrors aren't as safe as RAIDZ2 and they aren't as space/efficient. It all depends on circumstances, but if you want to store a ton of data, RAIDZ(2|3) seems the right way to go. Use RAIDZ(2|3) vdevs, not mirrors.
mrlonglong 6y ago
I recently built a ZFS raidz2 array with 4 disks. Is the article saying it's better to mirror over four vdevs (or 4 disks?)
- benlivengood 6y ago
  I did the same thing as you. I've had to sit through an RMA period on replacing a drive in a home RAID5 and just never wanted anything like that again; every replacement of a disk in a mirrored vdev would feel the same. Data is backed up but it's a headache to have data unavailable.
deleted 6y ago
[deleted]
sgarland 6y ago
As long as you aren't relying on it as backup, it doesn't matter for most use cases. I'm about to build a zpool consisting of nothing but 3-wide raidz1 vdevs. I can tolerate one drive dying. In the ~8 years or so I've been running a NAS, I've had precisely one drive failure. I am fully aware that survivorship bias is a thing, and anecdotes aren't data, but it's good enough for me. Anything important is backed up locally and to the cloud. Everything else is merely annoying to have to download again.
myrond 6y ago
I disagree. With co-located boxes and drop shipped drive replacements the time between a FAULT and the resilver event can be multiple days. Even though the resilver will go faster with a mirror having one disk remaining on a mirror vdev compared to raidz2 (or higher) mirrors will increase risk of data loss irrespective of resilver times because of drop ship drive replacement time. 3TB resilver on my last mechanical drive failure took 6 hours 30 minutes. Plus an additional 3 days for the drive to arrive. With mirror vdev setups you lose significantly more space as well. If you argue speed is worth it, then I would instead invest that money you saved going with a raidz2 with NVME cache and SLOG. Users won't notice the resilver event at all with a significant amount of memory and NVME cache + nvme SLOG tuned with a high /sys/module/zfs/parameters/zfs_dirty_data_max and larger than default /sys/module/zfs/parameters/zfs_txg_timeout.

"Use RAID10" is basically the storage equivalent of "use paper ballots". Sure, it feels suboptimal, but critically your intuition on how it can fail (mostly) works. That's a really nice property to have for storage.

chokeartist 6y ago

Parity pools are for poors. Mirror pools are for chads.

nix23 6y ago

For big installations i do striped (sometimes mirrors) vdevs (2 disks) and on top of them Raidz2/3. Works fast as hell and reliable.

npteljes 6y ago

I had my pitchforks out because I really like my raidz1 setup, but the arguments are very good for the mirror. When I set my pool up, I chose raidz1 because compared to mirroring, I still have 37.5% more space, which is huge if the budget is constrained. But if you feel paranoid and the raidz3 begin to sound good, that 12.5% extra space doesn't seem to worth the extra complexity over a much straightforward mirror.

gbrown_ 6y ago

> don’t be greedy. 50% storage efficiency is plenty. LOL.

deleted 6y ago
[deleted]

minimaul 6y ago

As always with RAID-style setups, there’s an inevitable trade off of cost vs capacity vs performance. There’s still a place for RAIDZ/RAIDZ2, and in my opinion that place is storing bulk data that isn’t too heavily accessed or that needs to be stored with an eye towards keeping £/GB down. Yes, mirrors are faster. Yes, mirrors are easier to expand. But across 12 4TB disks that is 24TB instead of 40TB with RAIDZ2 - and that’s a lot of capacity to lose if you’re on a budget. The rebuild times in this post seem high to me, though. I replaced 7x 2TB disks (nearly full) in a raidz1 in a backup pool with larger drives in about 30 hours.

minimaul 6y ago
The one thing in the article I agree with without reservation though is that you should always have a backup of your pool! ZFS makes doing good backups easy, with zfs send | zfs receive.
- gh02t 6y ago
  I mean, is this controversial? The problem for a budget conscious home user is: where do you back up to? Doing a complete backup to a remote storage provider isn't a practical option for many users, which leaves the affordable options being to only back up critical data and/or use something like a big USB external drive. Neither of these options are great, so in reality many people end up relying on RAID or mirroring to provide some measure of redundancy regardless of whether or not it's a good idea.
  - benlivengood 6y ago
    I just started playing around with s3blkdevd. S3-compatible storage at backblaze B2 is $0.005/GB-month and I'll see if I can manage a vdev made out of a huge nbd disk. Topical to the article, backblaze has piles of RAID6 and appears to be happy with it. Another thought has been to find a ZFS-friend in another part of the world and exchange snapshots. Inline encryption makes that a little more viable now. Right now I'm sending incremental snapshots to AWS Glacier and have another local box mirroring the data (mostly to fully verify the snapshots before sending them to Glacier).
  - unethical_ban 6y ago
    I have a raidz2 that I keep everything on: (A) Personal, (B) core media, and (C) all other media. I backup A and B on a workstation drive, and I further backup A to S3 Glacier for $1/month/100GB, syncing every few months. It isn't all or nothing, 3-copies vs. striped pool. Mirrors are fine if you have quick access to fix dead drives, but I like the idea of being able to lose any two drives vs. only one drive in a given mirror. And while I currently have a 4 drive RAIDz2 (same disk utilization as a mirror) I wish I had a 4+2 array, which I'll probably build next year.

geerlingguy 6y ago

Every time I make this argument (whether regular RAID or ZFS), everyone pulls out their pitchforks and tells me about how they have run RAID 5 or 6 for years and never had a problem, plus "I can have two drives fail! You would lose your array if the wrong two drives fail!" But drive failure is much more likely during an operation like a resilver/resync, and I wonder if the majority of people espousing more risky RAID a setups (especially single parity) are those who just want hundreds of TB for a growing media collection. I know for my critical data, I don't trust parity. Plus I can't afford the days or weeks long resilver operation, I need a working storage array that doesn't suffer drastically in performance if one drive goes bad.

Cyph0n 6y ago
Critical data for the average user typically does not require much space, so the cost of storage overhead in a mirrored ZFS pool is not too bad. RAIDZ is much more cost efficient when you need to reliably store large content that isn’t really critical. As you’ve noted, media is one good example of this. In my case, I’m using a RAIDZ1 vdev of 3 disks, with the understanding that things could go south during a resilver.
- nix23 6y ago
  Yeah true, in my Home-server i have one mirror (2 disks for OS/Swap etc) and one RaidZ1 (5 Disks for data), for me it's just important that i have no bit-rot etc. Daily backups (restic) are made so no problem whatsoever.
nix23 6y ago
I then tell a real horror-story about my past, not so long ago there was a EVA (A SAN from HP) with a freaking out hardware Raid out of nowhere (uptime about 200 days no Firmware update before nothing) it just started to freak out, the story goes on but the lesson...No more HW-Raid/proprietary stuff for me EVER again.
- labawi 6y ago
  While none for me either, I used to look down on HW RAID etc. Lately I've realized, such solutions are genuinely useful and more likely to survive/work properly, if the people managing it do not understand the technology, don't want to understand or just don't have the time for it. HW RAID with red blinking lights seems more likely to survive than a SW RAID that works until it doesn't, because you lost all the redundancy and noone there knows how either work.

jryb 6y ago

My gripe with this is that the author assumes everyone has the same workload and prefers the same set of tradeoffs. I use raidz1 on my home desktop precisely because I would absolutely prefer to have to wait for a resilver than lose data. “So backup your data!” - of course, but that’s just an implicitly larger pool.

aidenn0 6y ago
TFA suggests using mirrors instead of RAIDZ. There is no case in which RAIDZ1 is more durable against data loss than a mirror is.
- Naac 6y ago
  Doesn't that ignore the fact that RAIDZ1 gives you more storage space than a mirror?
  - aidenn0 6y ago
    The comment I was replying to had no mention whatsoever of storage space requirements.
caillou 6y ago
I use 10x8TB in RaidZ 2 in my home server. TimeMachine Backup for 6 people, docker volumes and an excessively huge media collection. The TimeMachine datasets are backed up offsite. Losing this pool would be a PITA, but not critical. My primary goal with ZFS is some data redundancy. At a good cost. And quick remote backup for a fraction of the pool. Not performance. At one point, 2 disks died within 2 days. While there was some panic involved, the data on the server could be reproduced with some time. There isn’t a best solution, that fits all needs. If there was, ZFS wouldn‘t offer all the options it does.

joekrill 6y ago

I've been using ZFS for a few years now and it's just been amazing. Storage is so cheap these days that it's just much simpler and straight-forward to use mirrored vdevs. If a single drive fails, the pool is still completely usable, and all I have to do is swap out the bad drive (when I get around to it, no hurry usually) and resilver. Resilvering can take a while, but everything is still completely usable while it's happening so it just runs in the background and I don't even notice. I just upgraded a pool of 4 drives from 4TB each to 10TB - I'd never done anything like that before and was rather nervous, but it was just so simple with the mirrored set up - just swap out each drive and resilver one- by-one.

jacob019 6y ago
That's risky. Better to add the new drive and re-silver before removing the old drive.

blueflame7 6y ago

Mirror pools are dangerous if you have a failing power-supply. All it takes is to lose the wrong two drives together (which are often in the same rack) and you lose a portion of your dataset. You should use mirrored raidz2 sets

js2 6y ago

This article is pretty hand-wavy. It doesn't give any empirical numbers at all. I've had a small FreeNAS server using 4 x 3TB SATA drives in a mirrored config for years now and it's out of space. I'm about to build a new server using used 10 x 3TB SAS drives and intend to put all ten disks into a RAIDZ2 vdev. I care more about space than performance or rebuild times. Before I load it with data, I'll do some testing of read/write performance and rebuilding times. If they're unacceptable, I'll try two smaller RAIDZ vdevs and if that still doesn't work, I'll go back to mirrors.

_jal 6y ago
The article is another entry in a long series of bad ZFS articles. For some reason a lot of people get to a point where they're comfortable with it and suddenly their use case is everyone's, they've become an expert, and you should Just Do What They Say. I highly recommend people ignore articles like this. ZFS is very flexible, and can serve a variety of workloads. It also assumes you know what you're doing, and the tradeoffs are not always apparent up-front. If you want to become comfortable-enough with ZFS to make your own choices, I recommend standing up your ZFS box well before you need it, and play with it. Set up configs you'd never use in production, just to see what happens. Yank a disk, figure out how to recover. If you have time, fill it up with garbage and look for yourself how fragmentation effects resilver times. If you're a serious user, join the mailing lists - it is high-signal. And value random articles on the interwebs telling you the Real Way at what they cost you. I'm convinced articles like this are a big part of what gives ZFS a bad name. People follow authoritative-sounding bad advice and blame their results on the file system.
- myrond 6y ago
  bang-on. This article should be thrown in the trash. The EXAMPLES for a raidz and raidz2 and raidz3 are mis-aligned in the article. They will obviously have terrible performance, and waste space due to blocks not fitting nicely due to non-standard physical block sizes. mirror's are easy because you don't have to worry about ASHIFT values, and physical block sizes of devices and making sure all of your blocks can be divided nicely into 128K block sizes. I get the feeling that the author of the article never actually has ran a properly configured raidz(x) with SLOG and cache devices.

Aaargh20318 6y ago

One problem with mirror vdevs vs. RAIDZ2 : In the RAIDZ2 case you can lose any two drives and still have your data, in the mirror vdev case, 2 drives failing in the same vdev means all your data is gone. You could potentially be okay with losing half your drives, but only if all the failed drives happen to be part of different vdevs.

heavenlyblue 6y ago
That's why you can RAIDZ3.

tjoff 6y ago

I don't agree. And if one of your disks failed, and age was a factor… you’re going to be sweating bullets wondering if another will fail before your resilver completes. So every every single time you loose any drive in a mirror setup you risk all the data on all drives in the entire pool. I sure do hope you aren't on vacation and/or have to order a drive online. It all depends on your use case. For me, raidz3 wins easily. Performance during resilvering is not something most users would suffer that much from anyway either. And that 8-drive recommendation comes from many factors and don't apply to most home-users anyway. Not that you should go overboard with it but it will easily pay for fast cache or whatever else you might want. If performance was the goal you would not be using spinning rust anyway, and if you are still limited by a single gbit then don't even think about it (but don't over-utilize your pool). But there are still lots of potential ways for your data to die, and you still need to back up your pool. Period. PERIOD! Of course! Yet for home users there is not a single reasonable way to do it if you have decently sized pool. There is bound to be sacrifices on what you choose to backup. Except for maybe another pool. Which is going to hurt since ZFS with buying everything up front is VASTLY more expensive than RAID where you can grow the array as needed - under the assumption that the storage needs grow slowly, which they typically do for home users. As a ZFS user and fanboy the temptation of a dual raid6 setup is tempting. But I can't compromise on the filesystem so in the end I've compromised on backups instead. Likely not the smartest move considering how rare bitrot is, I am very well aware. For many drives I'd go for raidz3. And for a 4-drive NAS I'd go with raidz2 rather than a mirror setup for the ease of mind. For fast SSD pools I'd go with mirrors. Much easier to backup the entire pool as well.

necheffa 6y ago
Repeat after me: RAID is not a backup solution, its an uptime solution.
- tjoff 6y ago
  RAID is not a backup solution, its an uptime solution. I know. Doesn't change the fact there exist no sensible way to backup a decently sized pool in a home-user scenario. Now we might disagree on what is sensible. And some things might not need a backup.

tw04 6y ago

There's no planet on which I would trust a mirror with a 10TB + SATA drive. There's a reason the major storage vendors already have or are working on 3-disk parity for large NL-SAS/SATAS drives. Give me RAID-Z2 or RAID-Z3 with dRAID all day long (although I wouldn't deploy dRAID quite yet on production workloads).

magicalhippo 6y ago
You could always go with 3-way or more mirror vdevs in ZFS if you're paranoid (and rich) enough.

angry_octet 6y ago

I would like to see an update for this for SSDs. The devices are black boxes of mystery, possibly likely to fail nearly simultaneously (at least in terms of being unable to write). Early on we left 20% if capacity unformatted to allow more wear leveling, but running them in RAID1 seemed both necessary and likely to cause synchronous errors. RAIDZ1 over three disks might be okay. Obviously huge R/W bandwidth so resilvering doesn't take long. Also, it isn't mentioned, but having the ZIL on battery backed flash can give huge improvements in IOPS to anything, and is far more valuable than extra TB or spindles.

Rafuino 6y ago
Battery backed flash sounds like a huge PitA to deal with though. Where do you even get that?
- angry_octet 6y ago
  The battery is on the HBA. If you lose power it will replay all uncommitted writes. Specifically, to the ZIL but for everything else too. (Originally these were batteries but they are mainly supercapacitors now.) Just saw this interesting SNIA presentation, which indicates that having NVDIMMs might be even better than NVMe for ZIL. https://www.snia.org/sites/default/files/SDC/2018/presentations/PM/Principe_Nick_Real-World_Performance_Advantages_of_NVDIMM_and_NVMe_A_Case_Study_with_OpenZFS.pdf https://www.snia.org/sites/default/files/SDC/2018/presentati... NVDIMM as RAM+FLASH+capacitor: https://www.vikingtechnology.com/products/nvdimm/ https://www.vikingtechnology.com/products/nvdimm/

xoa 6y ago

I looked for, and didn't see, "SSD" in this article (let alone "NVMe"). Maybe because it's from 2015? But at any rate, I'm not sure the logic applies there. High performance SSDs remain much more expensive, so losing major capacity is a much costlier issue, and simultaneously they rebuild vastly faster. I thought about this when making a pool out of U.2 NVMe drives, and with rebuild times measured in minutes and given the cost/GB I think RAIDZ2 (or even Z1) vdevs are plenty sufficient for most use cases. By the same token, what does the backup system and unique pool data lifetime look like? If someone is using a very fast/smaller/expensive pool as a local working space, but it's constantly being replicated to a much more heavily redundant pool of spinning rust in turn backing up sufficiently fast to remote, it may be perfectly acceptable to have minimal redundancy (I still like being able to heal from corruption) in the working pool. If the whole thing going kaput only means losing a few minutes of data it's totally reasonable to consider how much money that's actually worth. I guess a lot of the blanket advice for ZFS rubs me the wrong way. It's offers a very powerful toolbox full of options that are genuinely great in different circumstances, and there aren't many footguns (dedup being the biggest one that immediately comes to mind) that are hard to reason about. It's a shame if users aren't considering there own budgets, needs, hardware, and so on and taking advantage of it to get the most out of them.

jiveturkey 6y ago

needs 2015 tag

rsync 6y ago

We (rsync.net) have several PB of raidz3 deployed all over the world. We use conservatively sized (12-15 drive) vdevs and typically join 3 or 4 of those together to make a pool. I can see getting nervous about raidz2 (sort of analogous to "raid6") after a drive failure ... but losing 4 drives out of 12 in a single raidz3 failure cascade is extremely improbable. We all sleep quite well with this arrangement and have since we first migrated from UFS2 to ZFS in 2012.

tutfbhuf 6y ago
May I ask what your worst incident with ZFS was ever since 2012?
- rsync 6y ago
  Some background ... We have a fairly robust drive-proofing procedure prior to deployment - I think we still use 'badblocks' and really beat them up for 5-7 days. It's not enough to merely not fail - we have to see perfect returns on the SMART diagnostics and zero complaints from FreeBSD. So we're starting with a known-good population of drives. Then we monitor them very closely - again with both SMART and FreeBSD/ZFS system data and we are fairly aggressive about failing them out early if we see them start to misbehave. So we don't see things like straight up drive failures - the bad ones have already been rejected and drives that misbehave are culled out. The last time I got pulled in to make a judgement call (and got a little nervous) was (IIRC) we had a 15 drive vdev that had a drive failure and a candidate for early removal that had started to misbehave and the choice was made to yank them both because why do two big long resilvers ... and then during the subsequent resilver a third drive, while not failing, started to spit out errors. So the resilver completed and all was well but we had a potential third failure that could have died in the resilver and then we would be running with no protection. But that's why I love raidz3 - even if that had happened, we would have needed to lose yet another drive to lose the vdev (and the entire pool). I think the actionable recommendations here are to burn in your drives when you get them because we do, indeed, find rejects in most batches. Also, pick an error threshold that is low and be disciplined about sticking to it - don't let drives spin out SMART errors sporadically for months ... EDIT: Here is another thought and this goes back to pre-ZFS days and old-style RAID, etc. In normal operations we're aggressively failing out drives that misbehave BUT if you're in a marginal situation you need to flip that logic around - especially if the pool is in-use during resilver/rebuild. If you've lost 2 drives in a raidz3 (or, say, 1 drive in a RAID6) that remaining protection drive even if it is failing is like gold. If you fail it out, it's gone forever and has no relevance to the pool/array - but if you keep it, even as it's dying, you can either limp along OR you can even offline the pool/array and send that last drive to recovery or clone it or whatever ... the point is, when an array goes sideways every single bit of parity, no matter how poorly behaved, should be treated like gold.

somehnguy 6y ago

No thanks. I use raidz2. With mirror vdevs if you lose the wrong 2 drives you lose everything. I can lose any 2 drives and be totally fine. The probability of losing the wrong 2 drives at once is small, sure. But I would rather just not care about that probability. And I don't lose half my capacity, which for a home user (I don't have an unlimited budget!) matters a whole lot more than having the absolute best iops.

segfaultbuserr 6y ago
The original article already included a reply to this question. > But wait, why would I want to trade guaranteed two disk failure in RAIDZ2 with only 85.7% survival of two disk failure in a pool of mirrors? Because of the drastically shorter time to resilver, and drastically lower load placed on the pool while doing so. The only disk more heavily loaded than usual during a mirror vdev resilvering is the other disk in the vdev – which might sound bad, but remember that it’s no more heavily loaded than it would’ve been as a RAIDZ member. Each block resilvered on a RAIDZ vdev requires a block to be read from each surviving RAIDZ member; each block written to a resilvering mirror only requires one block to be read from a surviving vdev member. For a six-disk RAIDZ1 vs a six disk pool of mirrors, that’s five times the extra I/O demands required of the surviving disks. I think it's perfectly okay to disagree. But any comment must contain a counterargument to be useful, for example, one could've argued that load is not an issue in a small array and so on, and I may happily accept the other side of the argument. However, I respectfully criticize that your comment doesn't include any counterargument, it's not useful, please read the article more carefully next time.
- somehnguy 6y ago
  It’s not a question, it’s a statement and my opinion on the matter. I respectfully criticize that your comment isn’t helpful at all nor a counter argument and is therefore useless as well. Please read my comment more carefully next time.
67868018 6y ago
You don’t lose everything, only the files that were on that vdev. I’ve been through this. It’s not fun, though, and your pool is irreversibly damaged but your data is not all lost

louwrentius 6y ago

I strongly disagree with this old blogpost. I feel that this advice is a somewhat dishonest attempt to plaster over the fact that you can't expand a VDEV. https://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html https://louwrentius.com/the-hidden-cost-of-using-zfs-for-you... So you try to burry that fact by promoting mirrors. But mirrors aren't as safe as RAIDZ2 and they aren't as space/efficient. It all depends on circumstances, but if you want to store a ton of data, RAIDZ(2|3) seems the right way to go. Use RAIDZ(2|3) vdevs, not mirrors.

mrlonglong 6y ago

I recently built a ZFS raidz2 array with 4 disks. Is the article saying it's better to mirror over four vdevs (or 4 disks?)

benlivengood 6y ago
I did the same thing as you. I've had to sit through an RMA period on replacing a drive in a home RAID5 and just never wanted anything like that again; every replacement of a disk in a mirrored vdev would feel the same. Data is backed up but it's a headache to have data unavailable.

deleted 6y ago

[deleted]

sgarland 6y ago

As long as you aren't relying on it as backup, it doesn't matter for most use cases. I'm about to build a zpool consisting of nothing but 3-wide raidz1 vdevs. I can tolerate one drive dying. In the ~8 years or so I've been running a NAS, I've had precisely one drive failure. I am fully aware that survivorship bias is a thing, and anecdotes aren't data, but it's good enough for me. Anything important is backed up locally and to the cloud. Everything else is merely annoying to have to download again.

myrond 6y ago

I disagree. With co-located boxes and drop shipped drive replacements the time between a FAULT and the resilver event can be multiple days. Even though the resilver will go faster with a mirror having one disk remaining on a mirror vdev compared to raidz2 (or higher) mirrors will increase risk of data loss irrespective of resilver times because of drop ship drive replacement time. 3TB resilver on my last mechanical drive failure took 6 hours 30 minutes. Plus an additional 3 days for the drive to arrive. With mirror vdev setups you lose significantly more space as well. If you argue speed is worth it, then I would instead invest that money you saved going with a raidz2 with NVME cache and SLOG. Users won't notice the resilver event at all with a significant amount of memory and NVME cache + nvme SLOG tuned with a high /sys/module/zfs/parameters/zfs_dirty_data_max and larger than default /sys/module/zfs/parameters/zfs_txg_timeout.