6 ms·

I use ZoL for my home-server (5x3TB RAIDZ) and couldn't be happier, never had an issue too. The features really spoiled me compared to other filesystems/volume-

by vmp 9y ago

I use ZoL for my home-server (5x3TB RAIDZ) and couldn't be happier, never had an issue too. The features really spoiled me compared to other filesystems/volume-managers. The only issue that irks me is that I can't resize a pool after it was created, can't add new disks, can't remove one (without risking parity). Recently I tried running a simple BTRFS SSD Mirror for only a single KVM virtual machine, I thought compression would be neat since they are just two cheap 120GB SSDs and I wanted to have a spare if one of them gave up the ghost. At first everything was great but after I ran a btrfs scrub, and it found ~4000 uncorrectable errors, my VM was broken beyond repair and had to be restored from backup. The SSDs SMART data was fine, there weren't any loose cables, everything worked great, no reported (ECC) memory errors... I have no proof but it seems that BTRFS just decided to destroy itself. I have since moved my SSDs to a ZoL mirror and (after running scrubs every two days for two weeks) had no further corruption, silent or otherwise. To me, this means that btrfs just isn't stable enough for production use - while ZoL is.

gregmac 9y ago
> The only issue that irks me is that I can't resize a pool after it was created, can't add new disks, can't remove one (without risking parity). This is what eliminates it from home use for me. It means to expand the pool the only way is to copy everything off (so you need enough spare storage to hold your entire pool), rebuild, then copy it back. The alternative is don't expand, and just add additional pools, but that starts losing benefits quickly (not having to think about where there's free space when adding something, not having to look in multiple locations to find something). Can anyone who has been using ZFS long-term at home comment? How do you add more space?
- bestham 9y ago
  I have run ZFS at home since 2011 and migrated to larger drives one time: from 6x3TB to 6x4TB. The only downside is that the slabs are smaller compared to a native 6x4TB array.
  - tscs37 9y ago
    That's not the problem here. The problem is having a 6x3TB pool and turning it into a 7x3TB pool, which is arguably much much cheaper than buying 6x4TB.
  - deleted 9y ago
    [deleted]
- ScottBurson 9y ago
  There is another way to grow a pool: replace each disk with a larger one, resilvering after each replacement. But yeah, sizing your pool very generously when first building it is a good idea. I found that 6x 4TB drives in a raidz2 pool was within my budget, and it will take me a long time to fill 16TB.
  - StavrosK 9y ago
    What does the gp mean by saying there's no way to resize a pool? You can resize a pool by adding disks like you say, so what's the alternative? Resizing by only adding one disk? But it's RAID-Z, where will the parity go? Does anyone support this use case?
    - Ded7xSEoPKYNsDd 9y ago
      Btrfs supports it. The parity stays exactly where it is, but new files will use the new device. You can (and far too often have to, when btrfs decides there is no free space left) also perform a 'balance' operation, which recreates all the b-trees on the disks, optionally with different parity options.
      - StavrosK 9y ago
        Hmm, I don't understand how that's possible on RAID-Z? You can only have as much space per disk as the parity, no? I.e. you can't have three 500 GB disks for a total of 1 GB space, and replace one with a 1 GB disk and get more space, can you?
        rocqua 9y ago
        The idea is to add disks rather than replace. So to go from 3x500GB to 4x500GB.
        StavrosK 9y ago
        Oh, I see. In that case, it's odd to me that ZFS doesn't support that, hmm.
- gruturo 9y ago
  Uh? Let's say you have a simple mirror of 2 vdevs, you go like this: zpool attach <poolname> <first existing small vdev> <first larger new vdev> zpool attach <poolname> <second existing small vdev> <second larger new vdev> [...wait for the resilvering of the new vdevs...] You now have a 4-way mirror with 2 small and 2 large vdevs. Detach the small ones: zpool detach <poolname> <first old vdev> zpool detach <poolname> <second old vdev> Now you have a pool made only of large vdevs. You just give the pool the permission to expand and occupy all this new space: zpool set autoexpand=on <poolname> Done. Did it oodles of times on Solaris with SAN storage, but did it at home too, and in the weirdest ways (no SATA ports available? No problem, attach the new drive via USB3, then when finished take it out of the enclosure and install it inside displacing the old drive), sometimes even rather unsafely (creating a pre-broken raidz with 4 good disks plus a sparse file, migrating data off a different pool, then decommissioning that old pool and using one of its disk to replace the sparse file).
  - detaro 9y ago
    The commonly wished-for feature is not to entirely replace disks as you describe, but to expand it bit by bit. To be able to have e.g. a z2 setup spanning 4 disks of equal size and then add a 5th one later to get one disk's worth of extra capacity.
    - gruturo 9y ago
      If you mean turning a 4-disk raidz2 into a 5-disk raidz2, indeed that's unfortunately not yet possible (unless you had built it with an extra vdev sparse file and run it in this degraded state - effectively you had a raidz and not a z2, and it only works once). If you want, though, you can add capacity by attaching a second raidz2 to an existing pool - this forces you to add 4 new drives instead of one, but it works: zpool add <poolname> raidz2 <first new vdev> <second new vdev> <third new vdev> <fourth new vdev> You now have a concat of 2 raidz2's, in a single pool. Nowhere as elegant as merging a new slice in an existing z2, I concur. Does the job though. Oh and if you are insane you could probably even add a single device, resulting in a concat of a z2 + unmirrored single device. I don't think it will stop you from doing that.
- hks12 9y ago
  It's much easier to grow if you use stripe of mirrors instead of raidz. You waste more space, but gain more flexibility and performance. More details here: http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-... For example, you can start with 1 mirrored vdev, then add another mirrored vdev. You can upgrade vdevs separately, so you'll need to replace just 2 disks to grow your pool. The only thing you have to keep in mind is that data is striped across vdevs only when written, so if you add another vdev, you won't get performance gains for data which was written just to one vdev.
  - tscs37 9y ago
    It's also important to keep in mind that a stripe of mirrors, aka RAID10, has a higher failure probability compared to RAIDZ2 or higher on most common disk setups by several orders of magnitude.
    - rleigh 9y ago
      You also have to factor in the speed of a resilver and account for the chance of a failure during this at risk window. A mirror can resilver much faster, and it doesn't significantly affect the vdev performance while doing so. A RAIDZ resilver after a disc replacement can take a significant amount of time, and degrade performance seriously as it thrashes every single disc in the vdev. Allan Jude and Michael Lucas' books on ZFS have tables describing the tradeoffs of the different possible vdev layouts, and they are worth a read for anyone setting up ZFS storage.
      - tscs37 9y ago
        The danger of loosing data on a RAID10 resilver is much higher than people expect. A 6x8TB RAID10 has a 47.3% chance of encountering a URE during a rebuild, which means data loss. A 6x8TB RAID6 has a 0.0002% chance of a URE. (Assuming URE rate of 10^-14, in reality this rate is lower)
- lmm 9y ago
  I have a pool of two raidz2s of four disks each, one twice the size of the other; every so often I replace one set of disks with disks that are 4x the size (i.e. I started with 4x250gb + 4x500gb, after a few years I replaced the 250gb disks with 1tb disks, right now I have 4xt2b and 4x4tb - and I get to store half as much data as the total capacity, so 12tb at the moment). If a disk dies close to the time I was thinking of upgrading then I'll replace that disk with one of the "new" size (but can't use the extra space until I do the rest of the replacement). It works pretty well - by the time I'm buying disks that are 4x as large I don't mind throwing the old disks away. I've definitely avoided data loss in scenarios where I'd previously lost data under linux md (which lacks checksums and handles disks with isolated UREs very poorly).
- luca_ing 9y ago
  > Can anyone who has been using ZFS long-term at home comment? How do you add more space? My solution is to use RAID1 exclusively. That means that I can keep attaching pairs of devices if I run out of diskspace. I can never get them out again, however :-)
- arwineap 9y ago
  Whenever I run out of space, I hop on newegg, and search for drives twice my current size. I buy the first one I find immediately. Once I get it, I hot swap a drive, and start searching for sales on more drives. Rinse, repeat, all 5 disks swapped, more space appears \o/
Annatar 9y ago
On ZFS, a pool is resized by replacing all the drives in the pool. When autoexpand=on and the last drive is replaced, the pool will have expanded capacity. And drives are cheap. When another RAID device is added to the pool instead, it is used as a stripe. Different types of RAID can be combined in this way, for example one can add a mirrored stripe to an already existing RAID-Z. Whether this is desired or not is a different discussion, since the point here is that both scenarios are possible. Unlike other volume managers, ZFS expands on the disk boundary, rather than physical or logical elements; it's a larger boundary, but since no arcane or complex procedures or knowledge are required and drives are cheap, it's very practical, as well as elegant.
vpeters25 9y ago
> The only issue that irks me is that I can't resize a pool after it was created, can't add new disks, can't remove one (without risking parity). Not sure why ZFS would not auto resize for you. It's the reason I have been using it for several years as my home NAS under a linux server. In my case I just replace all the disk on a RAIDZ with bigger ones and automatically resizes up.
- scottLobster 9y ago
  Which forces you to waste $/GB and $$ in general on multiple higher capacity disks as opposed to just buying a single new drive. Not that my 10 TB RAIDZ is running out of space any time soon, but as soon as Btrfs gets their shit together I'm switching. At the rate of a few blu rays a year it'll fill up sooner or later.
FractalNerve 9y ago
Can't you just use ZFS without RAIDZ and still have your data protected from corruption/drive failures? I think storage is hard and I never understood the advantage of RAID (at least for home usage). It really only looks like an inflexible option, with too much risk to me. What's the benefit of RAIDZ over say, you choose to have X copies distributed over your disk(s)? Answer: zfs: copies=n is not a substitute for device redundancy! source: http://jrs-s.net/2016/05/02/zfs-copies-equals-n/ http://jrs-s.net/2016/05/02/zfs-copies-equals-n/ Here's a discussion about it: https://www.reddit.com/r/DataHoarder/comments/4hbn8v/raidz_vs_copiesn/ https://www.reddit.com/r/DataHoarder/comments/4hbn8v/raidz_v... Anyone who wants to have real security, relies on off-site backups, isn't that right? And aren't RAID(Z)s slow to recover also? (serious questions, I'm a zfs noob) RAIDZ: I don't know what stripe-set configuration is good for me and don't want to waste time comparing RAID controllers or if I even need one. Then configuring the beast on the hardware and software side just seems to be too tedious. Why not just (de)/attach another disk and let zfs expand/shrink my total disk space without loosing consistency? Startup idea: Someone clever should find a flexible storage solution that uses aufs, unionfs etc. to give you the flexibility we need.

by vmp 9y ago

gregmac 9y ago

> The only issue that irks me is that I can't resize a pool after it was created, can't add new disks, can't remove one (without risking parity). This is what eliminates it from home use for me. It means to expand the pool the only way is to copy everything off (so you need enough spare storage to hold your entire pool), rebuild, then copy it back. The alternative is don't expand, and just add additional pools, but that starts losing benefits quickly (not having to think about where there's free space when adding something, not having to look in multiple locations to find something). Can anyone who has been using ZFS long-term at home comment? How do you add more space?

bestham 9y ago
I have run ZFS at home since 2011 and migrated to larger drives one time: from 6x3TB to 6x4TB. The only downside is that the slabs are smaller compared to a native 6x4TB array.
- tscs37 9y ago
  That's not the problem here. The problem is having a 6x3TB pool and turning it into a 7x3TB pool, which is arguably much much cheaper than buying 6x4TB.
- deleted 9y ago
  [deleted]
ScottBurson 9y ago
There is another way to grow a pool: replace each disk with a larger one, resilvering after each replacement. But yeah, sizing your pool very generously when first building it is a good idea. I found that 6x 4TB drives in a raidz2 pool was within my budget, and it will take me a long time to fill 16TB.
- StavrosK 9y ago
  What does the gp mean by saying there's no way to resize a pool? You can resize a pool by adding disks like you say, so what's the alternative? Resizing by only adding one disk? But it's RAID-Z, where will the parity go? Does anyone support this use case?
  - Ded7xSEoPKYNsDd 9y ago
    Btrfs supports it. The parity stays exactly where it is, but new files will use the new device. You can (and far too often have to, when btrfs decides there is no free space left) also perform a 'balance' operation, which recreates all the b-trees on the disks, optionally with different parity options.
    - StavrosK 9y ago
      Hmm, I don't understand how that's possible on RAID-Z? You can only have as much space per disk as the parity, no? I.e. you can't have three 500 GB disks for a total of 1 GB space, and replace one with a 1 GB disk and get more space, can you?
      - rocqua 9y ago
        The idea is to add disks rather than replace. So to go from 3x500GB to 4x500GB.
        StavrosK 9y ago
        Oh, I see. In that case, it's odd to me that ZFS doesn't support that, hmm.
gruturo 9y ago
Uh? Let's say you have a simple mirror of 2 vdevs, you go like this: zpool attach <poolname> <first existing small vdev> <first larger new vdev> zpool attach <poolname> <second existing small vdev> <second larger new vdev> [...wait for the resilvering of the new vdevs...] You now have a 4-way mirror with 2 small and 2 large vdevs. Detach the small ones: zpool detach <poolname> <first old vdev> zpool detach <poolname> <second old vdev> Now you have a pool made only of large vdevs. You just give the pool the permission to expand and occupy all this new space: zpool set autoexpand=on <poolname> Done. Did it oodles of times on Solaris with SAN storage, but did it at home too, and in the weirdest ways (no SATA ports available? No problem, attach the new drive via USB3, then when finished take it out of the enclosure and install it inside displacing the old drive), sometimes even rather unsafely (creating a pre-broken raidz with 4 good disks plus a sparse file, migrating data off a different pool, then decommissioning that old pool and using one of its disk to replace the sparse file).
- detaro 9y ago
  The commonly wished-for feature is not to entirely replace disks as you describe, but to expand it bit by bit. To be able to have e.g. a z2 setup spanning 4 disks of equal size and then add a 5th one later to get one disk's worth of extra capacity.
  - gruturo 9y ago
    If you mean turning a 4-disk raidz2 into a 5-disk raidz2, indeed that's unfortunately not yet possible (unless you had built it with an extra vdev sparse file and run it in this degraded state - effectively you had a raidz and not a z2, and it only works once). If you want, though, you can add capacity by attaching a second raidz2 to an existing pool - this forces you to add 4 new drives instead of one, but it works: zpool add <poolname> raidz2 <first new vdev> <second new vdev> <third new vdev> <fourth new vdev> You now have a concat of 2 raidz2's, in a single pool. Nowhere as elegant as merging a new slice in an existing z2, I concur. Does the job though. Oh and if you are insane you could probably even add a single device, resulting in a concat of a z2 + unmirrored single device. I don't think it will stop you from doing that.
hks12 9y ago
It's much easier to grow if you use stripe of mirrors instead of raidz. You waste more space, but gain more flexibility and performance. More details here: http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-... For example, you can start with 1 mirrored vdev, then add another mirrored vdev. You can upgrade vdevs separately, so you'll need to replace just 2 disks to grow your pool. The only thing you have to keep in mind is that data is striped across vdevs only when written, so if you add another vdev, you won't get performance gains for data which was written just to one vdev.
- tscs37 9y ago
  It's also important to keep in mind that a stripe of mirrors, aka RAID10, has a higher failure probability compared to RAIDZ2 or higher on most common disk setups by several orders of magnitude.
  - rleigh 9y ago
    You also have to factor in the speed of a resilver and account for the chance of a failure during this at risk window. A mirror can resilver much faster, and it doesn't significantly affect the vdev performance while doing so. A RAIDZ resilver after a disc replacement can take a significant amount of time, and degrade performance seriously as it thrashes every single disc in the vdev. Allan Jude and Michael Lucas' books on ZFS have tables describing the tradeoffs of the different possible vdev layouts, and they are worth a read for anyone setting up ZFS storage.
    - tscs37 9y ago
      The danger of loosing data on a RAID10 resilver is much higher than people expect. A 6x8TB RAID10 has a 47.3% chance of encountering a URE during a rebuild, which means data loss. A 6x8TB RAID6 has a 0.0002% chance of a URE. (Assuming URE rate of 10^-14, in reality this rate is lower)
lmm 9y ago
I have a pool of two raidz2s of four disks each, one twice the size of the other; every so often I replace one set of disks with disks that are 4x the size (i.e. I started with 4x250gb + 4x500gb, after a few years I replaced the 250gb disks with 1tb disks, right now I have 4xt2b and 4x4tb - and I get to store half as much data as the total capacity, so 12tb at the moment). If a disk dies close to the time I was thinking of upgrading then I'll replace that disk with one of the "new" size (but can't use the extra space until I do the rest of the replacement). It works pretty well - by the time I'm buying disks that are 4x as large I don't mind throwing the old disks away. I've definitely avoided data loss in scenarios where I'd previously lost data under linux md (which lacks checksums and handles disks with isolated UREs very poorly).
luca_ing 9y ago
> Can anyone who has been using ZFS long-term at home comment? How do you add more space? My solution is to use RAID1 exclusively. That means that I can keep attaching pairs of devices if I run out of diskspace. I can never get them out again, however :-)
arwineap 9y ago
Whenever I run out of space, I hop on newegg, and search for drives twice my current size. I buy the first one I find immediately. Once I get it, I hot swap a drive, and start searching for sales on more drives. Rinse, repeat, all 5 disks swapped, more space appears \o/

Annatar 9y ago

On ZFS, a pool is resized by replacing all the drives in the pool. When autoexpand=on and the last drive is replaced, the pool will have expanded capacity. And drives are cheap. When another RAID device is added to the pool instead, it is used as a stripe. Different types of RAID can be combined in this way, for example one can add a mirrored stripe to an already existing RAID-Z. Whether this is desired or not is a different discussion, since the point here is that both scenarios are possible. Unlike other volume managers, ZFS expands on the disk boundary, rather than physical or logical elements; it's a larger boundary, but since no arcane or complex procedures or knowledge are required and drives are cheap, it's very practical, as well as elegant.

vpeters25 9y ago

> The only issue that irks me is that I can't resize a pool after it was created, can't add new disks, can't remove one (without risking parity). Not sure why ZFS would not auto resize for you. It's the reason I have been using it for several years as my home NAS under a linux server. In my case I just replace all the disk on a RAIDZ with bigger ones and automatically resizes up.

scottLobster 9y ago
Which forces you to waste $/GB and $$ in general on multiple higher capacity disks as opposed to just buying a single new drive. Not that my 10 TB RAIDZ is running out of space any time soon, but as soon as Btrfs gets their shit together I'm switching. At the rate of a few blu rays a year it'll fill up sooner or later.

FractalNerve 9y ago

Can't you just use ZFS without RAIDZ and still have your data protected from corruption/drive failures? I think storage is hard and I never understood the advantage of RAID (at least for home usage). It really only looks like an inflexible option, with too much risk to me. What's the benefit of RAIDZ over say, you choose to have X copies distributed over your disk(s)? Answer: zfs: copies=n is not a substitute for device redundancy! source: http://jrs-s.net/2016/05/02/zfs-copies-equals-n/ http://jrs-s.net/2016/05/02/zfs-copies-equals-n/ Here's a discussion about it: https://www.reddit.com/r/DataHoarder/comments/4hbn8v/raidz_vs_copiesn/ https://www.reddit.com/r/DataHoarder/comments/4hbn8v/raidz_v... Anyone who wants to have real security, relies on off-site backups, isn't that right? And aren't RAID(Z)s slow to recover also? (serious questions, I'm a zfs noob) RAIDZ: I don't know what stripe-set configuration is good for me and don't want to waste time comparing RAID controllers or if I even need one. Then configuring the beast on the hardware and software side just seems to be too tedious. Why not just (de)/attach another disk and let zfs expand/shrink my total disk space without loosing consistency? Startup idea: Someone clever should find a flexible storage solution that uses aufs, unionfs etc. to give you the flexibility we need.