5 ms·

TL;DR: Btrfs is usable in mirror mode. Raid is still unusable (as in: you will lose data) Usage and tooling are unintuitive and dangerous. I have to say, it

TL;DR: Btrfs is usable in mirror mode. Raid is still unusable (as in: you will lose data) Usage and tooling are unintuitive and dangerous. I have to say, it doesn't sound so bad when you know what you're doing. The example from the article - the fact that you need to re-silver and re-balance a mirror after a failure - yes that is dangerous. But then again, when you're starting with zero ZFS knowledge, you also need to learn the particular way it's designed. And let's remember that some mdadm raids don't auto-scrub periodically. So the essence is: Storage is still hard, it's still necessary to think before you manage disks.

creshal 5y ago
> But then again, when you're starting with zero ZFS knowledge, you also need to learn the particular way it's designed. Except… there's nothing to really learn. All these weird edge cases that can kill your BTRFS simply don't exist in ZFS. The tooling is also much more polished and in the few cases that something does require user attention, it'll tell you in easily understandable terms, so as a new user the only command you really need to learn is `zpool status`, which you now did.
- WastingMyTime89 5y ago
  > Except… there's nothing to really learn. Would you mind explaining to us how to add a disk to an array with ZFS? Then same question but with a disk which is not the same size that the rest of the array.
  - uniqueuid 5y ago
    You're teasing, of course, but it shows the real problems. To take you up on that with a dangerous example that reinforces the point: To add a disk to an array whose used space is less than the capacity of a single disk, you can (DON'T DO THIS, IT'S DANGEROUS) (1) remove a disk from the array (it's degraded now) (2) reformat the single disk, put all data on there (3) create a sparse file with the size of one disk (4) create a new array with N disks, in place of the temporary disk use the sparse file as a device (5) remove the sparse file (the new array is now degraded) (6) bring the new array up, copy all data from temp disk on it (7) replace the sparse file with the temp disk (8) copy files around to make them re-balance Congratulations, you've now grown your array, and put your data seriously at risk in the process! :)
    - mavhc 5y ago
      How do you do it on another FS?
      - uniqueuid 5y ago
        mdadm --add /dev/md1 DEVICE mdadm --grow /dev/md1 --size=max
        mavhc 5y ago
        My friend did that, lost all his data, now uses ZFS
  - xoa 5y ago
    To be clear, I'm assuming by "array" you mean "vdev" not the overall pool, since adding new disks to a pool has always been the normal way to grow a pool. PR was opened by Matthew Ahrens (one of the OpenZFS founding devs) back in June for RAID-Z expansion if you're actually curious. It almost certainly won't hit any stable major releases until next year while everyone kicks the tires, though should be noted ixSystems has a history of pulling new ZFS features into TrueNAS early (they might be more cautious to be ultra careful about this one though). But working code is there, it just adds another capability to the existing [zpool attach] command. FWIW I honestly don't think growing an array vs adding to a pool is actually that interesting given the modern state of drives and how capacity growth has lined up with data growth across the spectrum. At the consumer/soho/small business scale growth curves for storage blew past data needs a while ago, so by the time somebody needs more storage the drives are old and it makes more sense to just replace them with much bigger ones (which ZFS supports fine, replace all the drives in an array one by one then it'll expand), or small drives will be so much cheaper that just adding another set to the pool makes sense. Much bigger then that and people aren't going to be doing array expansion ever anyway. Still, cool to see that one checked off the list, and if the functionality ultimately enables generalization to RAID-Z type conversion or better rebalancing that'd be very interesting. >Then same question but with a disk which is not the same size that the rest of the array. Don't do that simple as that? There isn't any great way to handle that with full performance and redundancy, and any possible value seems way way too niche to be worth bothering with. Although again if you want to add uneven vdevs (single or multidisks) to a pool you can always do that if you want. Edit: One other practical consideration I've always had regarding expansion, adding disks and so on, is that the troublesome brute force method to me is an excuse to promote good habits and practices most of us are very lazy about in general. Having to tear down a pool and restore to get best performance and flexibility out of adding a bunch of disks is certainly a pain. But we should be able to do this at any time anyway, we should be semi-regularly testing that it works, and we do all do that testing right? Right? OK probably not, plenty of people aren't in the habit which is why automation is so important. But even if ZFS gained full perfect flexibility, I still think I wouldn't use it in general because I think the exercise of making sure that yes I can do a full restore and everything is working at least once every few years is so valuable.
  - bluGill 5y ago
    You don't. There is NO real world use case to do that. It seems like they exist, but every time you dig into the details you discover that there is a horrible edge case that makes it a stupid idea. What you do is buy not one drive, but a set of drives and add an second array to the system. Then RAID-0 between the two arrays.
    - Dylan16807 5y ago
      There is NO real world use case to buy fewer than six to eight drives at a time? (assuming I'm using raidz2 or so) Really?
      - bluGill 5y ago
        The smallest number you can safely go is 3 at a time - put them in a mirror configuration. This is expensive for the amount of storage you get, so few people are willing to spend that much. Once you want to be cost effective 6-8 is the sweet spot - enough drives for a good amount of usable space, but not so many that multiple drive failures are likely to compromise the array. Nobody is stopping you from other configurations, but 6-8 drives in your array is about the best compromise between performance, cost effective and total storage.
        Dylan16807 5y ago
        I want to work up to 8 drives in a vdev as my storage needs grow, because that's cost-effective. But I don't want to spend 8 drives of money up front, because it's a lot and because drive prices drop over time. Am I not a real use case?
        bluGill 5y ago
        For the first 8 drives, but after that no. BRTFS-raid2 seems to be good for up to 8 drives if that is your use case. But there is no easy upgrade path to that 9th drive. You should not be running less than 3 drives for any data you care about. Fortunately, most people don't have data they care about. Their photos and documents are on cloud providers like google/dropbox/facebook (and dozens of others I don't even know of). If disaster happens they might lose a couple days work. There is a lot to not like about cloud providers, but they are probably better than what the average person does - no backups at all. Note that the RAID system I've described here is for data integrity, but there are still house burning down cases where you lose it all,
  - mindslight 5y ago
    zpool add tank mirror /dev/sdc1 /dev/sdd1 But I presume you're referring to adding disks to an existing raidz. Such an operation was never really considered by ZFS, as it was made for business environments where they buy disks in quantity. So they buy enough disks for a full redundancy set (another mirror or raidz), and append onto the existing array. From the perspective of an individual user it's definitely a suboptimal part of ZFS. But standing in the shadow of business usage is the price to pay for it having been developed in the first place. As an individual user you can play games with sparse files and failed members as a sibling comment says (and dragging some external drives into service can help, as well as restoring from your backups). But really, just try to plan for the space you want and buy at once. Disks are large enough these days that you should aim to not be tinkering with your storage. Back in the day I had something like an 8 x 250GB disk external USB drive array for my FLAC collection, with each disk partitioned into 10 slices, raid6'd, and then LVM'd back together. This gave me the wiggle room to take down a single slice at a time for reshaping, and then I could even online grow the filesystem thanks to reiserfs. No backups of course. I did the online expanding thing once or twice and it was neat, but ultimately the whole thing was replaced by a simple 2TB mirror after several years. There's no reason you couldn't do something similar with ZFS besides performance. But do you really want to?
- uniqueuid 5y ago
  Sure, I give you that ZFS' tooling is much, much more polished. But of course there are peculiarities that have 'bitten' people (softly) in the past: - dedup has hardly any use case, novices get enticed to try something that doesn't do what they think it does - ZIL and SLOG sound like "a simple cache" when in reality they are much more complex - You can add single devices to a (redundant) pool, at which point you've lost redundancy, perhaps without knowing. - No defrag - No re-balance - No growing pools - ARC still surprises people (also, I've had kernel crashes under heavy load when the ARC didn't free ram fast enough ... edge case maybe, but still) That said, of course, ZFS is awesome and I encourage everybody to give it a try.
  - codys 5y ago
    Some points I think bear some clarification wrt current zfs state: - current versions of zfs (ie: openzfs) error (unless forced) when adding non-redundant devices to a redundant pool, and have device removal to handle the case where someone bypasses the error. - pools can grow in zfs (ie: the things one allocates storage out of). what is probably being referenced here is that one can't reshape a zraid vdev to add more disks to it (basically the reshape operation in classic raid setups). This does make expanding in some cases less flexible that old-style raid setups. Also, I really have to agree that ARC behavior is still not great under load, and I don't think that's that much of an edge case.
  - Dylan16807 5y ago
    - No copy-on-write file copies. - Very limited ability to make snapshots writable.

creshal 5y ago

> But then again, when you're starting with zero ZFS knowledge, you also need to learn the particular way it's designed. Except… there's nothing to really learn. All these weird edge cases that can kill your BTRFS simply don't exist in ZFS. The tooling is also much more polished and in the few cases that something does require user attention, it'll tell you in easily understandable terms, so as a new user the only command you really need to learn is `zpool status`, which you now did.

WastingMyTime89 5y ago
> Except… there's nothing to really learn. Would you mind explaining to us how to add a disk to an array with ZFS? Then same question but with a disk which is not the same size that the rest of the array.
- uniqueuid 5y ago
  You're teasing, of course, but it shows the real problems. To take you up on that with a dangerous example that reinforces the point: To add a disk to an array whose used space is less than the capacity of a single disk, you can (DON'T DO THIS, IT'S DANGEROUS) (1) remove a disk from the array (it's degraded now) (2) reformat the single disk, put all data on there (3) create a sparse file with the size of one disk (4) create a new array with N disks, in place of the temporary disk use the sparse file as a device (5) remove the sparse file (the new array is now degraded) (6) bring the new array up, copy all data from temp disk on it (7) replace the sparse file with the temp disk (8) copy files around to make them re-balance Congratulations, you've now grown your array, and put your data seriously at risk in the process! :)
  - mavhc 5y ago
    How do you do it on another FS?
    - uniqueuid 5y ago
      mdadm --add /dev/md1 DEVICE mdadm --grow /dev/md1 --size=max
      - mavhc 5y ago
        My friend did that, lost all his data, now uses ZFS
- xoa 5y ago
  To be clear, I'm assuming by "array" you mean "vdev" not the overall pool, since adding new disks to a pool has always been the normal way to grow a pool. PR was opened by Matthew Ahrens (one of the OpenZFS founding devs) back in June for RAID-Z expansion if you're actually curious. It almost certainly won't hit any stable major releases until next year while everyone kicks the tires, though should be noted ixSystems has a history of pulling new ZFS features into TrueNAS early (they might be more cautious to be ultra careful about this one though). But working code is there, it just adds another capability to the existing [zpool attach] command. FWIW I honestly don't think growing an array vs adding to a pool is actually that interesting given the modern state of drives and how capacity growth has lined up with data growth across the spectrum. At the consumer/soho/small business scale growth curves for storage blew past data needs a while ago, so by the time somebody needs more storage the drives are old and it makes more sense to just replace them with much bigger ones (which ZFS supports fine, replace all the drives in an array one by one then it'll expand), or small drives will be so much cheaper that just adding another set to the pool makes sense. Much bigger then that and people aren't going to be doing array expansion ever anyway. Still, cool to see that one checked off the list, and if the functionality ultimately enables generalization to RAID-Z type conversion or better rebalancing that'd be very interesting. >Then same question but with a disk which is not the same size that the rest of the array. Don't do that simple as that? There isn't any great way to handle that with full performance and redundancy, and any possible value seems way way too niche to be worth bothering with. Although again if you want to add uneven vdevs (single or multidisks) to a pool you can always do that if you want. Edit: One other practical consideration I've always had regarding expansion, adding disks and so on, is that the troublesome brute force method to me is an excuse to promote good habits and practices most of us are very lazy about in general. Having to tear down a pool and restore to get best performance and flexibility out of adding a bunch of disks is certainly a pain. But we should be able to do this at any time anyway, we should be semi-regularly testing that it works, and we do all do that testing right? Right? OK probably not, plenty of people aren't in the habit which is why automation is so important. But even if ZFS gained full perfect flexibility, I still think I wouldn't use it in general because I think the exercise of making sure that yes I can do a full restore and everything is working at least once every few years is so valuable.
- bluGill 5y ago
  You don't. There is NO real world use case to do that. It seems like they exist, but every time you dig into the details you discover that there is a horrible edge case that makes it a stupid idea. What you do is buy not one drive, but a set of drives and add an second array to the system. Then RAID-0 between the two arrays.
  - Dylan16807 5y ago
    There is NO real world use case to buy fewer than six to eight drives at a time? (assuming I'm using raidz2 or so) Really?
    - bluGill 5y ago
      The smallest number you can safely go is 3 at a time - put them in a mirror configuration. This is expensive for the amount of storage you get, so few people are willing to spend that much. Once you want to be cost effective 6-8 is the sweet spot - enough drives for a good amount of usable space, but not so many that multiple drive failures are likely to compromise the array. Nobody is stopping you from other configurations, but 6-8 drives in your array is about the best compromise between performance, cost effective and total storage.
      - Dylan16807 5y ago
        I want to work up to 8 drives in a vdev as my storage needs grow, because that's cost-effective. But I don't want to spend 8 drives of money up front, because it's a lot and because drive prices drop over time. Am I not a real use case?
        bluGill 5y ago
        For the first 8 drives, but after that no. BRTFS-raid2 seems to be good for up to 8 drives if that is your use case. But there is no easy upgrade path to that 9th drive. You should not be running less than 3 drives for any data you care about. Fortunately, most people don't have data they care about. Their photos and documents are on cloud providers like google/dropbox/facebook (and dozens of others I don't even know of). If disaster happens they might lose a couple days work. There is a lot to not like about cloud providers, but they are probably better than what the average person does - no backups at all. Note that the RAID system I've described here is for data integrity, but there are still house burning down cases where you lose it all,
- mindslight 5y ago
  zpool add tank mirror /dev/sdc1 /dev/sdd1 But I presume you're referring to adding disks to an existing raidz. Such an operation was never really considered by ZFS, as it was made for business environments where they buy disks in quantity. So they buy enough disks for a full redundancy set (another mirror or raidz), and append onto the existing array. From the perspective of an individual user it's definitely a suboptimal part of ZFS. But standing in the shadow of business usage is the price to pay for it having been developed in the first place. As an individual user you can play games with sparse files and failed members as a sibling comment says (and dragging some external drives into service can help, as well as restoring from your backups). But really, just try to plan for the space you want and buy at once. Disks are large enough these days that you should aim to not be tinkering with your storage. Back in the day I had something like an 8 x 250GB disk external USB drive array for my FLAC collection, with each disk partitioned into 10 slices, raid6'd, and then LVM'd back together. This gave me the wiggle room to take down a single slice at a time for reshaping, and then I could even online grow the filesystem thanks to reiserfs. No backups of course. I did the online expanding thing once or twice and it was neat, but ultimately the whole thing was replaced by a simple 2TB mirror after several years. There's no reason you couldn't do something similar with ZFS besides performance. But do you really want to?
uniqueuid 5y ago
Sure, I give you that ZFS' tooling is much, much more polished. But of course there are peculiarities that have 'bitten' people (softly) in the past: - dedup has hardly any use case, novices get enticed to try something that doesn't do what they think it does - ZIL and SLOG sound like "a simple cache" when in reality they are much more complex - You can add single devices to a (redundant) pool, at which point you've lost redundancy, perhaps without knowing. - No defrag - No re-balance - No growing pools - ARC still surprises people (also, I've had kernel crashes under heavy load when the ARC didn't free ram fast enough ... edge case maybe, but still) That said, of course, ZFS is awesome and I encourage everybody to give it a try.
- codys 5y ago
  Some points I think bear some clarification wrt current zfs state: - current versions of zfs (ie: openzfs) error (unless forced) when adding non-redundant devices to a redundant pool, and have device removal to handle the case where someone bypasses the error. - pools can grow in zfs (ie: the things one allocates storage out of). what is probably being referenced here is that one can't reshape a zraid vdev to add more disks to it (basically the reshape operation in classic raid setups). This does make expanding in some cases less flexible that old-style raid setups. Also, I really have to agree that ARC behavior is still not great under load, and I don't think that's that much of an edge case.
- Dylan16807 5y ago
  - No copy-on-write file copies. - Very limited ability to make snapshots writable.