9 ms·

"Once you build a ZFS volume, it’s pretty much fixed for life." The ease of growing/shrinking existing volumes and adding/removing storage is why I made the de

"Once you build a ZFS volume, it’s pretty much fixed for life." The ease of growing/shrinking existing volumes and adding/removing storage is why I made the decision to go with btrfs when I rebuilt my home file server.

ThatPlayer 9y ago
I'm also using Btrfs for my home file server, but Btrfs's current RAID5/6 status leaves a lot to be desired. I'm running Btrfs on top of mdadm right now because of that.
- sheepdestroyer 9y ago
  Should be "mostly" resolved from kernel 4.12 : https://www.phoronix.com/scan.php?page=news_item&px=Linux-4.12-Btrfs-RAID-Mostly-OK https://www.phoronix.com/scan.php?page=news_item&px=Linux-4....
  - pantalaimon 9y ago
    It used to be "usable for most purposes" at some point already, before those scary data corruption bugs were discovered. https://btrfs.wiki.kernel.org/index.php?title=RAID56&diff=30445&oldid=29769 https://btrfs.wiki.kernel.org/index.php?title=RAID56&diff=30...
- alyandon 9y ago
  My last file server used mdadm raid 5 across 4 drives and I just couldn't take the write performance loss anymore so I switched to btrfs raid1(ish) block duplication.
phil21 9y ago
This is only difficult with ZFS if you care about performance. If you are a typical home file server user, you can add vdevs in a rather ad-hoc fashion as they fill and it works pretty well. There is a huge performance penalty of course, as all the majority of new data will reside on the latest vdevs - but it generally works. I do agree this is one of the largest drawbacks of ZFS, but very few filesystems get it right.
- eriknstr 9y ago
  What I'd like to know is, can this issue be resolved in a future version of ZFS or is it too ingrained into the design of ZFS?
  - cbhl 9y ago
    It would be expensive -- you'd have to "rebalance" the data by copying it to the new volume / away from the old volume, which would take hours or days.
    - rocqua 9y ago
      People who want this feature are mostly frugal. They want it so they can upgrade by just buying a single new disk. That is a rare enough use case that the performance penalty would be worth it.
      - cbhl 9y ago
        Frugal enough to shut off their computer when they're not using it (to "save power"), so the rebalancing won't have time to complete in the background?
    - kalleboo 9y ago
      Isn't the fact that Drobo is still around and doing well a testament to people being fine with this tradeoff?
  - wtallis 9y ago
    For years the fix ("block pointer rewrite" feature) was promised as coming eventually, but that effort was abandoned. BTRFS will reach ZFS levels of stability before ZFS reaches BTRFS levels of flexibility.
- wmf 9y ago
  you can add vdevs in a rather ad-hoc fashion as they fill and it works pretty well As long as they're expensive mirror vdevs, right? My impression is that home users want the efficiency of RAID-6 and they want incremental expansion (regardless of whether this combination is "good for them"). ZFS can't do that.
bitL 9y ago
I hope you managed to use the fixed BTRFS unlike me that built home NAS with tens of terrabytes on top of BTRFS and then year later learned that the version I have was broken and the fix is not applicable to a running cluster... So I have to buy another batch of HDDs, install fixed BTRFS (who knows what other unknown issue awaits me in the darkness?) and copy over everything...
problems 9y ago
Btrfs is a mess too, especially with its metadata system that you have to manually resize constantly, data-loss bugs and fsck problems. I've had nothing but problems with it and will never use it on a production system. LVM on the other hand makes it easy to do snapshotting, RAID, volume resizing, adding more drives, etc - at the cost of some performance in some cases. For the most part, it's the best trade off available if expandability is a requirement and you can't get away with simple mdraid.
- cmurf 9y ago
  >its metadata system that you have to manually resize constantly ? Doesn't ring a bell. A lot of problems with Btrfs sounds like hardware problems of one sort or another. If it's a legit bug the only way such things get fixed is to report it to the developers <linux-btrfs@vger.kernel.org> with complete logs and system information. Did you? The fsck is definitely hit or miss, but my perspective is the emphasis is on fixing bugs that obviate file system problems in the first place. The reality is an offline fsck for large file systems is just not scalable, so the best bang for the buck is bug squashing. And there's a tons of that happening. For the initial 4.12 pull (now done, and probably had a few dozen changes during rc's) 40 files changed, 1629 insertions(+), 834 deletions(-) https://lkml.org/lkml/2017/5/9/510 https://lkml.org/lkml/2017/5/9/510 For the initial 4.13 pull 47 files changed, 1707 insertions(+), 1400 deletions(-) https://lkml.org/lkml/2017/7/4/436 https://lkml.org/lkml/2017/7/4/436
  - rrauenza 9y ago
    He may be talking about the need to rebalance ... I run a btrfs balance every month to make sure I don't run out of space again. It happened to me while on vacation -- I only had satellite internet and had to fix it over ssh at single characters per second. It might have to do with a bug I saw mentioned on the list that referred to the free space map getting corrupted (but easily fixed with a -oremount,clear_cache). So I also run that monthly as well. I've been too afraid to remove the cron job even though I moved my CentOS7 to the "mainline" kernel rpms from http://elrepo.org http://elrepo.org Also, I only trust RAID1 + Crashplan.
- 9y ago
ioquatix 9y ago
This advice isn't actually totally correct. If you build a volume, you can add more zdevs to it. My suggestion is to avoid RAIDZx and go with mirrors. To add more storage, just add another pair of drives and add it to your pool. Here is what my main `tank` looks like: pool: tank state: ONLINE scan: scrub repaired 0B in 4h59m with 0 errors on Tue Jul 4 12:47:02 2017 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 wwn-0x50014ee20eba1695 ONLINE 0 0 0 wwn-0x50014ee20eba337e ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 wwn-0x50014ee2640efb3a ONLINE 0 0 0 wwn-0x50014ee2b964ef3f ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 wwn-0x50014ee2b964f2d5 ONLINE 0 0 0 wwn-0x50014ee2b964f4f4 ONLINE 0 0 0 cache wwn-0x5002538d704e9ff1-part4 ONLINE 0 0 0 errors: No known data errors You can simply add another mirrored pair and get more storage. Or, you can update an existing mirror by adding new drives, then removing the old ones. Mirroring has better performance and IOPs than striped raid too. If you feel that mirrors might be less reliable than RAIDZx, I'd suggest you read up about it - generally mirrors are more reliable (and you could mirror 3 drives if you liked, get more throughput and IOPs - or have a hot spare).
- gaadd33 9y ago
  As you add mirrored pairs your data isn't evenly distributed across all stripes is it?
  - ioquatix 9y ago
    The simple answer to your question is no. The more complex answer is that over time as you write more data, ZFS will re-balance the distribution of data over the available zdevs.
    - takeda 9y ago
      I think the concern is that if you have two disks die it is not a big deal, unless those two disk happen to be of the same pair[1], then most likely all data is lost. [1] if you're not careful, this is actually very likely to happen. If you purchase 2 disks at the same time, and they are under the same usage patterns (which they will be when working in a pair), under the same temperature then they are very likely to die at the same time.

ThatPlayer 9y ago

I'm also using Btrfs for my home file server, but Btrfs's current RAID5/6 status leaves a lot to be desired. I'm running Btrfs on top of mdadm right now because of that.

sheepdestroyer 9y ago
Should be "mostly" resolved from kernel 4.12 : https://www.phoronix.com/scan.php?page=news_item&px=Linux-4.12-Btrfs-RAID-Mostly-OK https://www.phoronix.com/scan.php?page=news_item&px=Linux-4....
- pantalaimon 9y ago
  It used to be "usable for most purposes" at some point already, before those scary data corruption bugs were discovered. https://btrfs.wiki.kernel.org/index.php?title=RAID56&diff=30445&oldid=29769 https://btrfs.wiki.kernel.org/index.php?title=RAID56&diff=30...
alyandon 9y ago
My last file server used mdadm raid 5 across 4 drives and I just couldn't take the write performance loss anymore so I switched to btrfs raid1(ish) block duplication.

phil21 9y ago

This is only difficult with ZFS if you care about performance. If you are a typical home file server user, you can add vdevs in a rather ad-hoc fashion as they fill and it works pretty well. There is a huge performance penalty of course, as all the majority of new data will reside on the latest vdevs - but it generally works. I do agree this is one of the largest drawbacks of ZFS, but very few filesystems get it right.

eriknstr 9y ago
What I'd like to know is, can this issue be resolved in a future version of ZFS or is it too ingrained into the design of ZFS?
- cbhl 9y ago
  It would be expensive -- you'd have to "rebalance" the data by copying it to the new volume / away from the old volume, which would take hours or days.
  - rocqua 9y ago
    People who want this feature are mostly frugal. They want it so they can upgrade by just buying a single new disk. That is a rare enough use case that the performance penalty would be worth it.
    - cbhl 9y ago
      Frugal enough to shut off their computer when they're not using it (to "save power"), so the rebalancing won't have time to complete in the background?
  - kalleboo 9y ago
    Isn't the fact that Drobo is still around and doing well a testament to people being fine with this tradeoff?
- wtallis 9y ago
  For years the fix ("block pointer rewrite" feature) was promised as coming eventually, but that effort was abandoned. BTRFS will reach ZFS levels of stability before ZFS reaches BTRFS levels of flexibility.
wmf 9y ago
you can add vdevs in a rather ad-hoc fashion as they fill and it works pretty well As long as they're expensive mirror vdevs, right? My impression is that home users want the efficiency of RAID-6 and they want incremental expansion (regardless of whether this combination is "good for them"). ZFS can't do that.

bitL 9y ago

I hope you managed to use the fixed BTRFS unlike me that built home NAS with tens of terrabytes on top of BTRFS and then year later learned that the version I have was broken and the fix is not applicable to a running cluster... So I have to buy another batch of HDDs, install fixed BTRFS (who knows what other unknown issue awaits me in the darkness?) and copy over everything...

problems 9y ago

Btrfs is a mess too, especially with its metadata system that you have to manually resize constantly, data-loss bugs and fsck problems. I've had nothing but problems with it and will never use it on a production system. LVM on the other hand makes it easy to do snapshotting, RAID, volume resizing, adding more drives, etc - at the cost of some performance in some cases. For the most part, it's the best trade off available if expandability is a requirement and you can't get away with simple mdraid.

cmurf 9y ago
>its metadata system that you have to manually resize constantly ? Doesn't ring a bell. A lot of problems with Btrfs sounds like hardware problems of one sort or another. If it's a legit bug the only way such things get fixed is to report it to the developers <linux-btrfs@vger.kernel.org> with complete logs and system information. Did you? The fsck is definitely hit or miss, but my perspective is the emphasis is on fixing bugs that obviate file system problems in the first place. The reality is an offline fsck for large file systems is just not scalable, so the best bang for the buck is bug squashing. And there's a tons of that happening. For the initial 4.12 pull (now done, and probably had a few dozen changes during rc's) 40 files changed, 1629 insertions(+), 834 deletions(-) https://lkml.org/lkml/2017/5/9/510 https://lkml.org/lkml/2017/5/9/510 For the initial 4.13 pull 47 files changed, 1707 insertions(+), 1400 deletions(-) https://lkml.org/lkml/2017/7/4/436 https://lkml.org/lkml/2017/7/4/436
- rrauenza 9y ago
  He may be talking about the need to rebalance ... I run a btrfs balance every month to make sure I don't run out of space again. It happened to me while on vacation -- I only had satellite internet and had to fix it over ssh at single characters per second. It might have to do with a bug I saw mentioned on the list that referred to the free space map getting corrupted (but easily fixed with a -oremount,clear_cache). So I also run that monthly as well. I've been too afraid to remove the cron job even though I moved my CentOS7 to the "mainline" kernel rpms from http://elrepo.org http://elrepo.org Also, I only trust RAID1 + Crashplan.
9y ago

ioquatix 9y ago

This advice isn't actually totally correct. If you build a volume, you can add more zdevs to it. My suggestion is to avoid RAIDZx and go with mirrors. To add more storage, just add another pair of drives and add it to your pool. Here is what my main `tank` looks like: pool: tank state: ONLINE scan: scrub repaired 0B in 4h59m with 0 errors on Tue Jul 4 12:47:02 2017 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 wwn-0x50014ee20eba1695 ONLINE 0 0 0 wwn-0x50014ee20eba337e ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 wwn-0x50014ee2640efb3a ONLINE 0 0 0 wwn-0x50014ee2b964ef3f ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 wwn-0x50014ee2b964f2d5 ONLINE 0 0 0 wwn-0x50014ee2b964f4f4 ONLINE 0 0 0 cache wwn-0x5002538d704e9ff1-part4 ONLINE 0 0 0 errors: No known data errors You can simply add another mirrored pair and get more storage. Or, you can update an existing mirror by adding new drives, then removing the old ones. Mirroring has better performance and IOPs than striped raid too. If you feel that mirrors might be less reliable than RAIDZx, I'd suggest you read up about it - generally mirrors are more reliable (and you could mirror 3 drives if you liked, get more throughput and IOPs - or have a hot spare).

gaadd33 9y ago
As you add mirrored pairs your data isn't evenly distributed across all stripes is it?
- ioquatix 9y ago
  The simple answer to your question is no. The more complex answer is that over time as you write more data, ZFS will re-balance the distribution of data over the available zdevs.
  - takeda 9y ago
    I think the concern is that if you have two disks die it is not a big deal, unless those two disk happen to be of the same pair[1], then most likely all data is lost. [1] if you're not careful, this is actually very likely to happen. If you purchase 2 disks at the same time, and they are under the same usage patterns (which they will be when working in a pair), under the same temperature then they are very likely to die at the same time.