6 ms·

I keep seeing Btrfs improvements in each new kernel release but never anything about them fixing RAID 5/6. I was able to pickup multiple 20TB hard drives over B

I keep seeing Btrfs improvements in each new kernel release but never anything about them fixing RAID 5/6. I was able to pickup multiple 20TB hard drives over Black Friday but now I've got to figure out how to configure redundancy with Btrfs optimally or try out FreeNAS, or go the easy route with Synology. I'm leaning towards FreeNAS.

adql 4y ago
I live with 6 disk NAS(+ some VMs and services like Grafana) with part-RAID6 (6 drives, bulk storage) part-RAID10 (5+1, VMs and other stuff needing a bit more bw) and it has been entirely fine. But FreeNAS is perfect alternative if you don't want to fuck with penguin stuff and just have a blob of storage
shaicoleman 4y ago
"The 'write hole' bug, only affects metadata not data. Thus if you don't make metadata RAID 5/6, you avoid that problem. Fortunately BTRFS allows you to have different RAID levels for data and metadata so you could run RAID 1 for metadata and RAID 5/6 for data." https://www.reddit.com/r/linux/comments/yrlljv/comment/ivvopaj/ https://www.reddit.com/r/linux/comments/yrlljv/comment/ivvop...
- encryptluks2 4y ago
  Thanks... I'm going to look more into this and see if it is an option.
flas9sd 4y ago
there's ongoing work on fixing this by Johannes Thumshirn, see https://old.reddit.com/r/btrfs/comments/wcxfqs/btrfs_declustered_parity_raid_for_zoned_devices/ https://old.reddit.com/r/btrfs/comments/wcxfqs/btrfs_declust...
- flas9sd 4y ago
  people specifically looking for places where the read/write hole on raid56 is addressed, the problem is stated at minute 9 and solution of a new "raid-stripe-tree" presented from min 14 and on
PlutoIsAPlanet 4y ago
You could just keep to RAID 1. btrfs raid is at filesystem level, so RAID 1 with more than 2 drives just means 2 copies spread across those drives, so you benefit from the increased storage space like RAID 5 (albeit slightly less) but of course unlike normal RAID 1 still have 1 drive fault tolerance across the entire array.
gigatexal 4y ago
Or just use a filesystem with working Raid-5/6 code like ZFS. How Btrfs can keep shipping broken code and get a pass is beyond me. Here's to hoping Bcachefs ever gets merged into the kernel.
Geezus-42 4y ago
You could just use ZFS. I like butter, and maybe keeping your Metadata on a different raid level helps, but it torched my aray a couple times so I switched and have had no issues.
- encryptluks2 4y ago
  Yeah, I think I'm leaning in that direction but I may look into Btrfs with metadata on a different level. I've had good luck with Btrfs as my desktop filesystem.
- bravetraveler 4y ago
  Here to also suggest alternatives BTRFS is the only 'RAID' that I've encountered that will almost predictably fail if forcefully powered off RAID10 on gen4 NVMe drives, should sync pretty quick, you'd think. LVM RAID, ZFS, mdadm, all of them are considerably more reliable I can hear lamenting already: "that's not normal! you shouldn't expect consistency here!" I tested this because reality says we do unusual things all of the time, don't hate me - hate the results. Other implementations are demonstrably more robust
  - rektide 4y ago
    This is a you problem, that no one else reproduces, and which you have reported only here. It's degrading to the conversation to have this kind of casual slander. Btrfs raid is stupid reliable, if you dont f it up & dont put metadata in raid5/6. Having checksum consistency & checkpoints is a superpower most of the alternatives dont have. Btrfs is much more flexible about adding/removing drives, of various sizes even, to a raid pool, as compared to zfs, & is imo ridiculously easier to operate, from not needing out of tree modules to just requiring a lot less specialist knowledge in general of different zfs caches for tuning/setup. There's no alternative close & it runs great across millions of systems, with operators such as Facebook/Meta.
    - bravetraveler 4y ago
      Everyone in production reproduces this. Intentionally or not, everything on a long enough timeline will suddenly lose power. I'm sorry to offend what is otherwise a decent filesystem, but I, and the systems I'm responsible for, will not use BTRFS with RAID and this characteristic. There's a lot of qualification there, read it again. I would much rather let the journal replay and do its thing, than rebuild the array. Something every alternative offers, consistently. BTRFS RAID has routinely burned me every time I have given it the opportunity. It hasn't survived nearly as much as the other test arrays. I'm fine discussing filesystems, benefits, and all of that - but I can't entertain it when you try to make this so heated. These are anecdotes from reliability tests I've been doing... I'm sharing them so that they can be considered. You accuse me, while opening like this? At least my anecdotes weren't insulting. edit: To add, this isn't something I've kept secret. I don't know why you say that. Have you read every communication I've written? I can show you screenshots that span the last three months discussing this exact situation I haven't reported it upstream because I have better things to do; ship. You might see this, but hopes aren't high. This is about the response I expected. It's an extreme edge case, but it consistently cuts. The filesystem we use is incredibly uninteresting. By design. I find it utterly hilarious you think I'm the only person who has had issues with BTRFS RAID
moloch-hai 4y ago
RAID 5 and 6 are obsolete anyway, a relic of times when 2x storage cost too much.
- willnonya 4y ago
  That just seems like an absurd statement to me. It ignores so many use cases and financial realities.
- encryptluks2 4y ago
  2x storage is still expensive for home NAS use. RAID 5/6 has other benefits too over RAID 10. Maybe RAID 5/6 isn't the right choice, but it seemed like the best choice when I was comparing options.
  - moloch-hai 4y ago
    The time it takes to initialize RAID 5 or 6, or to swap out a drive, was manageable when the drives were smaller. Now, the benefit of fewer drives is swamped by the extremely long time needed before you can do anything with them.
anonymousDan 4y ago
As someone who is looking to purchase their first NAS any advice on keeping the energy costs down (e.g. does particular NAS hardware or hard drives have running cost advantages in this area)? My main use case (at least initially) will be for storing and backing up photos for infrequent access.
- encryptluks2 4y ago
  I would assume most 7200 RPM drives have similar energy requirements, although it does appear some drives get hotter than others so efficient cooling might increase those requirements by a bit. Cost of storage though can be a bit expensive. I was able to find 20TB CMR drives for around $329 each.

adql 4y ago

I live with 6 disk NAS(+ some VMs and services like Grafana) with part-RAID6 (6 drives, bulk storage) part-RAID10 (5+1, VMs and other stuff needing a bit more bw) and it has been entirely fine. But FreeNAS is perfect alternative if you don't want to fuck with penguin stuff and just have a blob of storage

shaicoleman 4y ago

"The 'write hole' bug, only affects metadata not data. Thus if you don't make metadata RAID 5/6, you avoid that problem. Fortunately BTRFS allows you to have different RAID levels for data and metadata so you could run RAID 1 for metadata and RAID 5/6 for data." https://www.reddit.com/r/linux/comments/yrlljv/comment/ivvopaj/ https://www.reddit.com/r/linux/comments/yrlljv/comment/ivvop...

encryptluks2 4y ago
Thanks... I'm going to look more into this and see if it is an option.

flas9sd 4y ago

there's ongoing work on fixing this by Johannes Thumshirn, see https://old.reddit.com/r/btrfs/comments/wcxfqs/btrfs_declustered_parity_raid_for_zoned_devices/ https://old.reddit.com/r/btrfs/comments/wcxfqs/btrfs_declust...

flas9sd 4y ago
people specifically looking for places where the read/write hole on raid56 is addressed, the problem is stated at minute 9 and solution of a new "raid-stripe-tree" presented from min 14 and on

PlutoIsAPlanet 4y ago

You could just keep to RAID 1. btrfs raid is at filesystem level, so RAID 1 with more than 2 drives just means 2 copies spread across those drives, so you benefit from the increased storage space like RAID 5 (albeit slightly less) but of course unlike normal RAID 1 still have 1 drive fault tolerance across the entire array.

gigatexal 4y ago

Or just use a filesystem with working Raid-5/6 code like ZFS. How Btrfs can keep shipping broken code and get a pass is beyond me. Here's to hoping Bcachefs ever gets merged into the kernel.

Geezus-42 4y ago

You could just use ZFS. I like butter, and maybe keeping your Metadata on a different raid level helps, but it torched my aray a couple times so I switched and have had no issues.

encryptluks2 4y ago
Yeah, I think I'm leaning in that direction but I may look into Btrfs with metadata on a different level. I've had good luck with Btrfs as my desktop filesystem.
bravetraveler 4y ago
Here to also suggest alternatives BTRFS is the only 'RAID' that I've encountered that will almost predictably fail if forcefully powered off RAID10 on gen4 NVMe drives, should sync pretty quick, you'd think. LVM RAID, ZFS, mdadm, all of them are considerably more reliable I can hear lamenting already: "that's not normal! you shouldn't expect consistency here!" I tested this because reality says we do unusual things all of the time, don't hate me - hate the results. Other implementations are demonstrably more robust
- rektide 4y ago
  This is a you problem, that no one else reproduces, and which you have reported only here. It's degrading to the conversation to have this kind of casual slander. Btrfs raid is stupid reliable, if you dont f it up & dont put metadata in raid5/6. Having checksum consistency & checkpoints is a superpower most of the alternatives dont have. Btrfs is much more flexible about adding/removing drives, of various sizes even, to a raid pool, as compared to zfs, & is imo ridiculously easier to operate, from not needing out of tree modules to just requiring a lot less specialist knowledge in general of different zfs caches for tuning/setup. There's no alternative close & it runs great across millions of systems, with operators such as Facebook/Meta.
  - bravetraveler 4y ago
    Everyone in production reproduces this. Intentionally or not, everything on a long enough timeline will suddenly lose power. I'm sorry to offend what is otherwise a decent filesystem, but I, and the systems I'm responsible for, will not use BTRFS with RAID and this characteristic. There's a lot of qualification there, read it again. I would much rather let the journal replay and do its thing, than rebuild the array. Something every alternative offers, consistently. BTRFS RAID has routinely burned me every time I have given it the opportunity. It hasn't survived nearly as much as the other test arrays. I'm fine discussing filesystems, benefits, and all of that - but I can't entertain it when you try to make this so heated. These are anecdotes from reliability tests I've been doing... I'm sharing them so that they can be considered. You accuse me, while opening like this? At least my anecdotes weren't insulting. edit: To add, this isn't something I've kept secret. I don't know why you say that. Have you read every communication I've written? I can show you screenshots that span the last three months discussing this exact situation I haven't reported it upstream because I have better things to do; ship. You might see this, but hopes aren't high. This is about the response I expected. It's an extreme edge case, but it consistently cuts. The filesystem we use is incredibly uninteresting. By design. I find it utterly hilarious you think I'm the only person who has had issues with BTRFS RAID

moloch-hai 4y ago

RAID 5 and 6 are obsolete anyway, a relic of times when 2x storage cost too much.

willnonya 4y ago
That just seems like an absurd statement to me. It ignores so many use cases and financial realities.
encryptluks2 4y ago
2x storage is still expensive for home NAS use. RAID 5/6 has other benefits too over RAID 10. Maybe RAID 5/6 isn't the right choice, but it seemed like the best choice when I was comparing options.
- moloch-hai 4y ago
  The time it takes to initialize RAID 5 or 6, or to swap out a drive, was manageable when the drives were smaller. Now, the benefit of fewer drives is swamped by the extremely long time needed before you can do anything with them.

anonymousDan 4y ago

As someone who is looking to purchase their first NAS any advice on keeping the energy costs down (e.g. does particular NAS hardware or hard drives have running cost advantages in this area)? My main use case (at least initially) will be for storing and backing up photos for infrequent access.

encryptluks2 4y ago
I would assume most 7200 RPM drives have similar energy requirements, although it does appear some drives get hotter than others so efficient cooling might increase those requirements by a bit. Cost of storage though can be a bit expensive. I was able to find 20TB CMR drives for around $329 each.