4 ms·
Don't run 12TB of data with consumer hardware in a RAID5. If you have the money stick with RAID 1, 10, maybe 6 or even JBOD.
by MattSteelblade 10y ago
Don't run 12TB of data with consumer hardware in a RAID5. If you have the money stick with RAID 1, 10, maybe 6 or even JBOD.
- tachion 10y agoRAID 6 will be almost as slow and vulnerable as RAID 5. The best one in terms of speed and data security would be RAID 10 or RAID 100.
- X86BSD 10y agoGah! Why is anyone even uttering the word "RAID". Ugh. ZFS people. ZFS.
- tachion 10y agoI agree, you can see from my other comments in this thread, I am a strong advocate for FreeNAS :)
- fnord123 10y agoWhat's the implication here? Do you eat the 50% storage efficiency with mirrored pools? Or do you use a wider stripe for better resiliency (raidz2, raidz3)?
- Filligree 10y agoI've got an 11-disk system with raidz3. All the striped modes will (in effect) limit write rates to the bandwidth of a single disk, but ZFS is good about this: So long as the writes are reasonably large, as they'll likely be for a media server, you'll usually get the maximum streaming bandwidth of the disk. So, 150-200 MB/s. Reads are much faster. It's not something you'd want to use in a DC, but for mostly-idle storage at home it's perfectly fine. Add in an SSD cache, and it'll be downright speedy. That gives an effective capacity of 64 TB, given 8TB disks, and the chance of four disks dying before you can replace even a single one is pretty low. The usual limitation of ZFS applies, however: You have to decide the number of disks up-front, as you can't expand the size of a RAIDZ vdev later on. (Well, you can, but only by replacing all the disks with larger versions.)
- X86BSD 10y agoI run stripped mirrors with ZFS. 2x2. RAIDZ is simply to slow in comparison. Like A LOT slower. Ill just eat the storage loss with mirrors. Disk is cheap.
- raattgift 10y agoDisk is cheap, but how you attach the disks starts becoming an important cost component as you add more disks. :-( IOPS aren't cheap, and you will find out just how much that matters the moment you have to replace a disk in a pool with a lot of data in it. Resilver times of days can be pretty agonizing in a home setting. I won't build a pool without at least double redundancy per rotating-drive vdev, meaning at least 3-way mirrors (zpool create foo mirror disk0 disk1 disk2 mirror disk3 disk4 disk5) or raidz2, as I have had to go to backups for failures during resilvers of raidz (never again!) and 2-way mirrors. 3-way mirrors conveniently provide high read IOPS. OpenZFS does a good job aggregating and scheduling writes to the point that writes are mainly choked by random reads. (Very full pools induce quasi-random reads while hunting for space in metaslabs, which is the main reason writes to very full pools perform poorly). 2-way mirrors are only really fine for vdevs that have negligible random-access penalty (and thus likely high IOPS), or where rebuilding an entire pool from backups takes time comparable to resilver and scrub times when dealing with a replacement leaf device that might fail during resilvering (you cannot know the data on the replacment device is OK until you have read it, which generally means scrub-after-resilver), or errors reading the surviving leaf device(s).
- boredinballard 10y agoRAID is fine. As long as it's RAID 10 haha.
- Dylan16807 10y agoWhat's your reasoning behind that? I would trust a 6+2 RAID 6 more than a 4+4 RAID 10.
- X86BSD 10y agoIf your that concerned with redundancy use a 3-disk striped mirror with ZFS. Redundancy AND great I/O.
- weitzj 10y agoResilvering time. You are faster getting a broken drive fixed from a mirror. This gets more important with increased drive capacity. There is an article somewhere calculating the probability of two drives failing during resilvering and how this gets worse with the ever increasing capacities, which increases the resilvering times. But I guess you already account for this by using a Raid 6 with +2.
- Dylan16807 10y agoI can't think of any reason resilvering should be significantly slower with RAID 6. Even an extreme bottleneck of attaching all the drives to a single SATA channel would let you resilver a 10TB disk in roughly one day. Triple parity can give you pretty good integrity, but at a certain point it's wasted effort. Disk failures cluster too much for any single array to be safe; you need backups.
- fnord123 10y agoThat's good advice. I was trying to cut the price down a bit so OP didn't fall out of his chair. Maybe 10TB was an unlucky number plucked out of the air and just two 8TB drives running in a mirror will suffice.
- tesseract 10y agoI am running raidz1 on a 5-disk home fileserver with the philosophy that if I do get bitten by a multiple drive failure or latent bad sectors during a rebuild, I can count on zfs to tell me which files were affected and then I'll just restore them from backup. (Unlike md raid where rebuilds seem to be much more all-or-nothing.) My underlying assumptions here are: * A disk failure where some moderate number of sectors become unreadable is far more likely than a disk going completely toast. * Even if I were running raidz2 or raidz3 I'd still want the offsite backups in case of some disaster befalling the entire array (or more likely, me accidentally rm'ing something) * It's a home fileserver used mostly for archival purposes. Unlike a business scenario, the loss of availability while I restore from backup is not going to hurt too badly. Did I miss something here or do you think this is sound?
- MattSteelblade 10y agoLooks good to me.