4 ms·
By tolerating the loss of two disks, do you mean raidz2 or do you mean 3-way mirror? Raidz2 is not fast. In fact, it is slow. Also, it is less reliable than a
by barrkel 13y ago
By tolerating the loss of two disks, do you mean raidz2 or do you mean 3-way mirror?
Raidz2 is not fast. In fact, it is slow. Also, it is less reliable than a two way mirror in most configurations, because recovering from a disk loss requires reading the entirety of every other disk, whereas recovering from loss in a mirror requires reading the entirety of one disk. The multiplication of the probabilities don't work out particularly well as you scale up in disk count (even taking into account that raidz2 tolerates a disk failure mid-recovery). And mirroring is much faster, since it can distribute seeks across multiple disks, something raidz2 cannot do. Raidz2 essentially synchronizes the spindles on all disks.
Raidz2 is more or less suitable for archival-style storage where you can't afford the space loss from mirroring. For example, I have an 11 disk raidz2 array in my home NAS, spread across two separate PCIe x8 8-port 6Gbps SAS/SATA cards, and don't usually see read or write speeds for files[1] exceeding 200MB/sec. The drives individually are capable of over 100MB/sec - in a non-raidz2 setup, I'd be potentially seeing over 1GB/sec on reads of large contiguous files.
Personally I'm going to move to multiple 4-disk raid10 vdevs. I can afford the space loss, and the performance characteristics are much better.
[1] Scrub speeds are higher, but not really relevant to FS performance.
- JoeAltmaier 13y agoReading 2 disks is not very much slower than 1. Its always the I/o bandwidth that is at issue, so it very much depends on the connectivity. And it all pretty much impacts your customer. We've all suffered under the 4-hour mirror rebuild, the whole machine made inoperable by the constant disk load. The only way to alleviate that, is to design extra bandwidth (e.g. another cable and controller) that's used exclusively for recovery. I've designed/build storage that worked that way, but it was for enterprise. The home user doesn't want to pay for extra bandwidth and then not use it most of the time. And if the DO use it, then they notice when a rebuild is triggered, and complain. Its catch-22.
- chrissnell 13y agoCan you elaborate on the second controller? Are you referring to two disks on two controllers, using mdadm (or similar) to create a software RAID 1 across the two?
- JoeAltmaier 13y agoI'm not up on PC controllers these days. We were building a Fibre-Channel switch/router for Dell back then; all proprietary hardware.
- lsc 13y ago>Reading 2 disks is not very much slower than 1. Its always the I/o bandwidth that is at issue, so it very much depends on the connectivity. This was a problem in the old shared-bus u320 days... but now that we've got a 3 or 6 gigabit serial link to each disk? the bottleneck, unless you have some super-fancy SSD shit going on, is going to be getting the bits off the disk. I don't know of any spinning disk that can consistently saturate even a 2 gigabit link. That's the thing... random access on spinning rust is staggeringly slow compared to almost everything else your computer does... and while a rebuild is sequential access, mostly, if you are trying to use the system during the rebuild? well, simultaneous sequential accesses become random access, so yea, your system is gonna suck during the rebuild anyhow. Add to this, well, disk diagnostics suck. Quite often a single disk will perform under-spec for some time before failing, slowing down the whole raid. But SATA solved almost all of the bus bottleneck issues when it comes to disks.
- shiftpgdn 13y agoWhy not Raid 60 with Btrfs? It'll tolerate two disk loss with pro-active parity protection via btrfs and be faster and provide you with more disk space.
- rbanffy 13y agoThey are running BSD. I don't think BtrFS is an option and ZFS can do pretty much every trick BtrFS can. The data corruption on the first machine seems like a hardware problem.
- KaiserPro 13y agoZFS has inbuild raiding and much much friendlier tools. The BTRFS tools are like MDAM and lilo had a child.
- barrkel 13y agoDo you feel comfortable with reliability of btrfs? When I do a Google search, I want to see confirmed cases of fs lockups / data loss from fs bugs to be multiple years in the past. btrfs isn't there yet. I like btrfs's ability to e.g. switch raid levels without offlining the array. I think it's promising. But I wouldn't trust my data to it tomorrow.
- KaiserPro 13y agoRaid 60 is not really that good. the rebuild time of a raid 6 slice of 12 disks is 36 hours. (sas 3tb disks) Raid 10 is more expensive on space, but is the best compromise of traditional raid. LSI have some fancy magic virtual disk chunking mechanism to make rebuidling large raid 6 luns much quicker (4-8 hours)
- duaneb 13y agoBecause BRTFS isn't even feature stable yet.
- jlgaddis 13y ago... or supported on FreeBSD, which is what the server runs.