8 ms·
Understanding RAID: How performance scales from one disk to eight
- 411111111111111 7y agoGreat write-up. Personally I'd have liked a more explicit warning for using raid 5/6 as the bigger they get, the more likely you're to lose the whole array on failures because the remaining disks will be under heavy load for recovery and are likely about to fail as well. He did mention it as a side note, but it deserves more attention I think.
- scandox 7y agoI remember standing with a few people discussing a RAID failure of this kind and asking each person how it could have happened. All of them could explain the benefits of RAID 5 for the specific application but not one of them could explain its failure modes.
- jldugger 7y agoThat's just one of many. Another good one is the violent failure mode. We had a RAID 6 go bad because a head crashed hard, hard enough that two other disks in the array also head crashed. Since the enclosure puts them pretty much lines them up, this is kinda inevitable.
- blackflame7000 7y agoWow, was that an IBM Deathstar? Two additional reasons for raid 6: 1. Fan failure on a hard disk pod causes damage to multiple drives simultaneously. 2. Many adjacent drives are connected via the same power cable which always has the potential to destroy drives in groups via over-voltage from a failing power supply.
- bluegreyred 7y agoYes, there's also the topic of unrecoverable read errors (URE) and their effect on successful RAID rebuilds. [1][2] Most consumer drives are still sold rated as <10^14 bits read per error. That's 12.5 terabytes, so in the worst case you could end up in situations where — on average — you are unable read a full 16TB drive without an error. Needless to say this is less than optimal for rebuilding a failed RAID array. Anecdotal evidence (i.e. very low error rates during ZFS scrubs) suggests that manufacturers underrate their drives and they are much more reliable than that, but it is something to keep in mind. Fortunately drive capacity completely outpaced my needs for personal data storage in the recent years, so I am happy with JBOD or RAID1, with backups of course. [1]http://www.raidtips.com/raid5-ure.aspx http://www.raidtips.com/raid5-ure.aspx [2]https://magj.github.io/raid-failure/ https://magj.github.io/raid-failure/
- vardump 7y ago> Anecdotal evidence (i.e. very low error rates during ZFS scrubs) suggests that manufacturers underrate their drives and they are much more reliable than that, but it is something to keep in mind. I've seen plenty of drives returning invalid data with correct CRC over the years. On reliable server-grade Xeon + ECC hardware. Of course the vast majority of drives never do it, there's just no way to know which ones do until it happens. Firmware bugs in weird corner cases? Cosmic rays? Perhaps, but I think it's more reasonable just consider it one of those weird things that occasionally Just Happen (TM) and just need to be protected against at a higher level. All drives produced in the last 30 years or so are running ever more complicated software stacks. For example they all have features that move data at risk to safer locations without the host system requesting or even knowing about it. Their physical (like bits on spinning rust or NAND flash block) and logical (what the host sees) data representations can be completely different. Plain old CRC errors, though, are way more frequent. I feel much more comfortable about those errors, at least the drive knows the data is corrupted.
- cyphar 7y agoThe other issue is that RAID (even RAID 6 with no dead disks) fundamentally cannot deal with data corruption or any of other fun bugs hardware likes to throw at you like phantom reads and writes. This is because it has no checksumming and most implementations will overwrite the parity if a mismatch is found. There's also the RAID "write hole" problem, but most implementations have worked around that since the mid-2000s. I switched to ZFS a few years ago and it's really quite amazing. In fairness, raidz does suffer from the same "resilvering takes a long time and increases the chance of subsequent failures" problem, which is one of the reasons why mirrors are more commonly used. Though unlike RAID, ZFS knows which data on the disk is actually used so can reduce resilver times pretty drastically.
- skyde 7y agowould have loved to see raid-Z in this test.
- tobyhinloopen 7y agoThe conclusion is unclear to me. Is the author saying RAID is a backup?
- WantonQuantum 7y agoYes. And RAID10 is the best backup strategy because 10 is bigger than 5 or 6.
- hans_castorp 7y agoSo how do I create a RAID42 then? Wouldn't that be the ultimate solution to all RAID problems?
- Rafuino 7y agoThis is really well written and I learned a bunch. I wonder how much money vendors make by selling hardware RAID when customers could get away with kernel RAID. Also, thank you for this bit! "All tests performed here are random access, because nearly any real-world storage workload is random access." Question for you smart people... is there a reason to use such an old Linux kernel version? They're testing on 4.15, which was released over 20 months ago. Would there be any appreciable difference on 5.6.4 or something in the 5.x range? Perhaps not since hard drives can't really improve much, but I'm curious.
- Arnt 7y agoHint: You haven't read about any performance breakthrough in the past several years.
- Rafuino 7y agoNot in HDDs... there's been quite a bit of improvement in SSDs and new memory media in the past several years.
- jmg90 7y agoLSI controllers (adapter cards really no better) for VMS Alpha used to go for a couple thousand list for a retail sub 500 dollar part lavishly upgraded with some firmware that ensured the margins remained as high as the system clock on our ES45s. Fifteen years elapsed before we got the same bandwidth inside a Xeon.... I've still got the above mentioned dual 3.GHz boxen in archive because after that failure we copied all the internal data and mothballed them in case. Furloughed a office lot of twenty thousand dollars workstations because of a shoddy poxy consumer grade aic we were probably missold in the first place has we not had the full pr moves made on us to forget that... NDA expired years ago and only said we wouldn't be beastly to Dell who have been transformed since then the better imo though I am still not buying
- zxcvgm 7y ago> _One final warning about hardware RAID controllers: It's difficult to predict whether a hardware RAID array created under one controller will import successfully to a different model of controller later._ > _We find that with hardware RAID, it's frequently difficult to tell whether you're nuking your array or importing it safely. So in the event of a controller failure and replacement, you may end up sweating bullets, YOLOing, and hoping._ This was the main reason I decided to switch to Linux software RAID from a 3ware card years ago. A thunderstorm caused a power trip and after powering the system back on, the 3ware controller lost its EEPROM contents that held the PCI vid/pid. Fortunately, it was just a RAID 1 mirror. We used dd to cut past the metadata header and managed to mount the ext2 filesystem. Using software RAID, I’m confident that we can take the drives to another machine in a pinch and still be able to mount it without much hassle.
- jmg90 7y agoSomehow storage creates the most intriguing narratives - maybe we collectively can most everyone appreciate losing or having data at risk, while otter arcana can't quite find the same empathy. Anyway... In 05 Dell sold me a workstation and internal RAID array that suffered a drive fault requiring replacement and rebuild. Unfortunately the engineer who was handholding me through the steps just to make the changes under our support conditions of the job being theirs and otherwise we left alone, told me it was a LSI card and not a Adaptek installed. This was a very entry level card and the only such card I've experience of. (Accustomed to the luxury of the high end) So I had to take him on his assertion it really was a LSI OEM PERC controller. Erm, not exactly. I learned this when a second connector dislocated and I blithely yanked and reseated U32PSCSI ribbon neatly and promptly as I imagine anyone would. Only the PERC was Adaptek and not LSI and for reasons my recall may be protecting me from* failed the second drive in the R10 array. I vowed to never run any hardware RAID again and I have maintained my resolution until last month, if specifying a rack of dedicated storage counts for proprietary hardware in equivalece of black box opacity. I should have written my experiences and blogged them because I took this loss of the subsystem so badly I became obsessive about learning first how to reconstruct the borked array from raw disk, a pleasure indescribable in polite company, and carried on until I was holding forth in charge of a budget for storage space in my dreams I never imagined presiding over. Every which way I believe only in software control of stores. I'm a true disciple to the cause. This r as rack of proprietary storage hardware only got its PO after the methods of code escrow had been thrashed to death in the final thrills of fall and last winter. (Incidentally the outside legal lead wrote me privately afterwards and pondered out loud whether the escrow didn't create a backdoor kind of vendor lockup because of the potential for contaminating employees with IP during emergency contingencies operating. Thanks Buddy! I'm not sounding like it but I'm actually grateful for that consideration if only because it sounds good for avoiding any repetition of the process just endured. It's a dual licensed codebase but being on the commercial code and the open code trailing in develop... I'd love to see what eg DDN routines are like I'm absolutely sure eager audience awaits publications like that. Surely the lock down must have awoken the industry to the needs of self sufficient repair and diagnosis.
- ngcc_hk 7y agoGot my qnap 8 drive to backup my synology 8 drive. Now the zfs will come soon once those in place. Hope that is a good mutual backup.
- ptha 7y agoFollowing the link to libeatmydata that some hardware controllers use for asynchronous writes: libeatmydata is a small LD_PRELOAD library designed to (transparently) disable fsync (and friends, like open(O_SYNC)). This has two side-effects: making software that writes data safely to disk a lot quicker and making this software no longer crash safe. DO NOT use libeatmydata on software where you care about what it stores. It's called libEAT-MY-DATA for a reason. Also just some latin pedantry, the article mentions the phrase: Caveat imperator a couple of times. I assume to mean Let the buyer beware, which as far as I know is Caveat emptor. Google latin translating: https://translate.google.com/#view=home&op=translate&sl=la&tl=en&text=caveat%20emptor%0Acaveat%20imperator https://translate.google.com/#view=home&op=translate&sl=la&t... Does caveat imperator have another meaning (perhaps the emperor is infallible)?
- ezzaf 7y agoI understood "caveat imperator" to mean operator (commander) beware https://en.wikipedia.org/wiki/Imperator https://en.wikipedia.org/wiki/Imperator
- avianlyric 7y agoYeah the author clarifies this in the comments. https://arstechnica.com/information-technology/2020/04/understanding-raid-how-performance-scales-from-one-disk-to-eight/?comments=1&post=38815324#comment-38815324 https://arstechnica.com/information-technology/2020/04/under...
- louwrentius 7y agoI have a passion for storage and I really liked this article. I'm also very interested in the follow-up about ZFS. Please note that the Author Jim Salter also has a nice podcast: https://techsnap.systems https://techsnap.systems Also interesting: https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/ https://arstechnica.com/gadgets/2020/02/how-fast-are-your-di... My main gripe with the article is that the topic of latency seems underdeveloped. The testing I've done using some of the parameters discussed on the article shows extreme, probably unrealistic latencies on the storage.
- louwrentius 7y agoA note about RAID arrays: Yes it is true that drives have become bigger but not faster. This causes RAID rebuilds to take longer. But I am not aware of any hard evidence that this means that this increases the risk of a failed rebuild. There are more sectors that could be bad but you need to do a patrol read / scrub at least one month to detect them in advance. I think there are a lot of scares about RAID of which I really wonder how much of it is rooted in anekdote and folklore.
- Craighead 7y agoThe thinking goes that the more total storage available the more unrecoverable bit errors occur
- montjoy 7y agoThe main issue with big drives is that you are sitting in a degraded state for longer while you await the rebuild. That allows more time for another disk to fail while your rebuild is incomplete.
- zepearl 7y agoRight - more time in a degraded state and more stress on the remaining drives during the rebuild (because there will be more data to synchronize).
- 7y ago
- juskrey 7y agoI have avoided RAID for the past 20 years and will avoid it in future. The whole technology and its benefits is about picking pennies before the steamroller, with explosive complexity of failures. I prefer much more simple and understandable replication and backup solutions - there are dozens of them.
- throwaway-9320 7y agoHave you tried out ZFS? It's quite good at resolving failures and data corruption and can be configured in all sorts of combinations, depending on your performance and disk failure tolerance requirements.
- juskrey 7y agoYes, it is good!
- bouncycastle 7y agoIt's very important to understand that RAID is not a backup solution. You still need to backup even if you have RAID, no matter what level. What it offers (in parity mode) is improved reliability and more chance that your server will be online if your drive dies. In majority of cases, it saves you from restoring from backup. Just replace the bad drive, and away you go! However, never assume that it will protect you from data loss. Backup. Backup. Backup!
- MaxBarraclough 7y agoAlso, test your backups regularly.
- prussian 7y agoYes, but what the parent comment is saying is that simple replication can replace the need for parity arrays entirely.
- juskrey 7y ago
- mohammedhdotio 7y agoGreat writing. however a question for geeks. what is wrong with using RAID0 for performance along with external backup/deduplication solution like borg backup/duplicati etc... ?
- vetinari 7y agoIt depends only on whether you can afford to lose the data. I'm using two SATA SSDs in md RAID0 for VMs on my desktop (where each VM gets its own LV). Dangerous thing, but if one of them fails, nothing happens, I will shrug and reinstall.
- tecleandor 7y agoAnd how many time will take to reinstall. You can take as an example the Recovery Point Objective and Recovery Time Objective terms used on disaster recovery. Depending on: a) how often you do backups, b) how many data are you backing up, and c) how fast are your network and transfer medium, you can calculate how much data you may lose in case of failure and how much time you'll use on recovery.
- jlgaddis 7y agoAs long as your choice of RAID0 is an informed one (i.e., you are fully aware of the risks and failure modes as well as the benefits -- and it sounds like you are), there's absolutely nothing wrong with using it.
- oofabz 7y agoThere are three drawbacks to the setup you describe: - If a drive fails, the system is offline until you restore from backup. Fine for personal use, but not okay if customers are paying you for a service. - If a drive fails, any changes since the last backup are lost. Fine for long-term archival, not so great for a bank ledger. - No capability to detect data corruption. RAID with parity can run a scrub to find and repair bit rot. If data is corrupted on your RAID 0, you probably won't notice, and you'll back up the corrupted. If your filesystem has checksums you will be able to detect the corruption but not repair it. Your filesystem probably doesn't have checksums. If you frequently run I/O intensive workloads, the extra performance might be worth the tradeoffs. If you're looking for your PC to feel slightly faster it seems foolish.
- rconti 7y agoIt seems every post here saying negative things about "hardware RAID" should prefix the term with "inexpensive consumer" or at least "single-system integrated". Which, to be fair, makes sense with the subject of the article limiting us to a discussion of 8 drives.
- SlowRobotAhead 7y agoRight. I struggle to see how anyone in this thread is seriously comparing their PC with ZFS to my server with 12x 3.84TB SAS SSD and a PERC 740 controller. Yet they are with absolutes like “Hardware RAID is dead”
- ecpottinger 7y agoWait a few years till something happens to your controller and you try to get a compatible version.
- SlowRobotAhead 7y agoYea, I think Dell makes more than 100 PERC740s. Plus, you’re talking about RAID as a backup, it is not. RAID is redundancy, not disaster recovery. If your backup solution is bas d on your RAID controller series, you’ve already lost.
- theevilsharpie 7y agoI manage a server fleet with local storage ranging from single-disk systems, to servers with 12+ internal disks and multiple attached JBODs, with disk tech including SATA and SAS HDDs of various RPMs, as well as SATA SSDs. Many of these machines are equipped with hardware RAID controllers of various brands. Hopefully that helps to demonstrate that I do actually have experience with enterprise hardware, and I'm not just some kid with a home lab parroting what I've read online. So, here goes... The management tooling for hardware RAID is a pain to deal with. Dell's OpenManage tools aren't that bad (other than being stupidly bloated), but every time I have to use LSI's MegaCLI utility, I end up wanting to shoot myself. (Seriously, do a Google search for "MegaCLI cheat sheet". It's _bad_.) For the cheaper LSI FusionMPT-based cards, I've completely given up trying to manage them from the OS, because I've never been able to get it to work reliably. And because hardware RAID controller manufacturers have changed ownership so many times over the past decade, even _obtaining_ the software can be a challenge. And that's assuming that it supports your OS, which isn't guaranteed. Monitoring the health of hardware RAID arrays is challenging. Not only do you generally need to use a proprietary tool of varying quality to do so (with all the issues stated above), the information you get is typically not that helpful. Most RAID controllers will at least tell you if the array is degraded or not (i.e., one or more disks are missing), but the quantity and quality of that information can vary. Additionally, monitoring the health of the underlying physical disks that make up the array is also important, but getting this information from a hardware RAID controller is tedious (and sometimes impossible), which can leave you blind to the actual health of your storage. The disk health typically reported by RAID controllers (the greem/amber blinky light that indicates a failed or failing disk) is based entirely on the disks reported SMART health check status, which is so prone to false negatives that it may as well not even exist. Oh, and if you're using an OEM RAID controller with third-party disks, you often have to deal with a nag message informing you that the disks aren't "official," which at best is annoying, and at worst can hide legitimate disk issues. Hardware RAID can have annoying limitations. Want to mix a SAS and SATA disk in the same array? Most controllers won't support that. Want to mix RAID levels on the same disks, like in the case of a distributed database where you might want some redundancy for the OS volume, but the actual database volume can be RAID 0 for performance/space efficiency? Most controllers don't support that, either. Want to create multiple distinct logical volumes from a single array (e.g., a 20 GB volume for OS, and a 10+ TB volume for data)? That's usually possible, but in the case of many LSI MegaRAID controllers (including their Dell PERC rebrands), that can prevent you from expanding the space on the array without a rebuild. Want things like RAID 6, more advanced nested RAID levels like RAID 50 or 60, or tiered storage? If the controller supports it at all, it's probably a premium-feature that costs extra to unlock. Hardware RAID is slow. The onboard processors on most controllers have no problem keeping up with mechanical disks, but struggle with the performance potential of SSDs. The centralized cache on the controller can be useful in some situations, but the typical configuration is to use this cache while disabling the cache located on the disks themselves, which runs into scalability issues for obvious reasons. And lastly, you're ultimately limited by the throughput of the bus to which the controller's attached, which means that a hardware PCIe RAID controller can't possibly compete with NVMe disks that are directly attached to the processor via the PCIe bus. Software RAID (at least on Linux) is fast, simple, easy to use, the UX is consistent no matter the underlying hardware, it's well integrated with many existing monitoring tools, and frankly, it's been more reliable than hardware RAID. All of our newer machines use software RAID with either NVMe disks or SAS HBAs, and as our older machines are being repurposed, they are being converted to software RAID where possible. It's made my life a lot easier, and nothing of value has been lost. Hardware RAID is dead. The only reason I'd ever use hardware RAID again is if I had to use an OS which didn't have decent software RAID support.
- montjoy 7y agoI wish articles like this could afford production-grade hardware to test on. When you’ve run large Fibre or SAS arrays in the past on enterprise hardware it’s hard not to see gaps in the testing methods/setup. Sequential writes metrics ARE important when doing database backups. SATA (even nearline) and SAS disks have vastly different performance profiles so I would expect optimal configuration would be different. What about SSD vs. spinning? What about different rotational speeds? Were the disks aligned properly? How does performance scale with increasing the number of threads requesting I/O? The setup in the article really only applies to certain prosumer/small business requirements. I guess I’m not the right audience for the article.
- yardie 7y agoThis is a great article on the basics of RAID. Outside of consumer grade RAID systems it's a bit out of date. There are some perceptible performance differences between enterprise and consumer storage systems. I manage a small (<100TB) storage system for our office. We have servers using RAID10 SSDs and we have other servers using RAID10 10k SAS HDDs. The read performance on the SATA SSDs is great but the writes are atrocious.
- theevilsharpie 7y ago> The read performance on the SATA SSDs is great but the writes are atrocious. If a 10K SAS HDD array is outperforming even a single SATA SSD, either your I/O access pattern is extremely sequential, or there's something wrong with how your storage is configured.
- yardie 7y agoOur environment is almost entirely virtualized, so yes, extremely sequential. Each host has 2 or more 10Gbps NICs. So starting, suspending, moving, snapshoting and cloning a VM shows the limits in both. Our SAN is 4 years old and has tiered storage (10k SAS mainline HDDs and high-write SAS SSDs). We use cheaper SSDs in some instances. But there is a performance difference of 15%-25% between SAS and SATA and that can't be ignored.
- Diederich 7y ago> In our experience, administrators are overwhelmingly likely not to notice when a hardware controller's cache batteries fail. Frequently, those administrators will still be operating their systems at reduced performance and reliability levels for years afterward. Can anyone else weigh in on this claim? In the two jobs where I've been responsible for such equipment, monitoring both the performance and battery status was pretty critical and done with due care and focus.