6 ms·

> _One final warning about hardware RAID controllers: It's difficult to predict whether a hardware RAID array created under one controller will import successfu

by zxcvgm 6y ago

> _One final warning about hardware RAID controllers: It's difficult to predict whether a hardware RAID array created under one controller will import successfully to a different model of controller later._ > _We find that with hardware RAID, it's frequently difficult to tell whether you're nuking your array or importing it safely. So in the event of a controller failure and replacement, you may end up sweating bullets, YOLOing, and hoping._ This was the main reason I decided to switch to Linux software RAID from a 3ware card years ago. A thunderstorm caused a power trip and after powering the system back on, the 3ware controller lost its EEPROM contents that held the PCI vid/pid. Fortunately, it was just a RAID 1 mirror. We used dd to cut past the metadata header and managed to mount the ext2 filesystem. Using software RAID, I’m confident that we can take the drives to another machine in a pinch and still be able to mount it without much hassle.

jmg90 6y ago
Somehow storage creates the most intriguing narratives - maybe we collectively can most everyone appreciate losing or having data at risk, while otter arcana can't quite find the same empathy. Anyway... In 05 Dell sold me a workstation and internal RAID array that suffered a drive fault requiring replacement and rebuild. Unfortunately the engineer who was handholding me through the steps just to make the changes under our support conditions of the job being theirs and otherwise we left alone, told me it was a LSI card and not a Adaptek installed. This was a very entry level card and the only such card I've experience of. (Accustomed to the luxury of the high end) So I had to take him on his assertion it really was a LSI OEM PERC controller. Erm, not exactly. I learned this when a second connector dislocated and I blithely yanked and reseated U32PSCSI ribbon neatly and promptly as I imagine anyone would. Only the PERC was Adaptek and not LSI and for reasons my recall may be protecting me from* failed the second drive in the R10 array. I vowed to never run any hardware RAID again and I have maintained my resolution until last month, if specifying a rack of dedicated storage counts for proprietary hardware in equivalece of black box opacity. I should have written my experiences and blogged them because I took this loss of the subsystem so badly I became obsessive about learning first how to reconstruct the borked array from raw disk, a pleasure indescribable in polite company, and carried on until I was holding forth in charge of a budget for storage space in my dreams I never imagined presiding over. Every which way I believe only in software control of stores. I'm a true disciple to the cause. This r as rack of proprietary storage hardware only got its PO after the methods of code escrow had been thrashed to death in the final thrills of fall and last winter. (Incidentally the outside legal lead wrote me privately afterwards and pondered out loud whether the escrow didn't create a backdoor kind of vendor lockup because of the potential for contaminating employees with IP during emergency contingencies operating. Thanks Buddy! I'm not sounding like it but I'm actually grateful for that consideration if only because it sounds good for avoiding any repetition of the process just endured. It's a dual licensed codebase but being on the commercial code and the open code trailing in develop... I'd love to see what eg DDN routines are like I'm absolutely sure eager audience awaits publications like that. Surely the lock down must have awoken the industry to the needs of self sufficient repair and diagnosis.
vardump 6y ago
People still use hardware RAID? Windows exempted, as there's unfortunately nothing better available for Windows. (ReFS SW RAID might be ok, haven't checked its state in a while. I know Microsoft is definitely working hard to fix this.) Hardware RAID is just data corruption waiting to happen, simply because there are more vulnerable components and interconnects along the data path. It's much better to checksum data while it's still protected by ECC RAM and CPU ECC caches – which is something only software RAID can do. Although even that is not a total guarantee. There's no longer reason to assume a slow SoC on a RAID adapter (even with XOR acceleration) would perform better than an Intel or AMD server CPU, as long as I/O path is not bandwidth limited.
- wazoox 6y ago
  Hardware RAID controllers always use ECC. Hardware RAID controllers haven't been using battery backed-up cache for the past decade, but supercapacitors and flash memory. So they provide better cache safety than any software RAID stack. Also hardware RAID allows for a more consistent performance under heavy system load in my experience.
  - vardump 6y ago
    > Hardware RAID controllers always use ECC. This is not what I was talking about. Of course they do. The problem is that data needs to somehow be transferred to the adapter first, and this alone adds a point of failure. Controller can (and too often do) have hardware and firmware bugs. The less components the lower chances for corruption. > battery backed-up cache for the past decade, but supercapacitors and flash memory No difference to software RAID there. For example ZeusRAM and Optane are used for this. Of course high endurance SLC flash with supercaps is also an option. > Also hardware RAID allows for a more consistent performance under heavy system load in my experience. I'd tend to count this subjective, but I'll grant you this one, it's at least plausible. I haven't encountered this issue, though – today's hardware has insane bandwidth. SW RAID can saturate 100 Gbps ethernet, how much more performance you really need? Even if performance was slightly degraded, data integrity is more important.
    - 6y ago
mobilio 6y ago
Intel also have solutions like ICH10 vs ICH10R. Today they also support it on some chipsets. But you're 100% correct. I have friend who have few RAID controllers in his office just in case that one of it's servers fail. And all of them are purchased in same order to be from same batch.
mc32 6y ago
Is it a problem with raid controllers that write raid config to its member discs?
rambojazz 6y ago
One more: hardware controllers use proprietary software. When the controller dies this means pain.
matheusmoreira 6y ago
Isn't software RAID just as likely to cause additional failures in remaining disks during rebuilds? Rebuilding a degraded array causes a lot of I/O. Storage capacities have increased tremendously but improvements in I/O speed and read error rates are orders of magnitude smaller. So we ended up with rebuilds that take over a day to complete and are likely to fail despite the redundancy. Using this RAID calculator gives me worrying numbers for high capacity arrays, even with enterprise hardware: https://wintelguy.com/raidmttdl.pl https://wintelguy.com/raidmttdl.pl Also, expanding the capacity of an array is a needlessly complicated process: https://raid.wiki.kernel.org/index.php/Growing https://raid.wiki.kernel.org/index.php/Growing The best solution seems to be something like Ceph... But that isn't practical for a home lab.
- Dylan16807 6y ago
  > Isn't software RAID just as likely to cause additional failures in remaining disks during rebuilds? Yes? But that's unrelated to the comment you were replying to.

by zxcvgm 6y ago

jmg90 6y ago

Somehow storage creates the most intriguing narratives - maybe we collectively can most everyone appreciate losing or having data at risk, while otter arcana can't quite find the same empathy. Anyway... In 05 Dell sold me a workstation and internal RAID array that suffered a drive fault requiring replacement and rebuild. Unfortunately the engineer who was handholding me through the steps just to make the changes under our support conditions of the job being theirs and otherwise we left alone, told me it was a LSI card and not a Adaptek installed. This was a very entry level card and the only such card I've experience of. (Accustomed to the luxury of the high end) So I had to take him on his assertion it really was a LSI OEM PERC controller. Erm, not exactly. I learned this when a second connector dislocated and I blithely yanked and reseated U32PSCSI ribbon neatly and promptly as I imagine anyone would. Only the PERC was Adaptek and not LSI and for reasons my recall may be protecting me from* failed the second drive in the R10 array. I vowed to never run any hardware RAID again and I have maintained my resolution until last month, if specifying a rack of dedicated storage counts for proprietary hardware in equivalece of black box opacity. I should have written my experiences and blogged them because I took this loss of the subsystem so badly I became obsessive about learning first how to reconstruct the borked array from raw disk, a pleasure indescribable in polite company, and carried on until I was holding forth in charge of a budget for storage space in my dreams I never imagined presiding over. Every which way I believe only in software control of stores. I'm a true disciple to the cause. This r as rack of proprietary storage hardware only got its PO after the methods of code escrow had been thrashed to death in the final thrills of fall and last winter. (Incidentally the outside legal lead wrote me privately afterwards and pondered out loud whether the escrow didn't create a backdoor kind of vendor lockup because of the potential for contaminating employees with IP during emergency contingencies operating. Thanks Buddy! I'm not sounding like it but I'm actually grateful for that consideration if only because it sounds good for avoiding any repetition of the process just endured. It's a dual licensed codebase but being on the commercial code and the open code trailing in develop... I'd love to see what eg DDN routines are like I'm absolutely sure eager audience awaits publications like that. Surely the lock down must have awoken the industry to the needs of self sufficient repair and diagnosis.

vardump 6y ago

People still use hardware RAID? Windows exempted, as there's unfortunately nothing better available for Windows. (ReFS SW RAID might be ok, haven't checked its state in a while. I know Microsoft is definitely working hard to fix this.) Hardware RAID is just data corruption waiting to happen, simply because there are more vulnerable components and interconnects along the data path. It's much better to checksum data while it's still protected by ECC RAM and CPU ECC caches – which is something only software RAID can do. Although even that is not a total guarantee. There's no longer reason to assume a slow SoC on a RAID adapter (even with XOR acceleration) would perform better than an Intel or AMD server CPU, as long as I/O path is not bandwidth limited.

wazoox 6y ago
Hardware RAID controllers always use ECC. Hardware RAID controllers haven't been using battery backed-up cache for the past decade, but supercapacitors and flash memory. So they provide better cache safety than any software RAID stack. Also hardware RAID allows for a more consistent performance under heavy system load in my experience.
- vardump 6y ago
  > Hardware RAID controllers always use ECC. This is not what I was talking about. Of course they do. The problem is that data needs to somehow be transferred to the adapter first, and this alone adds a point of failure. Controller can (and too often do) have hardware and firmware bugs. The less components the lower chances for corruption. > battery backed-up cache for the past decade, but supercapacitors and flash memory No difference to software RAID there. For example ZeusRAM and Optane are used for this. Of course high endurance SLC flash with supercaps is also an option. > Also hardware RAID allows for a more consistent performance under heavy system load in my experience. I'd tend to count this subjective, but I'll grant you this one, it's at least plausible. I haven't encountered this issue, though – today's hardware has insane bandwidth. SW RAID can saturate 100 Gbps ethernet, how much more performance you really need? Even if performance was slightly degraded, data integrity is more important.
  - 6y ago

mobilio 6y ago

Intel also have solutions like ICH10 vs ICH10R. Today they also support it on some chipsets. But you're 100% correct. I have friend who have few RAID controllers in his office just in case that one of it's servers fail. And all of them are purchased in same order to be from same batch.

mc32 6y ago

Is it a problem with raid controllers that write raid config to its member discs?

rambojazz 6y ago

One more: hardware controllers use proprietary software. When the controller dies this means pain.

matheusmoreira 6y ago

Isn't software RAID just as likely to cause additional failures in remaining disks during rebuilds? Rebuilding a degraded array causes a lot of I/O. Storage capacities have increased tremendously but improvements in I/O speed and read error rates are orders of magnitude smaller. So we ended up with rebuilds that take over a day to complete and are likely to fail despite the redundancy. Using this RAID calculator gives me worrying numbers for high capacity arrays, even with enterprise hardware: https://wintelguy.com/raidmttdl.pl https://wintelguy.com/raidmttdl.pl Also, expanding the capacity of an array is a needlessly complicated process: https://raid.wiki.kernel.org/index.php/Growing https://raid.wiki.kernel.org/index.php/Growing The best solution seems to be something like Ceph... But that isn't practical for a home lab.

Dylan16807 6y ago
> Isn't software RAID just as likely to cause additional failures in remaining disks during rebuilds? Yes? But that's unrelated to the comment you were replying to.