4 ms·

The overall premise of this article is good and needs to be said, but the author missed an opportunity by lumping traditional hardware RAID solutions with softw

by Exmoor 6y ago

The overall premise of this article is good and needs to be said, but the author missed an opportunity by lumping traditional hardware RAID solutions with software solutions like ZFS. The general consensus I've seen from people spinning up large storage arrays in recent years is that the software solutions are far superior for many of the reasons the author mentions.

swills 6y ago
I agree and think it also needs to be pointed out that "traditional hardware RAID solutions" are software (firmware) on a separate processor, software which is often in my experience troublesome, buggy, inscrutable, difficult to update, when updates are available, etc.
- ajsnigrutin 6y ago
  But... nobody ever got fired for choosing IBM! If you set up "your own solution", you're always the "guilty one" if something fails. If you buy IBM (or HP or whatever), it's their fault if something fails. (i did some of that stuff in enterprise enviroments.... honestly, IBM/HP/... + paid support was always the better option for me, mainly due to "liability" issues)
  - zrm 6y ago
    "Nobody ever got fired for buying IBM." -IBM, just before PC servers ate their lunch. My experience has always been that it's better to do your homework so you don't have problems than to have a scapegoat to blame the problems on. And the extent to which it's "their fault" tends to be quite inadequate. If you build a 16-drive RAID5 on some enterprise vendor's equipment and then suffer a multiple disk failure during a rebuild because that was a Bad Idea, what's the support contract going to get you? An overnight replacement for both failed drives? All your data is still gone.
    - ajsnigrutin 6y ago
      > If you build a 16-drive RAID5 on some enterprise vendor's equipment and then suffer a multiple disk failure during a rebuild because that was a Bad Idea... If I do that with a linux/solaris/mdadm/zfs/btrfs/whatever, and it fails, who will be the one blamed for the failure? Me of course. If i select a propper raid setup, and it still failes, it's better if it fails on an IBM than on my duct-tape linux/mdadm/zfs/btrfs/whatever solution, because again, I personally can say, that i did what the vendor (IBM) recommended in their brochure, and their solution failed. If the duct-tape solution fails, there's noone else to blame, than me. Either way, you have to restore from backup. And if there is some random bug (eg. kernel oops when mounting volume), there's IBM support on the call. Working with standard options, means standard, well-known bugs, and that means that support can help fix them. I know, that I (personally) sleep better, knowing there is a professional solution from a big-brand running in a "boring, enterprise" production.
      - zrm 6y ago
        > If I do that with a linux/solaris/mdadm/zfs/btrfs/whatever, and it fails, who will be the one blamed for the failure? Me of course. So don't create a 16-drive RAID5 at all. > If i select a propper raid setup, and it still failes, it's better if it fails on an IBM than on my duct-tape linux/mdadm/zfs/btrfs/whatever solution, because again, I personally can say, that i did what the vendor (IBM) recommended in their brochure, and their solution failed. But you're the one who chose the vendor, and evaluated their solution, and accepted it. So you're still fired. > Either way, you have to restore from backup. The point is there's a third option. Use a level of redundancy appropriate to the importance of the system, and then it doesn't fail to begin with and it isn't necessary to divert blame. Meanwhile you get to show your boss how you saved the company millions of dollars. > And if there is some random bug (eg. kernel oops when mounting volume), there's IBM support on the call. Working with standard options, means standard, well-known bugs, and that means that support can help fix them. Which you can still get for linux/solaris/mdadm/zfs/btrfs/whatever from a variety of vendors if you really want it. But because they're generic technologies, this also means you can get that support from multiple competing providers (so the cost is lower), including per-incident support once something happens that requires it instead of having to buy a support contract you may not ever need. Which you generally won't need, because the likes of mdadm are more widely used (and so better tested) than any individual vendor's proprietary hardware RAID solution, and can also recover more easily from things like controller failures because the replacement controller needs no state and is not required to match in firmware version or even model.
  - emsign 6y ago
    That's funny I use an IBM rebranded RAID card. In AHCI mode with ZFS.
louwrentius 6y ago
I kept my article neutral and I realise I should have stated that I consider Linux software RAID and ZFS in the same boat in terms of perceptions about RAID. People forgo on both Linux software RAID and ZFS because of scary stories. ZFS is an interesting alternative to Linux software RAID that discussion is a different topic all together.
- garmaine 6y ago
  The two fundamentally are different in ways that you are conflating. A hardware raid card will eject a disk that shows a single URE. Your zfs volume will happily correct the error and just report it as a minor statistic that is cleared on your next reboot. Yes zfs is immune to the problem the zdnet article hyped up. Because it is immune by design in a way that is purposefully different from raid.
  - StillBored 6y ago
    That completely depends on the RAID controller/software. Smarter ones, with sufficient slots will leave the "failing" drive online long enough to assure that should a sector fail to be read from the remaining drives an attempt can be made to read it from the "Bad" drive as well. In cases where the drive is being kicked for excessive SMART relocation, or hard read errors this could be the diff between the array going offline and the rebuild succeeding. For something like mdadm, you can simulate this by upgrading /reshaping from RAID5 to RAID6 (or similar) and then pulling the failing drive and reshaping back to raid5. Or as did a few times a decade+ ago when we had a really bad batch of drives, pick the bad sector off the failed drive, compute the corrected sector "by hand" and write it back to the drive with a read error.
- layoutIfNeeded 6y ago
  Scary stories of ZFS? I thought it was the pinnacle of file systems in terms of reliability.
  - magicalhippo 6y ago
    I mean there are some people who have lost ZFS pools or a lot of data. But for each one of those, there are tons more that have lost entire hardware RAIDs. There's also tons more of those that had some horrible error and imagined their ZFS pool was gone for sure, only to find they just lost a couple of files, if anything at all.
dehrmann 6y ago
I, too, have had this experience, but how is this possible? So much other experience has taught me that hardware is better. The main downside I've had with ZFS (over hardware RAID) is performance, but it's been incredibly reliable.
- heavenlyblue 6y ago
  Which other experience?
  - dehrmann 6y ago
    Softmodems, various virtualization technologies, emulators.
    - heavenlyblue 6y ago
      You mean hardware emulators are better than software emulators?
- kietdlam 6y ago
  It's because ZFS is much more than RAID. RAID just exposes a disk to the OS. ZFS exposes a filesystem and knows more metadata, e.g. the checksum, so it knows when to repair the data using the redundancy from the disks.
tw04 6y ago
Software RAID has always been superior - it's just that the vendors who were doing it right built enterprise arrays on their software stack and printed money. There aren't many standalone entities anymore, but the remains still exist. 3Par and Nimble (now HPe), EMC (now Dell), Solidfire (now NetApp), NetApp, Infinidat, IBM, etc. And ironically, the biggest competitor in the NAS space to NetApp back in the day essentially went bankrupt BECAUSE they used hardware RAID and ran into a TFDL situation that cause customers to flee in droves. https://en.wikipedia.org/wiki/Auspex_Systems https://en.wikipedia.org/wiki/Auspex_Systems
- arsome 6y ago
  Software raid is better, but it is worth picking up some LSI cards, just to put them in IT mode and pass them all through for software raid. Most of the cheap SATA cards you can buy have absolute garbage controllers and if you mix them with absolute garbage drives, some terrible failure scenarios occur, I've seen one failing drive take out a whole card, constantly knocking several drives offline in my RAID6 array. This was an absolute hell staring at smartctl and RAID repairs every few weeks until I finally gave up and swapped the controller, once that was done, the single drive failed cleanly and got kicked out without knocking the whole array offline.
  - StillBored 6y ago
    Just make sure they aren't of the megaraid variety. The data integrity/speed trade-off on those boards is probably the worst in the industry. What you are describing though sounds like a SATA card with a port expander. Most normal AHCI cards are fairly decent and the ports are independent. So loss of a single port won't take the whole thing offline. This is basically the same setup you have for the motherboard ports.

by Exmoor 6y ago

swills 6y ago

I agree and think it also needs to be pointed out that "traditional hardware RAID solutions" are software (firmware) on a separate processor, software which is often in my experience troublesome, buggy, inscrutable, difficult to update, when updates are available, etc.

ajsnigrutin 6y ago
But... nobody ever got fired for choosing IBM! If you set up "your own solution", you're always the "guilty one" if something fails. If you buy IBM (or HP or whatever), it's their fault if something fails. (i did some of that stuff in enterprise enviroments.... honestly, IBM/HP/... + paid support was always the better option for me, mainly due to "liability" issues)
- zrm 6y ago
  "Nobody ever got fired for buying IBM." -IBM, just before PC servers ate their lunch. My experience has always been that it's better to do your homework so you don't have problems than to have a scapegoat to blame the problems on. And the extent to which it's "their fault" tends to be quite inadequate. If you build a 16-drive RAID5 on some enterprise vendor's equipment and then suffer a multiple disk failure during a rebuild because that was a Bad Idea, what's the support contract going to get you? An overnight replacement for both failed drives? All your data is still gone.
  - ajsnigrutin 6y ago
    > If you build a 16-drive RAID5 on some enterprise vendor's equipment and then suffer a multiple disk failure during a rebuild because that was a Bad Idea... If I do that with a linux/solaris/mdadm/zfs/btrfs/whatever, and it fails, who will be the one blamed for the failure? Me of course. If i select a propper raid setup, and it still failes, it's better if it fails on an IBM than on my duct-tape linux/mdadm/zfs/btrfs/whatever solution, because again, I personally can say, that i did what the vendor (IBM) recommended in their brochure, and their solution failed. If the duct-tape solution fails, there's noone else to blame, than me. Either way, you have to restore from backup. And if there is some random bug (eg. kernel oops when mounting volume), there's IBM support on the call. Working with standard options, means standard, well-known bugs, and that means that support can help fix them. I know, that I (personally) sleep better, knowing there is a professional solution from a big-brand running in a "boring, enterprise" production.
    - zrm 6y ago
      > If I do that with a linux/solaris/mdadm/zfs/btrfs/whatever, and it fails, who will be the one blamed for the failure? Me of course. So don't create a 16-drive RAID5 at all. > If i select a propper raid setup, and it still failes, it's better if it fails on an IBM than on my duct-tape linux/mdadm/zfs/btrfs/whatever solution, because again, I personally can say, that i did what the vendor (IBM) recommended in their brochure, and their solution failed. But you're the one who chose the vendor, and evaluated their solution, and accepted it. So you're still fired. > Either way, you have to restore from backup. The point is there's a third option. Use a level of redundancy appropriate to the importance of the system, and then it doesn't fail to begin with and it isn't necessary to divert blame. Meanwhile you get to show your boss how you saved the company millions of dollars. > And if there is some random bug (eg. kernel oops when mounting volume), there's IBM support on the call. Working with standard options, means standard, well-known bugs, and that means that support can help fix them. Which you can still get for linux/solaris/mdadm/zfs/btrfs/whatever from a variety of vendors if you really want it. But because they're generic technologies, this also means you can get that support from multiple competing providers (so the cost is lower), including per-incident support once something happens that requires it instead of having to buy a support contract you may not ever need. Which you generally won't need, because the likes of mdadm are more widely used (and so better tested) than any individual vendor's proprietary hardware RAID solution, and can also recover more easily from things like controller failures because the replacement controller needs no state and is not required to match in firmware version or even model.
- emsign 6y ago
  That's funny I use an IBM rebranded RAID card. In AHCI mode with ZFS.

louwrentius 6y ago

I kept my article neutral and I realise I should have stated that I consider Linux software RAID and ZFS in the same boat in terms of perceptions about RAID. People forgo on both Linux software RAID and ZFS because of scary stories. ZFS is an interesting alternative to Linux software RAID that discussion is a different topic all together.

garmaine 6y ago
The two fundamentally are different in ways that you are conflating. A hardware raid card will eject a disk that shows a single URE. Your zfs volume will happily correct the error and just report it as a minor statistic that is cleared on your next reboot. Yes zfs is immune to the problem the zdnet article hyped up. Because it is immune by design in a way that is purposefully different from raid.
- StillBored 6y ago
  That completely depends on the RAID controller/software. Smarter ones, with sufficient slots will leave the "failing" drive online long enough to assure that should a sector fail to be read from the remaining drives an attempt can be made to read it from the "Bad" drive as well. In cases where the drive is being kicked for excessive SMART relocation, or hard read errors this could be the diff between the array going offline and the rebuild succeeding. For something like mdadm, you can simulate this by upgrading /reshaping from RAID5 to RAID6 (or similar) and then pulling the failing drive and reshaping back to raid5. Or as did a few times a decade+ ago when we had a really bad batch of drives, pick the bad sector off the failed drive, compute the corrected sector "by hand" and write it back to the drive with a read error.
layoutIfNeeded 6y ago
Scary stories of ZFS? I thought it was the pinnacle of file systems in terms of reliability.
- magicalhippo 6y ago
  I mean there are some people who have lost ZFS pools or a lot of data. But for each one of those, there are tons more that have lost entire hardware RAIDs. There's also tons more of those that had some horrible error and imagined their ZFS pool was gone for sure, only to find they just lost a couple of files, if anything at all.

dehrmann 6y ago

I, too, have had this experience, but how is this possible? So much other experience has taught me that hardware is better. The main downside I've had with ZFS (over hardware RAID) is performance, but it's been incredibly reliable.

heavenlyblue 6y ago
Which other experience?
- dehrmann 6y ago
  Softmodems, various virtualization technologies, emulators.
  - heavenlyblue 6y ago
    You mean hardware emulators are better than software emulators?
kietdlam 6y ago
It's because ZFS is much more than RAID. RAID just exposes a disk to the OS. ZFS exposes a filesystem and knows more metadata, e.g. the checksum, so it knows when to repair the data using the redundancy from the disks.

tw04 6y ago

Software RAID has always been superior - it's just that the vendors who were doing it right built enterprise arrays on their software stack and printed money. There aren't many standalone entities anymore, but the remains still exist. 3Par and Nimble (now HPe), EMC (now Dell), Solidfire (now NetApp), NetApp, Infinidat, IBM, etc. And ironically, the biggest competitor in the NAS space to NetApp back in the day essentially went bankrupt BECAUSE they used hardware RAID and ran into a TFDL situation that cause customers to flee in droves. https://en.wikipedia.org/wiki/Auspex_Systems https://en.wikipedia.org/wiki/Auspex_Systems

arsome 6y ago
Software raid is better, but it is worth picking up some LSI cards, just to put them in IT mode and pass them all through for software raid. Most of the cheap SATA cards you can buy have absolute garbage controllers and if you mix them with absolute garbage drives, some terrible failure scenarios occur, I've seen one failing drive take out a whole card, constantly knocking several drives offline in my RAID6 array. This was an absolute hell staring at smartctl and RAID repairs every few weeks until I finally gave up and swapped the controller, once that was done, the single drive failed cleanly and got kicked out without knocking the whole array offline.
- StillBored 6y ago
  Just make sure they aren't of the megaraid variety. The data integrity/speed trade-off on those boards is probably the worst in the industry. What you are describing though sounds like a SATA card with a port expander. Most normal AHCI cards are fairly decent and the ports are independent. So loss of a single port won't take the whole thing offline. This is basically the same setup you have for the motherboard ports.