4 ms·

People still use hardware RAID? Windows exempted, as there's unfortunately nothing better available for Windows. (ReFS SW RAID might be ok, haven't checked its

People still use hardware RAID? Windows exempted, as there's unfortunately nothing better available for Windows. (ReFS SW RAID might be ok, haven't checked its state in a while. I know Microsoft is definitely working hard to fix this.) Hardware RAID is just data corruption waiting to happen, simply because there are more vulnerable components and interconnects along the data path. It's much better to checksum data while it's still protected by ECC RAM and CPU ECC caches – which is something only software RAID can do. Although even that is not a total guarantee. There's no longer reason to assume a slow SoC on a RAID adapter (even with XOR acceleration) would perform better than an Intel or AMD server CPU, as long as I/O path is not bandwidth limited.

wazoox 7y ago
Hardware RAID controllers always use ECC. Hardware RAID controllers haven't been using battery backed-up cache for the past decade, but supercapacitors and flash memory. So they provide better cache safety than any software RAID stack. Also hardware RAID allows for a more consistent performance under heavy system load in my experience.
- vardump 7y ago
  > Hardware RAID controllers always use ECC. This is not what I was talking about. Of course they do. The problem is that data needs to somehow be transferred to the adapter first, and this alone adds a point of failure. Controller can (and too often do) have hardware and firmware bugs. The less components the lower chances for corruption. > battery backed-up cache for the past decade, but supercapacitors and flash memory No difference to software RAID there. For example ZeusRAM and Optane are used for this. Of course high endurance SLC flash with supercaps is also an option. > Also hardware RAID allows for a more consistent performance under heavy system load in my experience. I'd tend to count this subjective, but I'll grant you this one, it's at least plausible. I haven't encountered this issue, though – today's hardware has insane bandwidth. SW RAID can saturate 100 Gbps ethernet, how much more performance you really need? Even if performance was slightly degraded, data integrity is more important.
  - throwaway_pdp09 7y ago
    > SW RAID can saturate 100 Gbps ethernet, how much more performance you really need? "640K ought to be enough for anyone". To answer more fully, you need as much as you need. You may not, but others will need more.
    - sjwright 7y ago
      Surely that’s obvious. Is there a non-obvious point you were trying to make?
    - PaulHoule 7y ago
      HW raid is going from the frying pan to the fire. Notoriously embedded CPUs have a much worse price/performance ratio than mainstream CPUs so almost inevitably they use a much weaker CPU than is needed. This is one reason why WD is involved in RISC-V, so they can escape this, and another reason why Intel's storage strategy (ex. optane) has worked out, because the main CPU has power to RAID NVMe SSDs and off-brand CPUs don't.
    - gibspaulding 7y ago
      This comparison seems a little disingenuous in this context. I read GP's post as saying simply that the current speed of SW RAID is enough to saturate current network speeds. If in the future we make some great breakthroughs in network technology without making the same progress in compute performance, then perhaps hardware RAID will become more relevant again, but right now software strikes a better balance.
      - throwaway_pdp09 7y ago
        Not my best post. I can't argue about bandwidth saturation aobviously but "how much more performance you really need?" is a silly question. It dismisses the possibility of it being not enough. But yes, not a good post.
        correctasshole 7y ago
- tedunangst 7y ago
  Somebody sure fucked up the labeling on the supercapacitor that came with my HP Enterprise smart array because it says li-ion battery on it.
e12e 7y ago
> Windows exempted, as there's unfortunately nothing better available for Windows. Something wrong with dynamic disks for software raid for the os in windows? https://docs.microsoft.com/en-us/windows/win32/fileio/basic-and-dynamic-disks https://docs.microsoft.com/en-us/windows/win32/fileio/basic-... Or storage spaces? (the latter i have not used) https://support.microsoft.com/en-gb/help/12438/windows-10-storage-spaces https://support.microsoft.com/en-gb/help/12438/windows-10-st...
- zerkten 7y ago
  I wanted to love Storage Spaces, but I had huge issue with a leak in the driver that caused the non-paged pool to grow on my machines. It got fixed, but I'm struggling with the idea of going back to it after the experience and stuck with dynamic disks.
- vardump 7y ago
  Create a mirrored volume out of two disks, both with ReFS (or ZFS) and dynamic disk software RAID. Remember to enable ReFS integrity streams! ZFS protects full integrity by default. NTFS supports only protecting metadata, file data integrity is not protected. Fill the volumes with known data. Randomly corrupt some blocks out of disks (but not from both disks at same offsets!). Observe which solution recovers and returns correct file data and which does not. Hint: you might want to stop using dynamic disk RAID after this experiment... Storage Spaces operates on a higher level, building on lower level volumes. I'd feel way more comfortable using it on ReFS, at least once it can be considered sufficiently mature in the future. See: https://docs.microsoft.com/en-us/windows-server/storage/refs/refs-overview#storage-spaces https://docs.microsoft.com/en-us/windows-server/storage/refs.... No qualms about trusting ZFS, it's very solid on good hardware. Of course running ZFS on Windows in production might be a Very Bad Idea, so do it on FreBSD, Solaris derivatives or Linux instead!
- Osiris 7y ago
  I'm using storage spaces on a home machine. It's very flexible, my machine is using 5 disks arranged in 2 separate logical volumes (one parity, one mirror). The write speeds are terrible, around 25MB/s. Read speeds are >100 MB/s.
SlowRobotAhead 7y ago
> People still use hardware RAID? Last I checked... VMWare VXSI does not support a software RAID at all anywhere. I think that’s still fairly common in server environments.
- vardump 7y ago
  I think you'd probably want to use SAN (or perhaps a high-end NAS) with your ESXi deployments. See for example https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/vsphere-esxi-vcenter-server-67-performance-best-practices.pdf https://www.vmware.com/content/dam/digitalmarketing/vmware/e...
  - SlowRobotAhead 7y ago
    Pretty great example of how it is it to spend other people’s money. The cost difference in two servers with 10-20x SAS drives vs a performance comparable SAN is not very close.
    - vardump 7y ago
      A classic pick any two out of three example. Compromises need to be made in real life. Not all VMs need high data integrity anyways. You get what you pay for.
- seized 7y ago
  That's exactly what vSAN is. Granted you need the license for it but that's "software RAID".
- xaldir 7y ago
  vSAN is software Raid. Having an HBA or a controller in pass-through is a requirement.
derefr 7y ago
> People still use hardware RAID? Windows exempted, as there's unfortunately nothing better available for Windows. I've always wondered whether anyone has gotten fed up with this, and decided to just virtualize Windows Server under Linux in order to feed the Windows Server VM a virtual disk that sits atop all the Linux storage-layer tech (but where otherwise the Windows VM gets all the rest of the computer's resources passed through.) (Come to think of it, I've considered doing the same for a Hackintosh at some point. It got too contrived, but I'd like to come back to it one day.)

wazoox 7y ago

Hardware RAID controllers always use ECC. Hardware RAID controllers haven't been using battery backed-up cache for the past decade, but supercapacitors and flash memory. So they provide better cache safety than any software RAID stack. Also hardware RAID allows for a more consistent performance under heavy system load in my experience.

vardump 7y ago
> Hardware RAID controllers always use ECC. This is not what I was talking about. Of course they do. The problem is that data needs to somehow be transferred to the adapter first, and this alone adds a point of failure. Controller can (and too often do) have hardware and firmware bugs. The less components the lower chances for corruption. > battery backed-up cache for the past decade, but supercapacitors and flash memory No difference to software RAID there. For example ZeusRAM and Optane are used for this. Of course high endurance SLC flash with supercaps is also an option. > Also hardware RAID allows for a more consistent performance under heavy system load in my experience. I'd tend to count this subjective, but I'll grant you this one, it's at least plausible. I haven't encountered this issue, though – today's hardware has insane bandwidth. SW RAID can saturate 100 Gbps ethernet, how much more performance you really need? Even if performance was slightly degraded, data integrity is more important.
- throwaway_pdp09 7y ago
  > SW RAID can saturate 100 Gbps ethernet, how much more performance you really need? "640K ought to be enough for anyone". To answer more fully, you need as much as you need. You may not, but others will need more.
  - sjwright 7y ago
    Surely that’s obvious. Is there a non-obvious point you were trying to make?
  - PaulHoule 7y ago
    HW raid is going from the frying pan to the fire. Notoriously embedded CPUs have a much worse price/performance ratio than mainstream CPUs so almost inevitably they use a much weaker CPU than is needed. This is one reason why WD is involved in RISC-V, so they can escape this, and another reason why Intel's storage strategy (ex. optane) has worked out, because the main CPU has power to RAID NVMe SSDs and off-brand CPUs don't.
  - gibspaulding 7y ago
    This comparison seems a little disingenuous in this context. I read GP's post as saying simply that the current speed of SW RAID is enough to saturate current network speeds. If in the future we make some great breakthroughs in network technology without making the same progress in compute performance, then perhaps hardware RAID will become more relevant again, but right now software strikes a better balance.
    - throwaway_pdp09 7y ago
      Not my best post. I can't argue about bandwidth saturation aobviously but "how much more performance you really need?" is a silly question. It dismisses the possibility of it being not enough. But yes, not a good post.
      - correctasshole 7y ago
tedunangst 7y ago
Somebody sure fucked up the labeling on the supercapacitor that came with my HP Enterprise smart array because it says li-ion battery on it.

e12e 7y ago

> Windows exempted, as there's unfortunately nothing better available for Windows. Something wrong with dynamic disks for software raid for the os in windows? https://docs.microsoft.com/en-us/windows/win32/fileio/basic-and-dynamic-disks https://docs.microsoft.com/en-us/windows/win32/fileio/basic-... Or storage spaces? (the latter i have not used) https://support.microsoft.com/en-gb/help/12438/windows-10-storage-spaces https://support.microsoft.com/en-gb/help/12438/windows-10-st...

zerkten 7y ago
I wanted to love Storage Spaces, but I had huge issue with a leak in the driver that caused the non-paged pool to grow on my machines. It got fixed, but I'm struggling with the idea of going back to it after the experience and stuck with dynamic disks.
vardump 7y ago
Create a mirrored volume out of two disks, both with ReFS (or ZFS) and dynamic disk software RAID. Remember to enable ReFS integrity streams! ZFS protects full integrity by default. NTFS supports only protecting metadata, file data integrity is not protected. Fill the volumes with known data. Randomly corrupt some blocks out of disks (but not from both disks at same offsets!). Observe which solution recovers and returns correct file data and which does not. Hint: you might want to stop using dynamic disk RAID after this experiment... Storage Spaces operates on a higher level, building on lower level volumes. I'd feel way more comfortable using it on ReFS, at least once it can be considered sufficiently mature in the future. See: https://docs.microsoft.com/en-us/windows-server/storage/refs/refs-overview#storage-spaces https://docs.microsoft.com/en-us/windows-server/storage/refs.... No qualms about trusting ZFS, it's very solid on good hardware. Of course running ZFS on Windows in production might be a Very Bad Idea, so do it on FreBSD, Solaris derivatives or Linux instead!
Osiris 7y ago
I'm using storage spaces on a home machine. It's very flexible, my machine is using 5 disks arranged in 2 separate logical volumes (one parity, one mirror). The write speeds are terrible, around 25MB/s. Read speeds are >100 MB/s.

SlowRobotAhead 7y ago

> People still use hardware RAID? Last I checked... VMWare VXSI does not support a software RAID at all anywhere. I think that’s still fairly common in server environments.

vardump 7y ago
I think you'd probably want to use SAN (or perhaps a high-end NAS) with your ESXi deployments. See for example https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/vsphere-esxi-vcenter-server-67-performance-best-practices.pdf https://www.vmware.com/content/dam/digitalmarketing/vmware/e...
- SlowRobotAhead 7y ago
  Pretty great example of how it is it to spend other people’s money. The cost difference in two servers with 10-20x SAS drives vs a performance comparable SAN is not very close.
  - vardump 7y ago
    A classic pick any two out of three example. Compromises need to be made in real life. Not all VMs need high data integrity anyways. You get what you pay for.
seized 7y ago
That's exactly what vSAN is. Granted you need the license for it but that's "software RAID".
xaldir 7y ago
vSAN is software Raid. Having an HBA or a controller in pass-through is a requirement.

derefr 7y ago

> People still use hardware RAID? Windows exempted, as there's unfortunately nothing better available for Windows. I've always wondered whether anyone has gotten fed up with this, and decided to just virtualize Windows Server under Linux in order to feed the Windows Server VM a virtual disk that sits atop all the Linux storage-layer tech (but where otherwise the Windows VM gets all the rest of the computer's resources passed through.) (Come to think of it, I've considered doing the same for a Hackintosh at some point. It got too contrived, but I'd like to come back to it one day.)