5 ms·

Seems like quite a bit of complexity for a dubious win. Today's CPUs are REALLY fast, even a single core of 64 or more cores that are common on servers today:

by sliken 2y ago

Seems like quite a bit of complexity for a dubious win. Today's CPUs are REALLY fast, even a single core of 64 or more cores that are common on servers today: [ 478.047970] raid6: avx2x2 gen() 60473 MB/s [ 478.115971] raid6: avx512x1 gen() 53469 MB/s [ 478.149971] raid6: avx512x2 gen() 57067 MB/s Especially since the data is coming from the CPU anyways, so likely caches are warm. It also means that node you have to send a stripe of data to a single NVME, which likely has much less than 60GB/sec of checksum speed, then initiate transfers to every other drive in the stripe. Not to mention the NVME drive likely doesn't have ECC memory and any resulting memory errors are unlikely to be visible to the OS. Just seems like hardware RAID with all the same problems, likely not as fast as software RAID, harder to manage, a unique set of tools per vendor, harder to have global spares, and doesn't work with filesystems that do their own redundancy like ZFS.

everfrustrated 2y ago
The problem is not the CPU per se, but the PCI bandwidth congestion between CPU<->PCI lanes. These NVME drives can talk directly to each other for raid which means a much larger total bandwidth is available, and potentially improved latency also. RDMA means you might not be serving via CPU at all.
- wmf 2y ago
  PCIe P2P transfers go through the CPU so... no? I think what it's saving is main memory bandwidth.
  - deleted 2y ago
    [deleted]
  - zamadatix 2y ago
    > PCIe P2P transfers go through the CPU PCIe P2P transfers can go direct through a downstream PCIe switch such as found on chipsets without having to bump back up through the CPU.
    - namibj 2y ago
      And even with e.g. AMD Matisse, aka Desktop Ryzen Zen3 on AM4, it turns around in the PCIe root complex instead of consuming infinity fabric bandwidth.
- sliken 2y ago
  The difference isn't that big though. Sure with software RAID you write 1GB to a 8 disk RAID6 you write 8/6 x 1GB = 1.33GB. But with RAID offload you nearly double the NVME bandwidth (n - 1) x 2 consumed. I also wonder, if you have 8 NVMe, write to a stripe to one, it does the RAID calc and sends each disk the share of the stripe. What happens if the master NVMe dies? It's not really a RAID if a single disk can kill the RAID.
zamadatix 2y ago
Based on the numbers from the article it seems the problem is less how fast a CPU core can crunch numbers and more how much extra memory bandwidth it consumes to do so. Testing the AVX throughput of a single core in a storage only test skips that consideration because there is no memory bandwidth contention or usage consideration.
- sliken 2y ago
  Sure, but to write 1GB you stream 1GB from ram -> CPU in either case. With software RAID you do the calcs (60GB/sec per core) and then write 1.3GB/sec to the storage controller. Just doesn't seem that much of a difference, the CPU overhead is near zero (actual I/O / 64*60GB), and writing an extra 1/3rd for the redundancy data seems in the noise for normal server loads. Not to mention I'd expect the parity calculations to be MUCH slower on the NVMe controllers.
  - creshal 2y ago
    I guess the interesting usecase would be to combine this with other hardware accelerates and do DMA between devices, e.g., stream network data directly to a RAID without ever touching the main CPU, after some initial setup work.
  - zamadatix 2y ago
    Your assumption that, from a memory perspective, the stream goes from 1 GB RAM Read -> Write to Disk to 1 GB RAM read -> Calculation -> Write to disk does not hold. There are intermediate forms of data that end up writing back to RAM then to disk. This is what the article is talking about here: > upwards of 90% reduction in system DRAM utilization
    - sliken 2y ago
      My understanding is that it's something like: stripe = read_from_ram(*ptr) # usually between 128k and 256k blobs[]=do_raid_calc(stripe) # blobs usually 25% to 33% larger than stripe for i in drives write(drive=i,blobs[i]) The above should be relatively cache friendly, my Zen 4 desktop (1 gen old) has 128MB of L3 cache, enough for 1000 ish stripes. > upwards of 90% reduction in system DRAM utilization That seems unbelievable, most ram isn't spend for anything I/O related let alone RAID releated. Now if it's 90% reduction in system DRAM utilization by RAID, sure. But that seems like a very small fraction of all ram. Even if 10,000 stripes are in flight simultaneously to 100s of drives that's only 2.5GB or 1% of a servers ram (256GB or more seems common). Especially since 2/3rd of that would be in ram even with hardware RAID. Not like the buffer/page cache which might reach 50% of ram has the extra RAID in data in it.
topspin 2y ago
> and doesn't work with filesystems that do their own redundancy like ZFS Really? You somehow can't create a ZFS file system on an hardware RAID block device? Seems like that means the hardware RAID isn't the otherwise transparent block device it's supposed to be for the OS and whatever file systems it cares to employ. You're concerns about management, tools and spares are correct for many use cases. Some uses cases, like cloud operators that don't suffer the burdens of long term management at that level of detail (where entire racks and generations of hardware are cycled in/out as a working unit, with ample spares at hand, under contract) won't care about that. They'll care about the nice efficiency gain. When you operate like that you can accommodate sophisticated integration such as this for efficiency gains.
- sliken 2y ago
  > Really? You somehow can't create a ZFS file system on an hardware RAID block device? Sure you can do it, have two layers of checksums and a volume manager on top of a volume manager. But ZFS is designed to talk directly to block devices and try to detect and complain about the numerous failure modes. Like say a parity calc that goes awry because of a memory error. For this and other reasons it's recommended that even with Hardware RAID it's recommended to configure it in JBOD mode. I've also seen numerous cases where software RAID on top of hardware RAID running in JBOD mode is faster than just using hardware RAID. > When you operate like that you can accommodate sophisticated integration such as this for efficiency gains. Sure, if there are efficiency gains. If the strong bottleneck for writing to the controller is your limiting factor you might get a 33% increase in I/O. But for that to be true you need: * The bottleneck not to be elsewhere * The controller inside a NVMe device (often passively cooled) to be faster than the one on the CPU * The bandwidth between the PCI controller or PCIe switch and the NVMe controller to not care about a 2x increase in needed bandwidth Seems unlikely to me.

by sliken 2y ago

everfrustrated 2y ago

The problem is not the CPU per se, but the PCI bandwidth congestion between CPU<->PCI lanes. These NVME drives can talk directly to each other for raid which means a much larger total bandwidth is available, and potentially improved latency also. RDMA means you might not be serving via CPU at all.

wmf 2y ago
PCIe P2P transfers go through the CPU so... no? I think what it's saving is main memory bandwidth.
- deleted 2y ago
  [deleted]
- zamadatix 2y ago
  > PCIe P2P transfers go through the CPU PCIe P2P transfers can go direct through a downstream PCIe switch such as found on chipsets without having to bump back up through the CPU.
  - namibj 2y ago
    And even with e.g. AMD Matisse, aka Desktop Ryzen Zen3 on AM4, it turns around in the PCIe root complex instead of consuming infinity fabric bandwidth.
sliken 2y ago
The difference isn't that big though. Sure with software RAID you write 1GB to a 8 disk RAID6 you write 8/6 x 1GB = 1.33GB. But with RAID offload you nearly double the NVME bandwidth (n - 1) x 2 consumed. I also wonder, if you have 8 NVMe, write to a stripe to one, it does the RAID calc and sends each disk the share of the stripe. What happens if the master NVMe dies? It's not really a RAID if a single disk can kill the RAID.

zamadatix 2y ago

Based on the numbers from the article it seems the problem is less how fast a CPU core can crunch numbers and more how much extra memory bandwidth it consumes to do so. Testing the AVX throughput of a single core in a storage only test skips that consideration because there is no memory bandwidth contention or usage consideration.

sliken 2y ago
Sure, but to write 1GB you stream 1GB from ram -> CPU in either case. With software RAID you do the calcs (60GB/sec per core) and then write 1.3GB/sec to the storage controller. Just doesn't seem that much of a difference, the CPU overhead is near zero (actual I/O / 64*60GB), and writing an extra 1/3rd for the redundancy data seems in the noise for normal server loads. Not to mention I'd expect the parity calculations to be MUCH slower on the NVMe controllers.
- creshal 2y ago
  I guess the interesting usecase would be to combine this with other hardware accelerates and do DMA between devices, e.g., stream network data directly to a RAID without ever touching the main CPU, after some initial setup work.
- zamadatix 2y ago
  Your assumption that, from a memory perspective, the stream goes from 1 GB RAM Read -> Write to Disk to 1 GB RAM read -> Calculation -> Write to disk does not hold. There are intermediate forms of data that end up writing back to RAM then to disk. This is what the article is talking about here: > upwards of 90% reduction in system DRAM utilization
  - sliken 2y ago
    My understanding is that it's something like: stripe = read_from_ram(*ptr) # usually between 128k and 256k blobs[]=do_raid_calc(stripe) # blobs usually 25% to 33% larger than stripe for i in drives write(drive=i,blobs[i]) The above should be relatively cache friendly, my Zen 4 desktop (1 gen old) has 128MB of L3 cache, enough for 1000 ish stripes. > upwards of 90% reduction in system DRAM utilization That seems unbelievable, most ram isn't spend for anything I/O related let alone RAID releated. Now if it's 90% reduction in system DRAM utilization by RAID, sure. But that seems like a very small fraction of all ram. Even if 10,000 stripes are in flight simultaneously to 100s of drives that's only 2.5GB or 1% of a servers ram (256GB or more seems common). Especially since 2/3rd of that would be in ram even with hardware RAID. Not like the buffer/page cache which might reach 50% of ram has the extra RAID in data in it.

topspin 2y ago

> and doesn't work with filesystems that do their own redundancy like ZFS Really? You somehow can't create a ZFS file system on an hardware RAID block device? Seems like that means the hardware RAID isn't the otherwise transparent block device it's supposed to be for the OS and whatever file systems it cares to employ. You're concerns about management, tools and spares are correct for many use cases. Some uses cases, like cloud operators that don't suffer the burdens of long term management at that level of detail (where entire racks and generations of hardware are cycled in/out as a working unit, with ample spares at hand, under contract) won't care about that. They'll care about the nice efficiency gain. When you operate like that you can accommodate sophisticated integration such as this for efficiency gains.

sliken 2y ago
> Really? You somehow can't create a ZFS file system on an hardware RAID block device? Sure you can do it, have two layers of checksums and a volume manager on top of a volume manager. But ZFS is designed to talk directly to block devices and try to detect and complain about the numerous failure modes. Like say a parity calc that goes awry because of a memory error. For this and other reasons it's recommended that even with Hardware RAID it's recommended to configure it in JBOD mode. I've also seen numerous cases where software RAID on top of hardware RAID running in JBOD mode is faster than just using hardware RAID. > When you operate like that you can accommodate sophisticated integration such as this for efficiency gains. Sure, if there are efficiency gains. If the strong bottleneck for writing to the controller is your limiting factor you might get a 33% increase in I/O. But for that to be true you need: * The bottleneck not to be elsewhere * The controller inside a NVMe device (often passively cooled) to be faster than the one on the CPU * The bandwidth between the PCI controller or PCIe switch and the NVMe controller to not care about a 2x increase in needed bandwidth Seems unlikely to me.