3 ms·

Kioxia Demonstrates Raid Offload Scheme for NVMe Drives

36 points by vanburen 2y ago

mrktf 2y ago
I imagine these kind of schemes can be implemented as sort of on device eBPF filter (in layman terms CUDA, but for storage). It would allow deeper integration with system for example have hardware accelerated/integrated lvm (obviously speed would depend on use case, less win for thin volumes, more advantages for raid and so on). Or from other side have deeper integration with filesystems such as zfs, btrfs, bcachefs.
- rkagerer 2y ago
  Is this similar to Graid's products that have been out for a while? They basically use a GPU as a raid controller
  - Dylan16807 2y ago
    A GPU RAID card does give you some flexibility benefits. But it's also bottlenecked by the single slot.
- benlwalker 2y ago
  We tried to standardize exactly this - eBPF programs offloaded onto the device. The NVMe standard now has a lot of infrastructure for this standardized, including commands to discover device memory topology, transfer to/from that memory, and discover and upload programs. But one of the blockers is that eBPF isn't itself standardized. The other blockers are vendors ready and willing to build these devices and customers ready to buy them in volume. The extra compute ability will introduce some extra cost. I'm still hopeful that we see it happen some day.
  - doctorpangloss 2y ago
    > The NVMe standard now has a lot of infrastructure for this standardized, including commands to discover device memory topology, transfer to/from that memory, and discover and upload programs. On the other hand, Windows and Linux still cannot just upgrade the vast majority of firmwares on NVMe devices, least of all consumer ones, despite being completely and utterly standardized. You have to wonder, if Samsung makes bullshit, and then this https://github.com/chrivers/samsung-firmware-magic https://github.com/chrivers/samsung-firmware-magic becomes part of the ecosystem, why trust the vendors with anything else?
    - znpy 2y ago
      I think i remember upgrading the nvme disk firmware in work dell laptop (dell latitude 7390) from 2019 using fwupd some years ago (not more than 3 years ago). Also i think i remember fixing (upgrading?) the firmware on a crucial ssd like 5 or 6 years ago using some live linux system (downloaded off the crucial website i think?) Not sure about windows, but linux is getting incredibly better at this.
    - yencabulator 2y ago
      That repo was last updated 3 years ago, things seem to have changed since. There seem to be a lot of ISOs for download at https://semiconductor.samsung.com/consumer-storage/support/tools/ https://semiconductor.samsung.com/consumer-storage/support/t... The ISO contains an EFI filesystem with grub etc and boots fine without needing Windows.
  - sroussey 2y ago
    So, could you upload malware to the drive that way?
    - benlwalker 2y ago
      The eBPF programs are strictly bounded. And they're scoped to their own memory that you have to pre-load from the actual storage with separate commands issued from the CPU (presumably from the kernel driver which is doing access control checks). It's no different than uploading a shader to a GPU. You can burn resources but that's about the extent of the damage you can cause.
      - ChocolateGod 2y ago
        It was only a week ago that Google disclosed an exploit to get a root shell via eBPF. https://bughunters.google.com/blog/6303226026131456/a-deep-dive-into-cve-2023-2163-how-we-found-and-fixed-an-ebpf-linux-kernel-vulnerability https://bughunters.google.com/blog/6303226026131456/a-deep-d... I wouldn't want random applications (or web pages) to be able to load eBPF modules in the same way they can send shaders to a GPU through a graphics driver.
- cm2187 2y ago
  What I don't get is that RAID5 is a simple xor. It should be a trivial operation, that would be equally trivial to hardware accelerate. What I am the most puzzled by is how parity (i.e. RAID5) is so bad in windows storage space. A modern CPU should be able to xor data at several gigabytes per second. And it seems that even by optimizing the block sizes, windows storage space parity caps at a couple hundred MB/s.
deleted 2y ago
[deleted]
sliken 2y ago
Seems like quite a bit of complexity for a dubious win. Today's CPUs are REALLY fast, even a single core of 64 or more cores that are common on servers today: [ 478.047970] raid6: avx2x2 gen() 60473 MB/s [ 478.115971] raid6: avx512x1 gen() 53469 MB/s [ 478.149971] raid6: avx512x2 gen() 57067 MB/s Especially since the data is coming from the CPU anyways, so likely caches are warm. It also means that node you have to send a stripe of data to a single NVME, which likely has much less than 60GB/sec of checksum speed, then initiate transfers to every other drive in the stripe. Not to mention the NVME drive likely doesn't have ECC memory and any resulting memory errors are unlikely to be visible to the OS. Just seems like hardware RAID with all the same problems, likely not as fast as software RAID, harder to manage, a unique set of tools per vendor, harder to have global spares, and doesn't work with filesystems that do their own redundancy like ZFS.
- everfrustrated 2y ago
  The problem is not the CPU per se, but the PCI bandwidth congestion between CPU<->PCI lanes. These NVME drives can talk directly to each other for raid which means a much larger total bandwidth is available, and potentially improved latency also. RDMA means you might not be serving via CPU at all.
  - wmf 2y ago
    PCIe P2P transfers go through the CPU so... no? I think what it's saving is main memory bandwidth.
    - deleted 2y ago
      [deleted]
    - zamadatix 2y ago
      > PCIe P2P transfers go through the CPU PCIe P2P transfers can go direct through a downstream PCIe switch such as found on chipsets without having to bump back up through the CPU.
      - 2y ago
ein0p 2y ago
I’d much prefer NVMe colocated compute. Imagine a columnar storage engine able to filter and aggregate data during scans without reading it through PCIe, for example.
- jakedata 2y ago
  ScaleFlux https://scaleflux.com https://scaleflux.com computational storage might offer some of what you are imagining. Their NVMe drives have onboard ARM cores and perform hardware compression and advanced flash management with no drivers beyond standard NVMe. I believe you can tap into the computational capabilities with additional code.
przemub 2y ago
When I first bought Kioxia flash I thought it's a random Chinese knockoff. Shame they ditched Toshiba brand on these.
znpy 2y ago
I wonder how (if?) this will interact and integrate with the current software stack and the various volume-managing filesystems (zfs but also btrfs).
- wmf 2y ago
  It probably won't. These clever tricks usually don't come to market.
jamesfmilne 2y ago
HW RAID is dead, they need to get over it. We've had good experience with Xinnor, but it's a shame it's proprietary. I'd love to see a high-performance open-source erasure coding solution for NVMe. The built in offerings in Linux are not cutting it.
baruch 2y ago
NVMe drives fail at a fairly low rate so this is an optimization for a very small edge case and since they are also very fast it's not like you'll be doing a rebuild for 6+ hours like with HDDs. It also doesn't change anything for distributed storage.
- DaiPlusPlus 2y ago
  > NVMe drives fail at a fairly low rate But they still fail. Backups are great and all, but for hardware-failure nothing beats redundancy (while RAID1, RAID5, etc allow for faster reads - I don't know how-often NVMe SSDs saturate their PCIe links though...). Granted, you don't need hardware RAID for that (and HostRAID is a joke, lol): we still want redundancy, but today you'd do it with ZFS or similar so you aren't locked-in to some HW RAID vendor, or suffer the ironic consequences of having non-redundant HW RAID controllers.
  - okasaki 2y ago
    You would now typically use a distributed system for redundancy. Then you don't necessarily need RAID.
    - DaiPlusPlus 2y ago
      How do I fit a distributed system into my laptop?
  - iam-TJ 2y ago
    I have an NVME device that very rarely literally (but figuratively) falls off the PCI port and disappears [0]. It is one of several Physical Volumes (PV) in a Logical Volume Management (LVM) Volume Group (VG) that backs several RAID-1 mirror Logical Volumes (LV). When it drops off file-systems writes to the LVs are blocked and reads can also fail but the system survives sufficiently to do a controlled power off/on that recovers it. In some cases the LVs pair up a spinning disk with the NVME but due to how I've configured the LV the spinner is read-mostly and the NVME is write-mostly (RAID member syncing is delayed and in background). There isn't too much noticeable latency except for things like `git log -Sneedle` - and worth it for the resilience. [0] first time it happened it was spiders that had taken up residence around the M2 header and CPU (nice and warm!) and causing dust trails allowing current leakage between contacts (yes, I did do microscopic examination because I could not identify any other cause) that a simple blast with the air-compressor resolved. Later incidents turn out to be physical stress due to extreme thermal expansion and contraction as best as I can tell - ambient air temperature can fluctuate from 14C to 40C and back over 18 hours. Re-seating the M2 adapter fixes it for a a few months before it starts again! All NVME SMART self-tests pass; the failure is of the link not the storage - effectively being removed from the PCIe port. Firmware was at one stage suspected, although it had been fine for a couple of years on the same version, but updates haven't changed it in any way. ASPM is disabled.

36 points by vanburen 2y ago

I imagine these kind of schemes can be implemented as sort of on device eBPF filter (in layman terms CUDA, but for storage). It would allow deeper integration with system for example have hardware accelerated/integrated lvm (obviously speed would depend on use case, less win for thin volumes, more advantages for raid and so on). Or from other side have deeper integration with filesystems such as zfs, btrfs, bcachefs.

rkagerer 2y ago
Is this similar to Graid's products that have been out for a while? They basically use a GPU as a raid controller
- Dylan16807 2y ago
  A GPU RAID card does give you some flexibility benefits. But it's also bottlenecked by the single slot.
benlwalker 2y ago
We tried to standardize exactly this - eBPF programs offloaded onto the device. The NVMe standard now has a lot of infrastructure for this standardized, including commands to discover device memory topology, transfer to/from that memory, and discover and upload programs. But one of the blockers is that eBPF isn't itself standardized. The other blockers are vendors ready and willing to build these devices and customers ready to buy them in volume. The extra compute ability will introduce some extra cost. I'm still hopeful that we see it happen some day.
- doctorpangloss 2y ago
  > The NVMe standard now has a lot of infrastructure for this standardized, including commands to discover device memory topology, transfer to/from that memory, and discover and upload programs. On the other hand, Windows and Linux still cannot just upgrade the vast majority of firmwares on NVMe devices, least of all consumer ones, despite being completely and utterly standardized. You have to wonder, if Samsung makes bullshit, and then this https://github.com/chrivers/samsung-firmware-magic https://github.com/chrivers/samsung-firmware-magic becomes part of the ecosystem, why trust the vendors with anything else?
  - znpy 2y ago
    I think i remember upgrading the nvme disk firmware in work dell laptop (dell latitude 7390) from 2019 using fwupd some years ago (not more than 3 years ago). Also i think i remember fixing (upgrading?) the firmware on a crucial ssd like 5 or 6 years ago using some live linux system (downloaded off the crucial website i think?) Not sure about windows, but linux is getting incredibly better at this.
  - yencabulator 2y ago
    That repo was last updated 3 years ago, things seem to have changed since. There seem to be a lot of ISOs for download at https://semiconductor.samsung.com/consumer-storage/support/tools/ https://semiconductor.samsung.com/consumer-storage/support/t... The ISO contains an EFI filesystem with grub etc and boots fine without needing Windows.
- sroussey 2y ago
  So, could you upload malware to the drive that way?
  - benlwalker 2y ago
    The eBPF programs are strictly bounded. And they're scoped to their own memory that you have to pre-load from the actual storage with separate commands issued from the CPU (presumably from the kernel driver which is doing access control checks). It's no different than uploading a shader to a GPU. You can burn resources but that's about the extent of the damage you can cause.
    - ChocolateGod 2y ago
      It was only a week ago that Google disclosed an exploit to get a root shell via eBPF. https://bughunters.google.com/blog/6303226026131456/a-deep-dive-into-cve-2023-2163-how-we-found-and-fixed-an-ebpf-linux-kernel-vulnerability https://bughunters.google.com/blog/6303226026131456/a-deep-d... I wouldn't want random applications (or web pages) to be able to load eBPF modules in the same way they can send shaders to a GPU through a graphics driver.
cm2187 2y ago
What I don't get is that RAID5 is a simple xor. It should be a trivial operation, that would be equally trivial to hardware accelerate. What I am the most puzzled by is how parity (i.e. RAID5) is so bad in windows storage space. A modern CPU should be able to xor data at several gigabytes per second. And it seems that even by optimizing the block sizes, windows storage space parity caps at a couple hundred MB/s.

deleted 2y ago

[deleted]

sliken 2y ago

Seems like quite a bit of complexity for a dubious win. Today's CPUs are REALLY fast, even a single core of 64 or more cores that are common on servers today: [ 478.047970] raid6: avx2x2 gen() 60473 MB/s [ 478.115971] raid6: avx512x1 gen() 53469 MB/s [ 478.149971] raid6: avx512x2 gen() 57067 MB/s Especially since the data is coming from the CPU anyways, so likely caches are warm. It also means that node you have to send a stripe of data to a single NVME, which likely has much less than 60GB/sec of checksum speed, then initiate transfers to every other drive in the stripe. Not to mention the NVME drive likely doesn't have ECC memory and any resulting memory errors are unlikely to be visible to the OS. Just seems like hardware RAID with all the same problems, likely not as fast as software RAID, harder to manage, a unique set of tools per vendor, harder to have global spares, and doesn't work with filesystems that do their own redundancy like ZFS.

everfrustrated 2y ago
The problem is not the CPU per se, but the PCI bandwidth congestion between CPU<->PCI lanes. These NVME drives can talk directly to each other for raid which means a much larger total bandwidth is available, and potentially improved latency also. RDMA means you might not be serving via CPU at all.
- wmf 2y ago
  PCIe P2P transfers go through the CPU so... no? I think what it's saving is main memory bandwidth.
  - deleted 2y ago
    [deleted]
  - zamadatix 2y ago
    > PCIe P2P transfers go through the CPU PCIe P2P transfers can go direct through a downstream PCIe switch such as found on chipsets without having to bump back up through the CPU.
    - 2y ago

ein0p 2y ago

I’d much prefer NVMe colocated compute. Imagine a columnar storage engine able to filter and aggregate data during scans without reading it through PCIe, for example.

jakedata 2y ago
ScaleFlux https://scaleflux.com https://scaleflux.com computational storage might offer some of what you are imagining. Their NVMe drives have onboard ARM cores and perform hardware compression and advanced flash management with no drivers beyond standard NVMe. I believe you can tap into the computational capabilities with additional code.

przemub 2y ago

When I first bought Kioxia flash I thought it's a random Chinese knockoff. Shame they ditched Toshiba brand on these.

znpy 2y ago

I wonder how (if?) this will interact and integrate with the current software stack and the various volume-managing filesystems (zfs but also btrfs).

wmf 2y ago
It probably won't. These clever tricks usually don't come to market.

jamesfmilne 2y ago

HW RAID is dead, they need to get over it. We've had good experience with Xinnor, but it's a shame it's proprietary. I'd love to see a high-performance open-source erasure coding solution for NVMe. The built in offerings in Linux are not cutting it.

baruch 2y ago

NVMe drives fail at a fairly low rate so this is an optimization for a very small edge case and since they are also very fast it's not like you'll be doing a rebuild for 6+ hours like with HDDs. It also doesn't change anything for distributed storage.

DaiPlusPlus 2y ago
> NVMe drives fail at a fairly low rate But they still fail. Backups are great and all, but for hardware-failure nothing beats redundancy (while RAID1, RAID5, etc allow for faster reads - I don't know how-often NVMe SSDs saturate their PCIe links though...). Granted, you don't need hardware RAID for that (and HostRAID is a joke, lol): we still want redundancy, but today you'd do it with ZFS or similar so you aren't locked-in to some HW RAID vendor, or suffer the ironic consequences of having non-redundant HW RAID controllers.
- okasaki 2y ago
  You would now typically use a distributed system for redundancy. Then you don't necessarily need RAID.
  - DaiPlusPlus 2y ago
    How do I fit a distributed system into my laptop?
- iam-TJ 2y ago
  I have an NVME device that very rarely literally (but figuratively) falls off the PCI port and disappears [0]. It is one of several Physical Volumes (PV) in a Logical Volume Management (LVM) Volume Group (VG) that backs several RAID-1 mirror Logical Volumes (LV). When it drops off file-systems writes to the LVs are blocked and reads can also fail but the system survives sufficiently to do a controlled power off/on that recovers it. In some cases the LVs pair up a spinning disk with the NVME but due to how I've configured the LV the spinner is read-mostly and the NVME is write-mostly (RAID member syncing is delayed and in background). There isn't too much noticeable latency except for things like `git log -Sneedle` - and worth it for the resilience. [0] first time it happened it was spiders that had taken up residence around the M2 header and CPU (nice and warm!) and causing dust trails allowing current leakage between contacts (yes, I did do microscopic examination because I could not identify any other cause) that a simple blast with the air-compressor resolved. Later incidents turn out to be physical stress due to extreme thermal expansion and contraction as best as I can tell - ambient air temperature can fluctuate from 14C to 40C and back over 18 hours. Re-seating the M2 adapter fixes it for a a few months before it starts again! All NVME SMART self-tests pass; the failure is of the link not the storage - effectively being removed from the PCIe port. Firmware was at one stage suspected, although it had been fine for a couple of years on the same version, but updates haven't changed it in any way. ASPM is disabled.