4 ms·

Would this reduce the need to have expensive and power hungry RAID/HBA cards? I would assume splitting nvme/PCIe is a lot simpler than PCIe to SATA.

toast0 5y ago
Seems like it. The article mentions a PCIe switch, but PCIe bifurcation may also be an option. (That's splitting a multiple lane slot into multiple slots, requires system firmware suppport though)
- formerly_proven 5y ago
  Bifurcation has never been a thing on desktop platforms and even most entry-level (single socket, desktop-equivalent) servers don't support it. It seems to be reserved for HEDT and real server platforms. (This is of course purely a market segmentation decision by Intel/AMD).
  - toast0 5y ago
    My A520 mini-itx board supports it; can't get anymore desktop than that. Although, that has limited options, I think I can do either 2 x8 or one x8 and two x4. For this, it looks like each drive is expected to be x1, so you'd want one x16 to 16 x1s. It's doable, but not without mucking about in the firmware (either by the OEM, or dedicated enthusiasts), so a PCIe switch is probably advisable.
    - formerly_proven 5y ago
      Ah I see. It seems like previously bifurcation was not qualified for anything but the X-series chipset, but in the 500 series it's qualified for all. On top of that, it seems like some boards just allowed it regardless in prior generations. Another complication is of course that the non-PEG slots on the (non-HEDT) platforms are usually electrically only x4 or x1, so bifurcation really only makes sense in the PEG.
      - wtallis 5y ago
        PCIe root ports in CPUs are generally designed to provide an x16 with bifurcation down to x4x4x4x4 (or merely x8x4x4 for Intel consumer CPUs). Large-scale PCIe switches also commonly support bifurcation only down to x4 or sometimes x2, though x1 may start catching on with PCIe gen5. Smaller PCIe switches and motherboard chipsets usually support link widths from x4 down to x1. Treating each lane individually goes hand in hand with the fact that many of the lanes provided by a motherboard chipset can be reconfigured between some combination of PCIe, SATA and USB: they design a multi-purpose PHY, put down one or two dozen copies of it at the perimeter of the die, and connect them to an appropriate variety of MACs.
        formerly_proven 5y ago
        Yeah, but what I meant above was that if you only have an x4 slot electrically, sticking in an x16 -> 4x M.2 riser isn't going to do a whole lot, because the 12 lanes of 3 out of 4 slots aren't hooked up to anything. So in this scenario you'd really want a riser with a switch in it instead (which are more expensive than almost all motherboards). So on the consumer platforms that give you two PEGs best you could do while still having a GPU is stick that riser in the second PEG and use the x8/x8 split. Now the question becomes whether the UEFI allows you to use the x8/x8 bifurcation meant for dual GPU or similar use in an x8/(x4+x4) triple bifurcation kind of setup. Realistically this entire thing just doesn't make a lot of sense on the consumer platforms because they just don't have enough PCIe lanes out of the CPU. Intel used to be slightly worse here with 20 (of which 4 are reserved for the PCH but you know that), while AM4 has 28 (4 for the I/O hub again). On an HEDT platform with 40+ lanes though... (When I say bifurcation I meant bifurcation of a slot on the mainboard, not the various ways the slots and ports on the board itself can be configured, though that's technically bifurcation as well (or even switching protocols)
        wtallis 5y ago
        > Yeah, but what I meant above was that if you only have an x4 slot electrically, sticking in an x16 -> 4x M.2 riser isn't going to do a whole lot, because the 12 lanes of 3 out of 4 slots aren't hooked up to anything. So in this scenario you'd really want a riser with a switch in it instead (which are more expensive than almost all motherboards). True; but given how PCIe speeds are no longer stalled, we may soon see motherboards offering an x4 slot that can be operated as x1x1x1x1. Currently, the only risers you're likely to find that split a slot into independent x1 ports are intended for crypto mining, and they require switches. A passive (or retimer-only) quad-M.2 riser that only provides one PCIe lane per drive currently sounds a bit bandwidth-starved and wouldn't work with current motherboards. But given PCIe gen5 SSDs or widespread availability of PCIe-native hard drives, those uses for an x4 slot will start to make sense.
  - wtallis 5y ago
    PCIe bifurcation works fine on AMD consumer platforms. I've used a passive quad-M.2 riser card on an AMD B550 motherboard with no trouble other than changing a firmware setting to turn on bifurcation. It's only Intel that is strict about this aspect of product segmentation.
  - rasz 5y ago
    Intel supported PCIE bifurcation it on desktop chipsets up to Sandy Bridge. Just like Intel once supported memory parity on desktop chipsets until deciding to use it to upsell server products. Recent Intel chipsets were locked to Intel SSDs and only some magic hardware DRM handshake unlocked bifurcation https://www.asus.com/us/support/FAQ/1037507/ https://www.asus.com/us/support/FAQ/1037507/ But apparently AMD competition forced their hand and at lest some got unlocked https://forum.level1techs.com/t/new-bios-update-unlocks-4x-pcie-x4-bifurcation-and-raid-on-existing-x299-motherboards/1510701 https://forum.level1techs.com/t/new-bios-update-unlocks-4x-p... On AMD front there doesnt seem to be any problems, plenty of desktop boards with unlocked bifurcation BIOS menus.
wtallis 5y ago
PCIe switches are a lot simpler and more standardized than RAID and HBA controllers, but I'm not sure they're any cheaper for similar bandwidth and port counts. Broadcom/Avago/PLX and Microchip/Microsemi are the only two vendors for large-scale current generation PCIe switches, and starting with PCIe gen3 they decided to price them way out of the consumer market, contributing to the disappearance of multi-GPU from gaming PCs.
- semi-extrinsic 5y ago
  The main reason for multi-GPUs decline is that single GPU performance has grown much faster than both CPUs and RAM speeds. Going multi-GPU on a high-end GPU today lets you run a AAA game at 4K with all the eye candy at 150 fps rather than 110 fps, i.e. visually undistinguishable. And at the low end, it's always better performance per dollar to buy a single higher-end card. Of course this is all orthogonal to your point of PCIe switches. Anyone thinking about multi-GPU will buy a motherboard with at least 2 PCIe 4.0 x16 slots anyway, no switches required. You can even get motherboards with 2 PCIe 5.0 x16 slots without breaking the bank.
  - wtallis 5y ago
    > Anyone thinking about multi-GPU will buy a motherboard with at least 2 PCIe 4.0 x16 slots anyway, no switches required. You can even get motherboards with 2 PCIe 5.0 x16 slots without breaking the bank. I'm pretty sure you're referring to motherboards that have two slots which are mechanically x16, but actually using the second slot borrows half of the lanes from the first slot, leaving you operating at x8+x8. This is not the same as having a full 48+ lane PCIe packet switch between the CPU and two slots each wired for x16, which was very common for high-end consumer motherboards in the PCIe gen2 era (eg. with NVIDIA's NF200 switch). These days, you can get 32+ lanes straight from the CPU to enable two x16 slots, but that requires stepping up to a bigger CPU socket than the mainstream consumer CPU+motherboard platforms use. And I don't think you can yet buy a motherboard with two PCIe gen5 x16 slots at any price: as far as I can tell, none of the Intel Alder Lake motherboards announced so far include PCIe switches, because such switches have been cost-prohibitive for several generations now.
mgerdts 5y ago
I don't think so. The reason that I would use a RAID card with HDDs is to get the benefit of a battery backed write cache, optimally performing RAID operations. I've only seen one RAID card that supports connecting to NVMe drives. It is terribly expensive and aimed supporting the speed of flash-based NVMe drives. If you want a battery backed write cache with flash-based NVMe, get an enterprise drive with power loss protection, and you are golden. You don't generally need the write cache to get reasonable throughput, but you can save a few microseconds in write latency. I suspect that NVMe HDDs will have the same amount of write cache that existing SATA/SAS HDDs have. If you feel you need a RAID card for RAID and/or cache with SAS or SATA HDDs, you will probably have the same need for a RAID card with NVMe HDDs. Both Intel and AMD support limited hardware RAID for NVMe natively on some of their CPUs. I don't think this has any of the cache benefits you normally have with a RAID card. That is, it may work well with flash-based NVMe drives, particularly those with power loss protection. But really, if I'm using hard drives, I'd probably use ZFS with flash-based read and write cache or Storage Spaces with a flash tier and its integrated write cache on flash.
- wtallis 5y ago
  > Both Intel and AMD support limited hardware RAID for NVMe natively on some of their CPUs. "Limited" here means the motherboard firmware understands the same array format as their proprietary software RAID drivers, so you can boot an OS from the RAID volume.
p_l 5y ago
Not HBA adapters, in fact the design is pretty much directly for use with some very pricy HBAs - NVMe-OF seems obvious target, as I don't see much benefit from "just" allowing HDDs in U.2 bay and thus possibly standardising on single fabric bay in server chassis. NVMe-OF has options in form of InfiniBand and ROCE HBA chips that can be configured to act as PCI-E root complexes bifurcated over many many links (in place of default x16 endpoint) with minimal management logic or built-in SoC features. So a network JBOD with NVMe-OF can be essentially one chip + PCI-E links to disks + support/glue logic.

toast0 5y ago

Seems like it. The article mentions a PCIe switch, but PCIe bifurcation may also be an option. (That's splitting a multiple lane slot into multiple slots, requires system firmware suppport though)

formerly_proven 5y ago
Bifurcation has never been a thing on desktop platforms and even most entry-level (single socket, desktop-equivalent) servers don't support it. It seems to be reserved for HEDT and real server platforms. (This is of course purely a market segmentation decision by Intel/AMD).
- toast0 5y ago
  My A520 mini-itx board supports it; can't get anymore desktop than that. Although, that has limited options, I think I can do either 2 x8 or one x8 and two x4. For this, it looks like each drive is expected to be x1, so you'd want one x16 to 16 x1s. It's doable, but not without mucking about in the firmware (either by the OEM, or dedicated enthusiasts), so a PCIe switch is probably advisable.
  - formerly_proven 5y ago
    Ah I see. It seems like previously bifurcation was not qualified for anything but the X-series chipset, but in the 500 series it's qualified for all. On top of that, it seems like some boards just allowed it regardless in prior generations. Another complication is of course that the non-PEG slots on the (non-HEDT) platforms are usually electrically only x4 or x1, so bifurcation really only makes sense in the PEG.
    - wtallis 5y ago
      PCIe root ports in CPUs are generally designed to provide an x16 with bifurcation down to x4x4x4x4 (or merely x8x4x4 for Intel consumer CPUs). Large-scale PCIe switches also commonly support bifurcation only down to x4 or sometimes x2, though x1 may start catching on with PCIe gen5. Smaller PCIe switches and motherboard chipsets usually support link widths from x4 down to x1. Treating each lane individually goes hand in hand with the fact that many of the lanes provided by a motherboard chipset can be reconfigured between some combination of PCIe, SATA and USB: they design a multi-purpose PHY, put down one or two dozen copies of it at the perimeter of the die, and connect them to an appropriate variety of MACs.
      - formerly_proven 5y ago
        Yeah, but what I meant above was that if you only have an x4 slot electrically, sticking in an x16 -> 4x M.2 riser isn't going to do a whole lot, because the 12 lanes of 3 out of 4 slots aren't hooked up to anything. So in this scenario you'd really want a riser with a switch in it instead (which are more expensive than almost all motherboards). So on the consumer platforms that give you two PEGs best you could do while still having a GPU is stick that riser in the second PEG and use the x8/x8 split. Now the question becomes whether the UEFI allows you to use the x8/x8 bifurcation meant for dual GPU or similar use in an x8/(x4+x4) triple bifurcation kind of setup. Realistically this entire thing just doesn't make a lot of sense on the consumer platforms because they just don't have enough PCIe lanes out of the CPU. Intel used to be slightly worse here with 20 (of which 4 are reserved for the PCH but you know that), while AM4 has 28 (4 for the I/O hub again). On an HEDT platform with 40+ lanes though... (When I say bifurcation I meant bifurcation of a slot on the mainboard, not the various ways the slots and ports on the board itself can be configured, though that's technically bifurcation as well (or even switching protocols)
        wtallis 5y ago
        > Yeah, but what I meant above was that if you only have an x4 slot electrically, sticking in an x16 -> 4x M.2 riser isn't going to do a whole lot, because the 12 lanes of 3 out of 4 slots aren't hooked up to anything. So in this scenario you'd really want a riser with a switch in it instead (which are more expensive than almost all motherboards). True; but given how PCIe speeds are no longer stalled, we may soon see motherboards offering an x4 slot that can be operated as x1x1x1x1. Currently, the only risers you're likely to find that split a slot into independent x1 ports are intended for crypto mining, and they require switches. A passive (or retimer-only) quad-M.2 riser that only provides one PCIe lane per drive currently sounds a bit bandwidth-starved and wouldn't work with current motherboards. But given PCIe gen5 SSDs or widespread availability of PCIe-native hard drives, those uses for an x4 slot will start to make sense.
- wtallis 5y ago
  PCIe bifurcation works fine on AMD consumer platforms. I've used a passive quad-M.2 riser card on an AMD B550 motherboard with no trouble other than changing a firmware setting to turn on bifurcation. It's only Intel that is strict about this aspect of product segmentation.
- rasz 5y ago
  Intel supported PCIE bifurcation it on desktop chipsets up to Sandy Bridge. Just like Intel once supported memory parity on desktop chipsets until deciding to use it to upsell server products. Recent Intel chipsets were locked to Intel SSDs and only some magic hardware DRM handshake unlocked bifurcation https://www.asus.com/us/support/FAQ/1037507/ https://www.asus.com/us/support/FAQ/1037507/ But apparently AMD competition forced their hand and at lest some got unlocked https://forum.level1techs.com/t/new-bios-update-unlocks-4x-pcie-x4-bifurcation-and-raid-on-existing-x299-motherboards/1510701 https://forum.level1techs.com/t/new-bios-update-unlocks-4x-p... On AMD front there doesnt seem to be any problems, plenty of desktop boards with unlocked bifurcation BIOS menus.

wtallis 5y ago

PCIe switches are a lot simpler and more standardized than RAID and HBA controllers, but I'm not sure they're any cheaper for similar bandwidth and port counts. Broadcom/Avago/PLX and Microchip/Microsemi are the only two vendors for large-scale current generation PCIe switches, and starting with PCIe gen3 they decided to price them way out of the consumer market, contributing to the disappearance of multi-GPU from gaming PCs.

semi-extrinsic 5y ago
The main reason for multi-GPUs decline is that single GPU performance has grown much faster than both CPUs and RAM speeds. Going multi-GPU on a high-end GPU today lets you run a AAA game at 4K with all the eye candy at 150 fps rather than 110 fps, i.e. visually undistinguishable. And at the low end, it's always better performance per dollar to buy a single higher-end card. Of course this is all orthogonal to your point of PCIe switches. Anyone thinking about multi-GPU will buy a motherboard with at least 2 PCIe 4.0 x16 slots anyway, no switches required. You can even get motherboards with 2 PCIe 5.0 x16 slots without breaking the bank.
- wtallis 5y ago
  > Anyone thinking about multi-GPU will buy a motherboard with at least 2 PCIe 4.0 x16 slots anyway, no switches required. You can even get motherboards with 2 PCIe 5.0 x16 slots without breaking the bank. I'm pretty sure you're referring to motherboards that have two slots which are mechanically x16, but actually using the second slot borrows half of the lanes from the first slot, leaving you operating at x8+x8. This is not the same as having a full 48+ lane PCIe packet switch between the CPU and two slots each wired for x16, which was very common for high-end consumer motherboards in the PCIe gen2 era (eg. with NVIDIA's NF200 switch). These days, you can get 32+ lanes straight from the CPU to enable two x16 slots, but that requires stepping up to a bigger CPU socket than the mainstream consumer CPU+motherboard platforms use. And I don't think you can yet buy a motherboard with two PCIe gen5 x16 slots at any price: as far as I can tell, none of the Intel Alder Lake motherboards announced so far include PCIe switches, because such switches have been cost-prohibitive for several generations now.

mgerdts 5y ago

I don't think so. The reason that I would use a RAID card with HDDs is to get the benefit of a battery backed write cache, optimally performing RAID operations. I've only seen one RAID card that supports connecting to NVMe drives. It is terribly expensive and aimed supporting the speed of flash-based NVMe drives. If you want a battery backed write cache with flash-based NVMe, get an enterprise drive with power loss protection, and you are golden. You don't generally need the write cache to get reasonable throughput, but you can save a few microseconds in write latency. I suspect that NVMe HDDs will have the same amount of write cache that existing SATA/SAS HDDs have. If you feel you need a RAID card for RAID and/or cache with SAS or SATA HDDs, you will probably have the same need for a RAID card with NVMe HDDs. Both Intel and AMD support limited hardware RAID for NVMe natively on some of their CPUs. I don't think this has any of the cache benefits you normally have with a RAID card. That is, it may work well with flash-based NVMe drives, particularly those with power loss protection. But really, if I'm using hard drives, I'd probably use ZFS with flash-based read and write cache or Storage Spaces with a flash tier and its integrated write cache on flash.

wtallis 5y ago
> Both Intel and AMD support limited hardware RAID for NVMe natively on some of their CPUs. "Limited" here means the motherboard firmware understands the same array format as their proprietary software RAID drivers, so you can boot an OS from the RAID volume.

p_l 5y ago

Not HBA adapters, in fact the design is pretty much directly for use with some very pricy HBAs - NVMe-OF seems obvious target, as I don't see much benefit from "just" allowing HDDs in U.2 bay and thus possibly standardising on single fabric bay in server chassis. NVMe-OF has options in form of InfiniBand and ROCE HBA chips that can be configured to act as PCI-E root complexes bifurcated over many many links (in place of default x16 endpoint) with minimal management logic or built-in SoC features. So a network JBOD with NVMe-OF can be essentially one chip + PCI-E links to disks + support/glue logic.