5 ms·

It seems to me you might have a hard time adding 12 NVMe drives to a single CPU then? SATA might be slower, but I much prefer a slower interface that can break

by bullen 5y ago

It seems to me you might have a hard time adding 12 NVMe drives to a single CPU then? SATA might be slower, but I much prefer a slower interface that can break but supports 12x drives than a single NVMe drive!!! Also I need to see some CPU overhead graphs on those 12x SATA drives vs. a single NVMe?! My suspicion is that the overhead will be lower if the SATA control chip is buffering data to avoid the CPU waiting.

deleted 5y ago
[deleted]
fh973 5y ago
You can connect many NVMEs to a single CPU, but the PCI bandwidth becomes a bottleneck much earlier of course. Just as with SATA. Think of the SATA HBA as an adapter that bridges between PCI and SATA. For NVMe the counterpart is a PCI switch, with PCI on both sides.
- puzzlingcaptcha 5y ago
  Supermicro sells single-CPU servers with 24 NVMe bays. Not a big problem if your CPU has 128 PCIe lanes. You could get 12 quite easily even on consumer hardware. AMD Threadripper gives you 64 PCIe 4.0 lanes (so up to 72 with the TRX40 chipset), so you just use three PCIe x16 expansion cards each providing four NVMe 4x slots.
deleted 5y ago
[deleted]
MisterTea 5y ago
> It seems to me you might have a hard time adding 12 NVMe drives to a single CPU then? Attachment is a solved issue as all you do is buy a PCIe switch and hang more nvme off your cpu. The issue I see is no clear standard for nvme form factors or connectors. m.2 is a consumer standard that is more mobile friendly than desktop or server yet it appears to be the more common standard in all three. U.2 nvme in the classic 2.5" spinning rust form factor but I don't see them used in desktops. Then there are numerous "ruler" form factors designed to plug into 1U servers. Connectors are also a mixed bag with m.2, U.2, a 36pin quad channel sff motherboard connector, sata express connector, and OCulink. And further this whole article is more old-is-new again blogspam as all it talks about yet another SAN technology.
- wyager 5y ago
  I had a very hard time finding an efficient way to add more NVMe slots to my personal storage server. If you want more than 1 NVMe drive per PCI slot (even if the slot is x16), it seems you need an expensive card that only works with certain motherboards. What are these PCI switches you mention?
  - voxadam 5y ago
    They're talking about a PCIe card with a PCIe-to-PCIe switch chip from the likes of the ones available from PLX Technology (now Broadcom). In the absence of chipset and firmware level PCIe bifurcation support a PCIe switch is your only option. https://en.wikipedia.org/wiki/PCI_Express#SWITCH https://en.wikipedia.org/wiki/PCI_Express#SWITCH https://en.wikipedia.org/wiki/PLX_Technology https://en.wikipedia.org/wiki/PLX_Technology https://www.anandtech.com/show/13511/highpoint-releases-the-ssd7102-a-bootable-quad-m2-pcie-x16-nvme-ssd-raid-card https://www.anandtech.com/show/13511/highpoint-releases-the-... https://www.ebay.com/p/18021887628 https://www.ebay.com/p/18021887628
  - MisterTea 5y ago
    > I had a very hard time finding an efficient way to add more NVMe slots to my personal storage server. I think this has more to do with the form factor/connector issue than the interface. m.2 is an awkward form factor even though it is the most popular. An m.2 device is a low cost bare pcb which relies on a cheap board-to-board connector and secured via a screw remotely located from the connector. You cant locate these off board without a carrier and cable and you cant connect them vertically because of the way they are secured. Great for laptops though... Another thing to consider is NVMe uses a PCI express x4 link meaning you need eight high speed differential pairs in a single cable along with clocking signals and so on. Cables aren't as simple nor cheap to make as sata which only had two differential pairs. Hopefully OCulink will become the standard. And don't forget that disk form factors and interfaces are tied to desktop standards like ATX which are not the driving force behind computer design anymore. So does it make sense to continue to promulgate these form factors, e.g. U.2? I personally say keep what works but some will argue that. Until we get this whole physical part of nvme sorted, we're going to be stuck in this awkward transition phase for a bit.

by bullen 5y ago

deleted 5y ago

[deleted]

fh973 5y ago

You can connect many NVMEs to a single CPU, but the PCI bandwidth becomes a bottleneck much earlier of course. Just as with SATA. Think of the SATA HBA as an adapter that bridges between PCI and SATA. For NVMe the counterpart is a PCI switch, with PCI on both sides.

puzzlingcaptcha 5y ago
Supermicro sells single-CPU servers with 24 NVMe bays. Not a big problem if your CPU has 128 PCIe lanes. You could get 12 quite easily even on consumer hardware. AMD Threadripper gives you 64 PCIe 4.0 lanes (so up to 72 with the TRX40 chipset), so you just use three PCIe x16 expansion cards each providing four NVMe 4x slots.

deleted 5y ago

[deleted]

MisterTea 5y ago

> It seems to me you might have a hard time adding 12 NVMe drives to a single CPU then? Attachment is a solved issue as all you do is buy a PCIe switch and hang more nvme off your cpu. The issue I see is no clear standard for nvme form factors or connectors. m.2 is a consumer standard that is more mobile friendly than desktop or server yet it appears to be the more common standard in all three. U.2 nvme in the classic 2.5" spinning rust form factor but I don't see them used in desktops. Then there are numerous "ruler" form factors designed to plug into 1U servers. Connectors are also a mixed bag with m.2, U.2, a 36pin quad channel sff motherboard connector, sata express connector, and OCulink. And further this whole article is more old-is-new again blogspam as all it talks about yet another SAN technology.

wyager 5y ago
I had a very hard time finding an efficient way to add more NVMe slots to my personal storage server. If you want more than 1 NVMe drive per PCI slot (even if the slot is x16), it seems you need an expensive card that only works with certain motherboards. What are these PCI switches you mention?
- voxadam 5y ago
  They're talking about a PCIe card with a PCIe-to-PCIe switch chip from the likes of the ones available from PLX Technology (now Broadcom). In the absence of chipset and firmware level PCIe bifurcation support a PCIe switch is your only option. https://en.wikipedia.org/wiki/PCI_Express#SWITCH https://en.wikipedia.org/wiki/PCI_Express#SWITCH https://en.wikipedia.org/wiki/PLX_Technology https://en.wikipedia.org/wiki/PLX_Technology https://www.anandtech.com/show/13511/highpoint-releases-the-ssd7102-a-bootable-quad-m2-pcie-x16-nvme-ssd-raid-card https://www.anandtech.com/show/13511/highpoint-releases-the-... https://www.ebay.com/p/18021887628 https://www.ebay.com/p/18021887628
- MisterTea 5y ago
  > I had a very hard time finding an efficient way to add more NVMe slots to my personal storage server. I think this has more to do with the form factor/connector issue than the interface. m.2 is an awkward form factor even though it is the most popular. An m.2 device is a low cost bare pcb which relies on a cheap board-to-board connector and secured via a screw remotely located from the connector. You cant locate these off board without a carrier and cable and you cant connect them vertically because of the way they are secured. Great for laptops though... Another thing to consider is NVMe uses a PCI express x4 link meaning you need eight high speed differential pairs in a single cable along with clocking signals and so on. Cables aren't as simple nor cheap to make as sata which only had two differential pairs. Hopefully OCulink will become the standard. And don't forget that disk form factors and interfaces are tied to desktop standards like ATX which are not the driving force behind computer design anymore. So does it make sense to continue to promulgate these form factors, e.g. U.2? I personally say keep what works but some will argue that. Until we get this whole physical part of nvme sorted, we're going to be stuck in this awkward transition phase for a bit.