4 ms·

> It seems to me you might have a hard time adding 12 NVMe drives to a single CPU then? Attachment is a solved issue as all you do is buy a PCIe switch and ha

> It seems to me you might have a hard time adding 12 NVMe drives to a single CPU then? Attachment is a solved issue as all you do is buy a PCIe switch and hang more nvme off your cpu. The issue I see is no clear standard for nvme form factors or connectors. m.2 is a consumer standard that is more mobile friendly than desktop or server yet it appears to be the more common standard in all three. U.2 nvme in the classic 2.5" spinning rust form factor but I don't see them used in desktops. Then there are numerous "ruler" form factors designed to plug into 1U servers. Connectors are also a mixed bag with m.2, U.2, a 36pin quad channel sff motherboard connector, sata express connector, and OCulink. And further this whole article is more old-is-new again blogspam as all it talks about yet another SAN technology.

wyager 5y ago
I had a very hard time finding an efficient way to add more NVMe slots to my personal storage server. If you want more than 1 NVMe drive per PCI slot (even if the slot is x16), it seems you need an expensive card that only works with certain motherboards. What are these PCI switches you mention?
- voxadam 5y ago
  They're talking about a PCIe card with a PCIe-to-PCIe switch chip from the likes of the ones available from PLX Technology (now Broadcom). In the absence of chipset and firmware level PCIe bifurcation support a PCIe switch is your only option. https://en.wikipedia.org/wiki/PCI_Express#SWITCH https://en.wikipedia.org/wiki/PCI_Express#SWITCH https://en.wikipedia.org/wiki/PLX_Technology https://en.wikipedia.org/wiki/PLX_Technology https://www.anandtech.com/show/13511/highpoint-releases-the-ssd7102-a-bootable-quad-m2-pcie-x16-nvme-ssd-raid-card https://www.anandtech.com/show/13511/highpoint-releases-the-... https://www.ebay.com/p/18021887628 https://www.ebay.com/p/18021887628
- MisterTea 5y ago
  > I had a very hard time finding an efficient way to add more NVMe slots to my personal storage server. I think this has more to do with the form factor/connector issue than the interface. m.2 is an awkward form factor even though it is the most popular. An m.2 device is a low cost bare pcb which relies on a cheap board-to-board connector and secured via a screw remotely located from the connector. You cant locate these off board without a carrier and cable and you cant connect them vertically because of the way they are secured. Great for laptops though... Another thing to consider is NVMe uses a PCI express x4 link meaning you need eight high speed differential pairs in a single cable along with clocking signals and so on. Cables aren't as simple nor cheap to make as sata which only had two differential pairs. Hopefully OCulink will become the standard. And don't forget that disk form factors and interfaces are tied to desktop standards like ATX which are not the driving force behind computer design anymore. So does it make sense to continue to promulgate these form factors, e.g. U.2? I personally say keep what works but some will argue that. Until we get this whole physical part of nvme sorted, we're going to be stuck in this awkward transition phase for a bit.
- puzzlingcaptcha 5y ago
  I don't know why you were downvoted because this is a valid question. Most NVMe expansion cards ('carrier cards') do not integrate any extra IC (or just a retimer) and instead rely on motherboard feature called PCIe bifurcation which essentially turns a x16 slot into four 'virtual' x4 slots. These can be fairly inexpensive ($50-$80), e.g. Supermicro AOC-SLG3-2M2 which provides two NVMe x4 connectors or Linkreal LRNV95NF which provides four. If your motherboard does not support bifurcation then you need a more expensive ($150-$200) card with a PCI switch (e.g. SuperMicro AOC-SHG3-4M2P or Linkreal LRNV9524 which use PLX PEX switches).

wyager 5y ago

I had a very hard time finding an efficient way to add more NVMe slots to my personal storage server. If you want more than 1 NVMe drive per PCI slot (even if the slot is x16), it seems you need an expensive card that only works with certain motherboards. What are these PCI switches you mention?

voxadam 5y ago
They're talking about a PCIe card with a PCIe-to-PCIe switch chip from the likes of the ones available from PLX Technology (now Broadcom). In the absence of chipset and firmware level PCIe bifurcation support a PCIe switch is your only option. https://en.wikipedia.org/wiki/PCI_Express#SWITCH https://en.wikipedia.org/wiki/PCI_Express#SWITCH https://en.wikipedia.org/wiki/PLX_Technology https://en.wikipedia.org/wiki/PLX_Technology https://www.anandtech.com/show/13511/highpoint-releases-the-ssd7102-a-bootable-quad-m2-pcie-x16-nvme-ssd-raid-card https://www.anandtech.com/show/13511/highpoint-releases-the-... https://www.ebay.com/p/18021887628 https://www.ebay.com/p/18021887628
MisterTea 5y ago
> I had a very hard time finding an efficient way to add more NVMe slots to my personal storage server. I think this has more to do with the form factor/connector issue than the interface. m.2 is an awkward form factor even though it is the most popular. An m.2 device is a low cost bare pcb which relies on a cheap board-to-board connector and secured via a screw remotely located from the connector. You cant locate these off board without a carrier and cable and you cant connect them vertically because of the way they are secured. Great for laptops though... Another thing to consider is NVMe uses a PCI express x4 link meaning you need eight high speed differential pairs in a single cable along with clocking signals and so on. Cables aren't as simple nor cheap to make as sata which only had two differential pairs. Hopefully OCulink will become the standard. And don't forget that disk form factors and interfaces are tied to desktop standards like ATX which are not the driving force behind computer design anymore. So does it make sense to continue to promulgate these form factors, e.g. U.2? I personally say keep what works but some will argue that. Until we get this whole physical part of nvme sorted, we're going to be stuck in this awkward transition phase for a bit.
puzzlingcaptcha 5y ago
I don't know why you were downvoted because this is a valid question. Most NVMe expansion cards ('carrier cards') do not integrate any extra IC (or just a retimer) and instead rely on motherboard feature called PCIe bifurcation which essentially turns a x16 slot into four 'virtual' x4 slots. These can be fairly inexpensive ($50-$80), e.g. Supermicro AOC-SLG3-2M2 which provides two NVMe x4 connectors or Linkreal LRNV95NF which provides four. If your motherboard does not support bifurcation then you need a more expensive ($150-$200) card with a PCI switch (e.g. SuperMicro AOC-SHG3-4M2P or Linkreal LRNV9524 which use PLX PEX switches).