5 ms·

I had a very hard time finding an efficient way to add more NVMe slots to my personal storage server. If you want more than 1 NVMe drive per PCI slot (even if t

by wyager 5y ago

I had a very hard time finding an efficient way to add more NVMe slots to my personal storage server. If you want more than 1 NVMe drive per PCI slot (even if the slot is x16), it seems you need an expensive card that only works with certain motherboards. What are these PCI switches you mention?

voxadam 5y ago
They're talking about a PCIe card with a PCIe-to-PCIe switch chip from the likes of the ones available from PLX Technology (now Broadcom). In the absence of chipset and firmware level PCIe bifurcation support a PCIe switch is your only option. https://en.wikipedia.org/wiki/PCI_Express#SWITCH https://en.wikipedia.org/wiki/PCI_Express#SWITCH https://en.wikipedia.org/wiki/PLX_Technology https://en.wikipedia.org/wiki/PLX_Technology https://www.anandtech.com/show/13511/highpoint-releases-the-ssd7102-a-bootable-quad-m2-pcie-x16-nvme-ssd-raid-card https://www.anandtech.com/show/13511/highpoint-releases-the-... https://www.ebay.com/p/18021887628 https://www.ebay.com/p/18021887628
MisterTea 5y ago
> I had a very hard time finding an efficient way to add more NVMe slots to my personal storage server. I think this has more to do with the form factor/connector issue than the interface. m.2 is an awkward form factor even though it is the most popular. An m.2 device is a low cost bare pcb which relies on a cheap board-to-board connector and secured via a screw remotely located from the connector. You cant locate these off board without a carrier and cable and you cant connect them vertically because of the way they are secured. Great for laptops though... Another thing to consider is NVMe uses a PCI express x4 link meaning you need eight high speed differential pairs in a single cable along with clocking signals and so on. Cables aren't as simple nor cheap to make as sata which only had two differential pairs. Hopefully OCulink will become the standard. And don't forget that disk form factors and interfaces are tied to desktop standards like ATX which are not the driving force behind computer design anymore. So does it make sense to continue to promulgate these form factors, e.g. U.2? I personally say keep what works but some will argue that. Until we get this whole physical part of nvme sorted, we're going to be stuck in this awkward transition phase for a bit.
puzzlingcaptcha 5y ago
I don't know why you were downvoted because this is a valid question. Most NVMe expansion cards ('carrier cards') do not integrate any extra IC (or just a retimer) and instead rely on motherboard feature called PCIe bifurcation which essentially turns a x16 slot into four 'virtual' x4 slots. These can be fairly inexpensive ($50-$80), e.g. Supermicro AOC-SLG3-2M2 which provides two NVMe x4 connectors or Linkreal LRNV95NF which provides four. If your motherboard does not support bifurcation then you need a more expensive ($150-$200) card with a PCI switch (e.g. SuperMicro AOC-SHG3-4M2P or Linkreal LRNV9524 which use PLX PEX switches).

by wyager 5y ago

voxadam 5y ago

They're talking about a PCIe card with a PCIe-to-PCIe switch chip from the likes of the ones available from PLX Technology (now Broadcom). In the absence of chipset and firmware level PCIe bifurcation support a PCIe switch is your only option. https://en.wikipedia.org/wiki/PCI_Express#SWITCH https://en.wikipedia.org/wiki/PCI_Express#SWITCH https://en.wikipedia.org/wiki/PLX_Technology https://en.wikipedia.org/wiki/PLX_Technology https://www.anandtech.com/show/13511/highpoint-releases-the-ssd7102-a-bootable-quad-m2-pcie-x16-nvme-ssd-raid-card https://www.anandtech.com/show/13511/highpoint-releases-the-... https://www.ebay.com/p/18021887628 https://www.ebay.com/p/18021887628

MisterTea 5y ago

> I had a very hard time finding an efficient way to add more NVMe slots to my personal storage server. I think this has more to do with the form factor/connector issue than the interface. m.2 is an awkward form factor even though it is the most popular. An m.2 device is a low cost bare pcb which relies on a cheap board-to-board connector and secured via a screw remotely located from the connector. You cant locate these off board without a carrier and cable and you cant connect them vertically because of the way they are secured. Great for laptops though... Another thing to consider is NVMe uses a PCI express x4 link meaning you need eight high speed differential pairs in a single cable along with clocking signals and so on. Cables aren't as simple nor cheap to make as sata which only had two differential pairs. Hopefully OCulink will become the standard. And don't forget that disk form factors and interfaces are tied to desktop standards like ATX which are not the driving force behind computer design anymore. So does it make sense to continue to promulgate these form factors, e.g. U.2? I personally say keep what works but some will argue that. Until we get this whole physical part of nvme sorted, we're going to be stuck in this awkward transition phase for a bit.

puzzlingcaptcha 5y ago

I don't know why you were downvoted because this is a valid question. Most NVMe expansion cards ('carrier cards') do not integrate any extra IC (or just a retimer) and instead rely on motherboard feature called PCIe bifurcation which essentially turns a x16 slot into four 'virtual' x4 slots. These can be fairly inexpensive ($50-$80), e.g. Supermicro AOC-SLG3-2M2 which provides two NVMe x4 connectors or Linkreal LRNV95NF which provides four. If your motherboard does not support bifurcation then you need a more expensive ($150-$200) card with a PCI switch (e.g. SuperMicro AOC-SHG3-4M2P or Linkreal LRNV9524 which use PLX PEX switches).