3 ms·

Here is the easy example: A dual 100GbE NIC requires a PCIe Gen4 x16 slot to run both ports at full speed. Mellanox ConnectX-6 200GbE cards require either 2x G

Here is the easy example: A dual 100GbE NIC requires a PCIe Gen4 x16 slot to run both ports at full speed. Mellanox ConnectX-6 200GbE cards require either 2x Gen4 or 1x Gen5 slot for full dual port bandwidth. Most of the larger NVMEoF nodes we are seeing are using more than one 100GbE NIC. Enterprise SSDs will rapidly fill Gen4 x4 lanes later this year as we start to see them launch alongside more mainstream platforms. Four of those will fill a x16 slot. GPUs and PCIe switches are also great examples of uses. You are right that for the high-volume server segment, 160 PCIe lanes are overkill, as is even 128 today frankly. For some context, there is a huge portion of servers sold with less than 1 DIMM per RAM channel installed.

velox_io 8y ago
Hi Patrick, Before we had M2/U2 NVMe drives, there were quite a few manufacturers making drives that plug straight into the PCIe bus where they could easily access the full 16 lanes, and they still can... Intel seems to be the only ones making PCIe drives (Octane's are funny; they don't look fast only 2000 MB/s, although they do win on latency and random access). Anyway, there doesn't seem to be much demand for that speed, I guess it's a case that x4 U2/M2 are good enough. Speaking of which.. I'm am surprised there isn't a similar network interface utilising PCIe with a low overhead protocol the same as NVMe does. Getting towards the point where a small number of machines could share memory and CPU's between machines. I know we have Infiniband but that has always come at quite a cost, mainly in supercomputers.
- wmf 8y ago
  There are still a decent number of x8 AIC SSDs on the market but enterprises don't want them because they're not hot swap. Networking over PCIe exists but no one seems to care about it. https://semiaccurate.com/2014/09/23/look-avagos-expressfabric/ https://semiaccurate.com/2014/09/23/look-avagos-expressfabri... Part of the problem may be that 96x8G PCIe switches look pretty small compared to 256x50G Ethernet switches and thus you'd need a lot more of them.
- justinclift 8y ago
  > I know we have Infiniband but that has always come at quite a cost, mainly in supercomputers. Lower end Infiniband (eg 40/56Gb) isn't too expensive these days. Well, that's assuming people aren't paying list price. :)

velox_io 8y ago

Hi Patrick, Before we had M2/U2 NVMe drives, there were quite a few manufacturers making drives that plug straight into the PCIe bus where they could easily access the full 16 lanes, and they still can... Intel seems to be the only ones making PCIe drives (Octane's are funny; they don't look fast only 2000 MB/s, although they do win on latency and random access). Anyway, there doesn't seem to be much demand for that speed, I guess it's a case that x4 U2/M2 are good enough. Speaking of which.. I'm am surprised there isn't a similar network interface utilising PCIe with a low overhead protocol the same as NVMe does. Getting towards the point where a small number of machines could share memory and CPU's between machines. I know we have Infiniband but that has always come at quite a cost, mainly in supercomputers.

wmf 8y ago
There are still a decent number of x8 AIC SSDs on the market but enterprises don't want them because they're not hot swap. Networking over PCIe exists but no one seems to care about it. https://semiaccurate.com/2014/09/23/look-avagos-expressfabric/ https://semiaccurate.com/2014/09/23/look-avagos-expressfabri... Part of the problem may be that 96x8G PCIe switches look pretty small compared to 256x50G Ethernet switches and thus you'd need a lot more of them.
justinclift 8y ago
> I know we have Infiniband but that has always come at quite a cost, mainly in supercomputers. Lower end Infiniband (eg 40/56Gb) isn't too expensive these days. Well, that's assuming people aren't paying list price. :)