3 ms·

Pity I can't downvote you. You deserve it. You didn't read the whole post, especially the most critical part. Lack of PCI Express LANES is the problem, not sl

Pity I can't downvote you. You deserve it. You didn't read the whole post, especially the most critical part. Lack of PCI Express LANES is the problem, not slots. All this great shit you're talking about is wonderful... as long as you have the PCI Express lanes to support all those add-in cards.

justinlloyd 4y ago
I did read your entire post. And the OP post. I have Intel Xeon Skylake processors, 48 lanes of PCIe 3.0, all of equal priority so they aren't tiered like in some of the earlier blue team processors or the red team processors. Across two CPUs that gives me 96 lanes. The slots are switched so either of the CPUs can use an individual PCIe card, or I can assign a PCIe card to a specific CPU. Alternatively if there are not enough PCIe slots I can run the PCIe cards through a separate switch. There are a lot of non-consumer grade motherboards out there that offer these features. It is rare that any deployment will require all 48 lanes on a CPU to be maxed out simultaneously, though I've been involved with use cases where that has taken place (8x 4K non-compressed video stream captures directly to storage), though what happens then is you run in to DRAM bandwidth issues and other problems. In my workstation I have dual RTX A5000 GPUs, dual Blackmagic quad 4K capture cards, dual Highpoint USB 3.0 expanders with four separate USB controllers per board, a Highpoint M.2 RAID controller with on-board PLX, along with the onboard M.2, six channel U.2 through a switch, onboard USB & SATA. What we should be asking for is better switching, not more lanes. Again, it is a rare use case where we can max out the bandwidth of all PCIe lanes of a CPU. We should also be asking for better switching for the direct DMA between cards which is a sorely neglected area across all architectures/motherboards. P.S. Forgot to mention the NIC PCIe card.

justinlloyd 4y ago

I did read your entire post. And the OP post. I have Intel Xeon Skylake processors, 48 lanes of PCIe 3.0, all of equal priority so they aren't tiered like in some of the earlier blue team processors or the red team processors. Across two CPUs that gives me 96 lanes. The slots are switched so either of the CPUs can use an individual PCIe card, or I can assign a PCIe card to a specific CPU. Alternatively if there are not enough PCIe slots I can run the PCIe cards through a separate switch. There are a lot of non-consumer grade motherboards out there that offer these features. It is rare that any deployment will require all 48 lanes on a CPU to be maxed out simultaneously, though I've been involved with use cases where that has taken place (8x 4K non-compressed video stream captures directly to storage), though what happens then is you run in to DRAM bandwidth issues and other problems. In my workstation I have dual RTX A5000 GPUs, dual Blackmagic quad 4K capture cards, dual Highpoint USB 3.0 expanders with four separate USB controllers per board, a Highpoint M.2 RAID controller with on-board PLX, along with the onboard M.2, six channel U.2 through a switch, onboard USB & SATA. What we should be asking for is better switching, not more lanes. Again, it is a rare use case where we can max out the bandwidth of all PCIe lanes of a CPU. We should also be asking for better switching for the direct DMA between cards which is a sorely neglected area across all architectures/motherboards. P.S. Forgot to mention the NIC PCIe card.