3 ms·

This is a crucial consideration when choosing hardware (especially Intel vs AMD) [0]. Of note is the AMD Threadripper family supporting up to 56 PCIe lanes dire

This is a crucial consideration when choosing hardware (especially Intel vs AMD) [0]. Of note is the AMD Threadripper family supporting up to 56 PCIe lanes directly to the CPU. On the datacenter/server side AMD Epyc supports 128 CPU PCIe lanes [1]. Note this is per socket so with multi-socket systems these numbers increase accordingly. Massive amounts of I/O such as this are instrumental for high and extremely dense I/O applications whether it be for storage, network, GPUs, etc (or combinations of all, of course). I know it's been a consideration for companies such as Netflix, Cloudflare, and Nvidia. [0] https://www.cgdirector.com/guide-to-pcie-lanes/ https://www.cgdirector.com/guide-to-pcie-lanes/ [1] https://www.amd.com/en/processors/epyc-7002-series https://www.amd.com/en/processors/epyc-7002-series

cesarb 5y ago
> On the datacenter/server side AMD Epyc supports 128 CPU PCIe lanes [1]. Note this is per socket so with multi-socket systems these numbers increase accordingly. However, AMD Epyc uses half of the 128 lanes on each socket to talk to the other socket, so on two-socket systems each socket has only 64 PCIe lanes available, for a total of 128 lanes, the same as with a single socket. On the other hand, newer AMD Epyc processors can (depending on the motherboard) use only 48 lanes to talk to the other socket, freeing another 16 lanes per socket (for a total of 160 PCIe lanes) at the cost of slower inter-socket communication; and they also have an extra single PCIe lane per socket to be used to connect to a BMC. The article at https://www.servethehome.com/why-amd-epyc-rome-2p-will-have-128-160-pcie-gen4-lanes-and-a-bonus/ https://www.servethehome.com/why-amd-epyc-rome-2p-will-have-... has a great explanation of all that.
- kkielhofner 5y ago
  Oh wow! I didn't know that. I'm assuming this is for DMA between the two sockets (and potentially) cross-socket PCIe device routing? I still remember building a two socket Xeon workstation some years ago and being puzzled that video wasn't initializing. Turns out one CPU wasn't quite seated correctly and the GPU was in a slot wired to it. I wonder if this architecture avoids that?
  - loeg 5y ago
    It’s for cache coherency between the two sockets, in addition to remote device (eg, PCIe card local to the remote socket) and memory access.
  - cesarb 5y ago
    > I'm assuming this is for DMA between the two sockets (and potentially) cross-socket PCIe device routing? It's more than just that. The memory is attached directly to each socket, so for instance you could have 64GB of memory on each socket for a total of 128GB; for a core in one socket to access memory which happens to be attached to the other socket, it has to go through these inter-socket links. More than that, for a core in one socket to access memory in the same socket but which has been cached somewhere in the other socket, the cache coherence traffic has to go through these inter-socket links. > Turns out one CPU wasn't quite seated correctly and the GPU was in a slot wired to it. I wonder if this architecture avoids that? No, each PCIe link is wired to only one of the CPU sockets (PCIe is point-to-point, not a bus like classic PCI), or to an auxiliary chip which is wired to only one of the CPU sockets; if that CPU is not seated correctly, what you saw could happen. The architecture which avoids that is the older one in which all CPUs were wired together in a bus, with PCI and memory attached to an auxiliary chip also on that bus.
- Dylan16807 5y ago
  > On the other hand, newer AMD Epyc processors can (depending on the motherboard) use only 48 lanes to talk to the other socket, freeing another 16 lanes per socket (for a total of 160 PCIe lanes) at the cost of slower inter-socket communication; and they also have an extra single PCIe lane per socket to be used to connect to a BMC. For people that aren't reading the whole article, I want to make it clear that "slower" is only relative to other newer chips. The lanes on the old chips were half as fast, so even 48 lanes beats them by a factor of 1.5x.

cesarb 5y ago

> On the datacenter/server side AMD Epyc supports 128 CPU PCIe lanes [1]. Note this is per socket so with multi-socket systems these numbers increase accordingly. However, AMD Epyc uses half of the 128 lanes on each socket to talk to the other socket, so on two-socket systems each socket has only 64 PCIe lanes available, for a total of 128 lanes, the same as with a single socket. On the other hand, newer AMD Epyc processors can (depending on the motherboard) use only 48 lanes to talk to the other socket, freeing another 16 lanes per socket (for a total of 160 PCIe lanes) at the cost of slower inter-socket communication; and they also have an extra single PCIe lane per socket to be used to connect to a BMC. The article at https://www.servethehome.com/why-amd-epyc-rome-2p-will-have-128-160-pcie-gen4-lanes-and-a-bonus/ https://www.servethehome.com/why-amd-epyc-rome-2p-will-have-... has a great explanation of all that.

kkielhofner 5y ago
Oh wow! I didn't know that. I'm assuming this is for DMA between the two sockets (and potentially) cross-socket PCIe device routing? I still remember building a two socket Xeon workstation some years ago and being puzzled that video wasn't initializing. Turns out one CPU wasn't quite seated correctly and the GPU was in a slot wired to it. I wonder if this architecture avoids that?
- loeg 5y ago
  It’s for cache coherency between the two sockets, in addition to remote device (eg, PCIe card local to the remote socket) and memory access.
- cesarb 5y ago
  > I'm assuming this is for DMA between the two sockets (and potentially) cross-socket PCIe device routing? It's more than just that. The memory is attached directly to each socket, so for instance you could have 64GB of memory on each socket for a total of 128GB; for a core in one socket to access memory which happens to be attached to the other socket, it has to go through these inter-socket links. More than that, for a core in one socket to access memory in the same socket but which has been cached somewhere in the other socket, the cache coherence traffic has to go through these inter-socket links. > Turns out one CPU wasn't quite seated correctly and the GPU was in a slot wired to it. I wonder if this architecture avoids that? No, each PCIe link is wired to only one of the CPU sockets (PCIe is point-to-point, not a bus like classic PCI), or to an auxiliary chip which is wired to only one of the CPU sockets; if that CPU is not seated correctly, what you saw could happen. The architecture which avoids that is the older one in which all CPUs were wired together in a bus, with PCI and memory attached to an auxiliary chip also on that bus.
Dylan16807 5y ago
> On the other hand, newer AMD Epyc processors can (depending on the motherboard) use only 48 lanes to talk to the other socket, freeing another 16 lanes per socket (for a total of 160 PCIe lanes) at the cost of slower inter-socket communication; and they also have an extra single PCIe lane per socket to be used to connect to a BMC. For people that aren't reading the whole article, I want to make it clear that "slower" is only relative to other newer chips. The lanes on the old chips were half as fast, so even 48 lanes beats them by a factor of 1.5x.