3 ms·

They are direct to 1/2 of the CPUs, actually. This is part of what the GP was complaining about. The 2 dies which do not have memory also do not have PCIe wir

They are direct to 1/2 of the CPUs, actually. This is part of what the GP was complaining about. The 2 dies which do not have memory also do not have PCIe wired out of the socket, so they are one hop away from I/O as well. If you are trying to max out I/O, you'd probably be better off with a low-end Epyc that had fewer cores enabled per die. It is interesting to note that the memoryless dies do have PCIe root ports, for built-in things like the PSP crypto engines. That surprised me somewhat when I first noticed it, but it makes sense in retrospect, since the issue is that the TR socket was designed for 2 dies.

Dylan16807 8y ago
The interconnect latencies give you something like an extra 50 nanoseconds[1]. That matters when accessing memory. Not so much for PCIe, where your base latency is most of a microsecond[2][3]. There seems to be plenty of bandwidth to handle it, too. [1] https://www.servethehome.com/amd-epyc-infinity-fabric-latency-ddr4-2400-v-2666-a-snapshot/ https://www.servethehome.com/amd-epyc-infinity-fabric-latenc... [2] https://forum.stanford.edu/events/posterslides/LowLatencyNetworkInterfaces.pdf https://forum.stanford.edu/events/posterslides/LowLatencyNet... [3] https://gianniantichi.github.io/files/papers/pciebench.pdf https://gianniantichi.github.io/files/papers/pciebench.pdf
- drewg123 8y ago
  My comment was addressing the parent comment about bandwidth, not latency. Because you only have 1/2 the memory and I/O bandwidth on threadripper, if maxing out I/O is your concern, you would likely be better off with a low-end epyc that still had all the memory controllers and pcie lanes wired up.
  - Dylan16807 8y ago
    > My comment was addressing the parent comment about bandwidth, not latency. Oh, it is? It really seemed to be about direct vs. non-direct, and nothing else. And that pretty much only affects latency, because it only loads the infinity fabric by about 20-25% to route 16 PCIe lanes to each die.

Dylan16807 8y ago

The interconnect latencies give you something like an extra 50 nanoseconds[1]. That matters when accessing memory. Not so much for PCIe, where your base latency is most of a microsecond[2][3]. There seems to be plenty of bandwidth to handle it, too. [1] https://www.servethehome.com/amd-epyc-infinity-fabric-latency-ddr4-2400-v-2666-a-snapshot/ https://www.servethehome.com/amd-epyc-infinity-fabric-latenc... [2] https://forum.stanford.edu/events/posterslides/LowLatencyNetworkInterfaces.pdf https://forum.stanford.edu/events/posterslides/LowLatencyNet... [3] https://gianniantichi.github.io/files/papers/pciebench.pdf https://gianniantichi.github.io/files/papers/pciebench.pdf

drewg123 8y ago
My comment was addressing the parent comment about bandwidth, not latency. Because you only have 1/2 the memory and I/O bandwidth on threadripper, if maxing out I/O is your concern, you would likely be better off with a low-end epyc that still had all the memory controllers and pcie lanes wired up.
- Dylan16807 8y ago
  > My comment was addressing the parent comment about bandwidth, not latency. Oh, it is? It really seemed to be about direct vs. non-direct, and nothing else. And that pretty much only affects latency, because it only loads the infinity fabric by about 20-25% to route 16 PCIe lanes to each die.