4 ms·

Surely these processors introduce another new level of NUMA dynamics? Each group of 8 cores has its own memory controller, own PCIe root complex, and then there

Surely these processors introduce another new level of NUMA dynamics? Each group of 8 cores has its own memory controller, own PCIe root complex, and then there is a crosslink between each group of 8 cores. Up until now you would (potentially) have to consider which socket you are on, and where your memory or IO devices (PCIe) are. Now you have the same considerations within a socket, as well as between sockets?

wmf 9y ago
Previous Opterons were also MCMs with NUMA within the socket, although their performance was poor enough that many people probably never noticed they existed. If Intel makes cluster-on-die mode mandatory then they'll also have NUMA within the socket.
- sliken 9y ago
  Actually the opterons were pretty competitive for many years, but of course stalled with the opteron 63xx series. They still scaled better than intel, but often lost on single thread performance. But if you were looking at performance per node on parallel tasks they often won by a substantial amount on price/performance. However since 2012 (the opteron 63xx announce) AMD dropped the ball. First they were non-competitive on the dual sockets, and then a year or so later they were non-competitive on quad sockets. Not sure why it's taken them 5 years to bring out something newer, but surprisingly the ryzen/naples/epyc looks surprisingly competitive.
- geezerjay 9y ago
  > although their performance was poor enough that many people probably never noticed they existed. I've read HPC papers demonstrating Opteron processors outperformed Xeons in BLAS applications by a wide margin due to their greater throughput. I don't really know if that assertion ever had any basis on reality.
dom0 9y ago
For most server workloads this plays a relatively minor role.
loeg 9y ago
Yep.
sliken 9y ago
Not really. So the Epyc is basically 4 ryzens. So you get 4x the cores, 4x the memory channels, and 4x the pieces of silicon. So think of a single socket ryzen as a quad socket motherboard. In either case you have clusters of cores/cache connected to memory controllers and hypertransport. For most workloads a NUMA aware kernel does a pretty good job of minimizing hitting pages on other controllers. But it's not a particularly big deal when you miss, typically about 10% (latency and bandwidth). AMD makes all the I/O pins capable of hypertransport (or whatever they call it now) and PCI-e. This isn't particularly new btw. MCM (multiple chips per package) go way back to the pentium pro if not before. Intel Xeons are all single chip, but have similar on chip architectures. The 4,6,8 core chips are pretty simple, but the larger core chips have a ring bus for one set of cores, and another ring bus for the second. But generally for most workloads the NUMA issues related to the newer chips isn't a particularly large hurdle from getting good performance. What I am concerned about though is how good the Epyc floating point is, I fear they are bragging about integer performance and not FP because they are behind on FP.
Dylan16807 9y ago
It's more than that, I think the cores are still in clusters of 4 that talk over infinity fabric. But there's no avoiding some kind of complex inter-core dynamics at this level. The Intel alternative is a bunch of ring busses that have different speeds to each core from any point. And this design makes every memory access go over the infinity fabric, so latency might be surprisingly even.

wmf 9y ago

Previous Opterons were also MCMs with NUMA within the socket, although their performance was poor enough that many people probably never noticed they existed. If Intel makes cluster-on-die mode mandatory then they'll also have NUMA within the socket.

sliken 9y ago
Actually the opterons were pretty competitive for many years, but of course stalled with the opteron 63xx series. They still scaled better than intel, but often lost on single thread performance. But if you were looking at performance per node on parallel tasks they often won by a substantial amount on price/performance. However since 2012 (the opteron 63xx announce) AMD dropped the ball. First they were non-competitive on the dual sockets, and then a year or so later they were non-competitive on quad sockets. Not sure why it's taken them 5 years to bring out something newer, but surprisingly the ryzen/naples/epyc looks surprisingly competitive.
geezerjay 9y ago
> although their performance was poor enough that many people probably never noticed they existed. I've read HPC papers demonstrating Opteron processors outperformed Xeons in BLAS applications by a wide margin due to their greater throughput. I don't really know if that assertion ever had any basis on reality.

dom0 9y ago

For most server workloads this plays a relatively minor role.

loeg 9y ago

Yep.

sliken 9y ago

Not really. So the Epyc is basically 4 ryzens. So you get 4x the cores, 4x the memory channels, and 4x the pieces of silicon. So think of a single socket ryzen as a quad socket motherboard. In either case you have clusters of cores/cache connected to memory controllers and hypertransport. For most workloads a NUMA aware kernel does a pretty good job of minimizing hitting pages on other controllers. But it's not a particularly big deal when you miss, typically about 10% (latency and bandwidth). AMD makes all the I/O pins capable of hypertransport (or whatever they call it now) and PCI-e. This isn't particularly new btw. MCM (multiple chips per package) go way back to the pentium pro if not before. Intel Xeons are all single chip, but have similar on chip architectures. The 4,6,8 core chips are pretty simple, but the larger core chips have a ring bus for one set of cores, and another ring bus for the second. But generally for most workloads the NUMA issues related to the newer chips isn't a particularly large hurdle from getting good performance. What I am concerned about though is how good the Epyc floating point is, I fear they are bragging about integer performance and not FP because they are behind on FP.

Dylan16807 9y ago

It's more than that, I think the cores are still in clusters of 4 that talk over infinity fabric. But there's no avoiding some kind of complex inter-core dynamics at this level. The Intel alternative is a bunch of ring busses that have different speeds to each core from any point. And this design makes every memory access go over the infinity fabric, so latency might be surprisingly even.