4 ms·

Huge improvement for AMD EPYC, basically nothing for Intel. Anyone got an idea why?

jeffbee 3y ago
Probably the "machine of the day" for google websearch is an EPYC so they are focused on that. Possibly the issue didn't affect Xeon in the first place, leaving nothing to optimize. <strike>You can see in the absolute values it was already faster on Intel than it is on EPYC after the improvement.</strike> actually I am cross-eyed, I thought there was another zero there. I wonder if this set of patches was the result of visibility gained from recent `perf mem` features that show loads and stores for each struct member.
- TacticalCoder 3y ago
  > You can see in the absolute values it was already faster on Intel than it is on EPYC after the improvement I don't know because the table shows a 100 Gbit/s NIC on AMD and a 200 Gbit/s (!) NIC on Intel. Which for some benchmarks would now make AMD twice as fast as Intel. Note sure if the 100 Gbit vs 200 Gbit labels are correct though.
wmf 3y ago
Epyc has multiple L3 caches while Xeon has a unified L3. If the same TCP socket is accessed on multiple CCXes it would cause cache lines to bounce on Epyc which is pretty costly.
- briffle 3y ago
  I believe the new Xeon SP processors coming out are also based on the 'chiplet' design like the Epyc ones.
  - wmf 3y ago
    They are but Intel implements chiplets differently so they still have a single unified L3 cache.
- charleshn 3y ago
  On that subject, it's worth knowing that with the default BIOS settings the whole socket shows up as a single NUMA node, which means that most workloads don't benefit from automatic NUMA balancing or manual NUMA-aware placement within a socket. One needs to set NPS "Nodes per socket" to e.g. 4 to have one NUMA node per quadrant, and potentially also "L3 cache as NUMA" to have one per CCX [0]. [0] https://www.amd.com/content/dam/amd/en/documents/epyc-technical-docs/tuning-guides/58002_amd-epyc-9004-tg-hpc.pdf https://www.amd.com/content/dam/amd/en/documents/epyc-techni...

Probably the "machine of the day" for google websearch is an EPYC so they are focused on that. Possibly the issue didn't affect Xeon in the first place, leaving nothing to optimize. <strike>You can see in the absolute values it was already faster on Intel than it is on EPYC after the improvement.</strike> actually I am cross-eyed, I thought there was another zero there. I wonder if this set of patches was the result of visibility gained from recent `perf mem` features that show loads and stores for each struct member.

TacticalCoder 3y ago
> You can see in the absolute values it was already faster on Intel than it is on EPYC after the improvement I don't know because the table shows a 100 Gbit/s NIC on AMD and a 200 Gbit/s (!) NIC on Intel. Which for some benchmarks would now make AMD twice as fast as Intel. Note sure if the 100 Gbit vs 200 Gbit labels are correct though.

wmf 3y ago

Epyc has multiple L3 caches while Xeon has a unified L3. If the same TCP socket is accessed on multiple CCXes it would cause cache lines to bounce on Epyc which is pretty costly.

briffle 3y ago
I believe the new Xeon SP processors coming out are also based on the 'chiplet' design like the Epyc ones.
- wmf 3y ago
  They are but Intel implements chiplets differently so they still have a single unified L3 cache.
charleshn 3y ago
On that subject, it's worth knowing that with the default BIOS settings the whole socket shows up as a single NUMA node, which means that most workloads don't benefit from automatic NUMA balancing or manual NUMA-aware placement within a socket. One needs to set NPS "Nodes per socket" to e.g. 4 to have one NUMA node per quadrant, and potentially also "L3 cache as NUMA" to have one per CCX [0]. [0] https://www.amd.com/content/dam/amd/en/documents/epyc-technical-docs/tuning-guides/58002_amd-epyc-9004-tg-hpc.pdf https://www.amd.com/content/dam/amd/en/documents/epyc-techni...