3 ms·
https://people.freebsd.org/~gallatin/talks/euro2019.pdf https://people.freebsd.org/~gallatin/talks/euro2019.pdf Here's a talk about a lot of the NUMA-issues bro
by dragontamer 5y ago
https://people.freebsd.org/~gallatin/talks/euro2019.pdf https://people.freebsd.org/~gallatin/talks/euro2019.pdf
Here's a talk about a lot of the NUMA-issues brought up by Netflix + FreeBSD. Its 200 Gb/s (bits), not bytes, but I'd expect you to be running into a lot of the similar problems.
Its 200 Gb/s because they're reading from SSDs. But still, careful attention to NUMA was the only way they could reach full PCIe-bandwidths, which is a similar issue to yours (where NUMA is needed to reach full Memory/DRAM bandwidth).
--------
Did you set your BIOS into NUMA mode? If you're in default BIOS-settings, your RAM is being split up between all 4 nodes (Uniform-memory-access mode). At least, that's how it was for my Threadripper 1950x (I know you've got a totally different model, but who knows?)
I forgot exactly the name of the BIOS setting. But you gotta boot your computer up with 4-nodes-per-socket (or was it 8-nodes per socket for your computer?) or some similar setting. There were some programs on Linux (and Windows) that help you see what the OS is seeing.
- kristjansson 5y agoSee also the a more recent talk[0] on progress made on the same topic. On EPYC too, so perhaps more applicable to OP? [0]: https://people.freebsd.org/~gallatin/talks/euro2021.pdf https://people.freebsd.org/~gallatin/talks/euro2021.pdf
- dragontamer 5y agoThe original talk was on 2019 era hardware, for a 2019-era talk. Xeon and EPYC are both discussed, albeit somewhat dated now but still relevant IMO. The newer talk is 2021 hardware. IMO, the newer talk seems to focus more on the specific details of kTLS and PCIe TLS... but there's still stuff in there about NUMA for sure. But yeah, the 2021 talk definitely is on more recent hardware: 2021 Xeons and EPYCs.
- kristjansson 5y agoAh, dang, that's on me, I forgot the earlier talk discussed EPYC chips too.
- bradfa 5y agoAre you thinking of "NUMA per socket" setting in the BIOS? Dell has a short overview of this focused on Epyc 7000 series: https://downloads.dell.com/manuals/common/dell-emc-dfd-numa-amd-epyc-2ndgen.pdf https://downloads.dell.com/manuals/common/dell-emc-dfd-numa-...
- dragontamer 5y agoThat sounds like the one. Different motherboards call these settings different names in their BIOS programs / setup however. So just keep an eye out for it.