3 ms·
I still don't get the NUMA obsession here. It seems like they could have saved a lot of effort and a huge number of powerpoint slides by building a box with hal
by jeffbee 4y ago
I still don't get the NUMA obsession here. It seems like they could have saved a lot of effort and a huge number of powerpoint slides by building a box with half of these resources and no NUMA: one CPU socket with all the memory and one PCIe root complex and all the disks and NICs attached thereto. It would be half the size, draw half the power, and be way easier to program.
- muststopmyths 4y agoCan you buy non NUMA mainstream CPUs though? Honest question because I’d love to be rid of that BS too
- jeffbee 4y agoNUMA is an outcome of system configuration. You can make a non-NUMA platform using any CPU. You just limit yourself to 1 CPU socket. Here's a Facebook engineering blog post about how they left NUMA behind. https://engineering.fb.com/2016/03/09/data-center-engineering/facebook-s-new-front-end-server-design-delivers-on-performance-without-sucking-up-power/ https://engineering.fb.com/2016/03/09/data-center-engineerin...
- Dylan16807 4y ago> You can make a non-NUMA platform using any CPU. You just limit yourself to 1 CPU socket. Well, not on Epyc generation 1. Those have four NUMA segments in each socket. Also those Xeon Platinum 9200 processors Intel made as an attention grab.
- jeffbee 4y agoEPYC Naples wasn't good for much of anything though, so I am trying to forget it.
- drewg123 4y agoThis is a testbed to see what breaks at higher speed. Our normal production platforms are indeed single socket and run at 1/2 this speed. I've identified all kinds of unexpected bottlenecks on this testbed, so it has been worth it. We invested in NUMA back when Intel was the only game in town, and they refused to give enough IO and memory bandwidth per-socket to scale to 200Gb/s. Then AMD EPYC came along. And even though Naples was single-socket, you had to treat it as NUMA to get performance out of it. With Rome and Milan, you can run them in 1NPS mode and still get good performance, so NUMA is used mainly for forward looking performance testbeds.
- Bluecobra 4y agoIf you are buying servers at scale the costs will certainly add up vs. buying two processors. If you buy single proc servers, that is double the amount of chassis, rail kits, power supplies, power cables, drives, iLO/iDRAC licenses, etc.
- dboreham 4y agoYou can build motherboards with two or more completely isolated sets of CPU and memory, that are physically compatible with standard racks etc.
- Bluecobra 4y agoGood point, I forgot about those. It would be interesting to see if 1x PowerEdge C6525 with four single processor nodes is cheaper than 2x Dell R7525 servers. The C6525 does support dual processor, so it does seem a bit wasteful to me.
- jiggawatts 4y agoModern CPUs like the AMD EPYC server processor are "always NUMA", even in single-socket configurations! They have 9 chips on what is essentially a tiny, high-density motherboard. Effectively they are 8-socket server boards that fit in the palm of your hand. The dual-socket version is effectively a 16-socket motherboard with a complex topology configured in a hierarchy. Take a look at some "core-to-core" latency diagrams. They're quite complex because of the various paths possible: https://www.anandtech.com/show/16214/amd-zen-3-ryzen-deep-dive-review-5950x-5900x-5800x-and-5700x-tested/5 https://www.anandtech.com/show/16214/amd-zen-3-ryzen-deep-di... Intel is not immune from this either.Their higher core-count server processors have two internal ring-bus networks, with some cores "closer" to PCIe devices or certain memory buses: https://semiaccurate.com/2017/06/15/intel-talks-skylakes-mesh-interconnect/ https://semiaccurate.com/2017/06/15/intel-talks-skylakes-mes...