5 ms·
A delightful set of slides and references that tickles many of my pet topics. In particular, one I’d love to hear more about is why so many deployments are stil
by jeffbee 5y ago
A delightful set of slides and references that tickles many of my pet topics. In particular, one I’d love to hear more about is why so many deployments are still choosing 2-socket servers by default when managing them is such a pain in the neck and the performance when you do it badly is so poor. Live the life of the future, today: choose single sockets!
- soulbadguy 5y agoIn world where most work loads are containerized, and where each container can be pinned to numa region doesn't it really matter?
- wmf 5y agoDoes any container runtime/orchestrator perform this optimization yet? Why wait?
- syoc 5y agoKernel scheduling is NUMA aware and will localize workloads. Threads will mostly have their RAM on the sticks local to their node. The core the thread is delegated to is also more likely to be the core local to the disk or NIC being used for IO. This is at least my experience, though I am no expert.
- solarkennedy 5y agoTitus (Netflix's container orchestrator that I work on) does this via: https://github.com/Netflix-Skunkworks/titus-isolate https://github.com/Netflix-Skunkworks/titus-isolate
- jeffbee 5y agok8s, by default, is oblivious to NUMA topology. You have to enable unreleased features and configure them correctly, which is the unwanted complexity to which I referred earlier. Simply aligning your containers to NUMA domains does not solve the problem that your arriving network frames or your NVMe completion queues can still be on the wrong domain. Isn't it simpler to just have 1 socket and not need to care? The number of cores available on a single socket system is pretty high these days, and in general the 1S parts are cheaper and faster.
- eloff 5y agoYeah, it makes a lot of sense to go with single socket servers unless you can't scale horizontally (e.g. database server). Why deal with the complexity when you can just side step it.
- dragontamer 5y agoWhy would you switch from a 100GBps NUMA connection (800 gigabits per second) over NUMA fabric into a 10 Gbps Ethernet fabric? If you are scaling horizontally, NUMA is the superior fabric than Ethernet or Infiniband (100Gbps) Horizontal scaling seems to favor NUMA. 1000 chips over Ethernet is less efficient than 500 dual socket nodes over Ethernet. Anything you can do over Ethernet seems easier and cheaper over NUMA instead.
- wmf 5y agoThis is correct if your software is NUMA-optimized (or if auto-NUMA works well for you) but if it isn't you can end up with slowdowns.
- dragontamer 5y agoSurely that can be fixed with just a well placed numactl command to set node affinity and CPU affinity. The root article is discussing rewriting code to fit on FPGAs. If NUMA is too complex then... I dunno. The FPGA argument seems dead on arrival.
- kortilla 5y agoYour scaling architecture sucks if it depends on that kind of throughput. If you need that you’ve only can kicked your way to more capacity without a real scaling fix.
- dragontamer 5y agoDepend on? Heavens no. Dual socket has numerous advantages in density and rack space. The fact that performance is better is pretty much icing on the cake. It's easier to manage 500 dual socket servers than 1000 single socket servers. Less equipment, higher utilization of parts, etc. Etc. To suggest dual socket NUMA is going away is... just very unlikely to me. I don't see what the benefits would be at all. Not just performance, but also routine maintenance issues (power, Ethernet, local storage, etc etc)
- syoc 5y agoRack space can be quite expensive. Sometimes you need a lot of computing power in one or two rack units. Would be interested in what the management pains are. I agree that 2 socket machines require more thought in a lot of scenarios, especially IO heavy workloads.
- zaroth 5y agoIn my very limited experience it seems like space is much less an issue than power density. You can fit far more kW/U than the datacenter can possibly cool. In the commodity space that I rent, I ran out of power before filling even half the rack. I’m sure higher power/cooling density is possible to obtain, but I would think you’re primarily paying for that versus square footage?
- zozbot234 5y agoWhat do you need that power density for? It's a rack, not a supercomputer. (I sure hope it's not "mining coins" or anything like that.)
- wmf 5y agoIt's not really about needing power density; high density can easily happen accidentally. 40 1S servers in a rack could be 20 kW and 40 2S servers could be 30+ kW.
- deleted 5y ago[deleted]
- jeffbee 5y agoThe OpenCompute "Delta Lake" machine mentioned in the article occupies only one third of 1RU and peaks at 400W. You will certainly be power/cooling limited, rather than volume limited, with that kind of density.
- toast0 5y agoAll things considered, managing fewer hosts is nicer than more hosts. In some hardware generations, dual socket has pretty good cost and complexity tradeoffs. And if you benefit from having a large dataset in memory on a single machine, dual socket often gets you twice the DIMM sockets and therefore twice the ram. Quad socket has been very expensive (and not great performance) for quite some time, so that's usually out. Single socket Epyc looks pretty impressive though; although I'm retired and probably won't get to work with those anytime soon.
- rektide 5y ago> In particular, one I’d love to hear more about is why so many deployments are still choosing 2-socket servers by default bit of a guess but connectivity has been coming at a stupidly high premium for too long. there are sweet sweet blade chassis with price optimized less than full power 1p designs but 10g is still kind of novel there. bigger form factors are starting to see 25gbit at not-astronomical prices & switches in some rare cases are reasonable too. power supplies, storage, networking... a computer has a lot of not-entirely-ancillary needs. having multiple chips sharing the perhipetals should make sense, should be cheap. it's not though. the SMP tax is huge huge huge. thing is we don't need smp. we just need multihost peripherals. we needs nic's that like the grouphug ocp board can support 4 separate modes via pcie srv-io, a nice that can present multiple different virtual functions that different hosts can use. NVME similarly could be multiport- was was. power supplies are shared in ocp designs, with big bus rails, some 48v. I'm notad yet but sure seems dead obvious to me the future of multi-socket is non-coherency. build a big board with a couple different isolated computers on it, but connected via shared nic or nics to the top-of-rack. we get close with the 3 per width ope compute systems but those each need to be self contained, and there's an obvious leap in efficiency to be had by merging those three separate computers onto a single motherboard, while sharing some network, maybe storage devices. also like throw in some gratis pcie ntb maybe for a medium speed (32gbps on pcie 4.0 x16) direct server to server interconnect. ideally add another ntb unit on most chips so we can make a little mediums speed nearly free ring, or other topology. choose single sockets but choose many of them, each sharing some common peripherals.