4 ms·
A lot of the reasons they've had to build most of this stuff themselfs is because they decided for some reason to use freeBSD. The NUMA work they did, I rememb
by Moral_ 4y ago
A lot of the reasons they've had to build most of this stuff themselfs is because they decided for some reason to use freeBSD.
The NUMA work they did, I remember being in a meeting with them as a Linux Developer at Intel at the time. They bought NVMe drives or were saying they were going to buy NVMe drives from Intel which got them access to "on the ground" kernel developers and CPU people from Intel. Instead of talking about NVMe they spent the entire meeting asking us about howt the Linux kernel handles NUMA and corner cases around memory and scheudling. If I recall correctly I think they asked if we could help them upstream BSD code for NVMe and NUMA. I think in that meeting there was even some L9 or super high up NUMA CPU guy from Hillsborough they some how convinced to join.
The conversation and technical discussion was quite fun, but it was sort of funny to us at the time they were having to do all this work on the BSD kernel that was solved years ago for linux.
Technical debt I guess.
- cperciva 4y agoNetflix tried Linux. FreeBSD worked better.
- dboreham 4y agoBy some definition of better.
- trasz 4y agoIt worked faster. It's a common misconception among newbies that "Linux has NUMA" automatically means it will use NUMA properly in a given workload. What it actually means is you _should_ be able to use existing functionality. Sometimes you'll only need to configure it, sometimes you'll need to reimplement it from the scratch, and doing that in FreeBSD is easier because there's less bloat.
- throw0101c 4y ago*At the time when they created the OCA project. If someone was going to do a similar comparison now the results could be different.
- Thaxll 4y agoIt's hard to believe in 2022, Google, Amazon, FB etc .. all use Linux, all CDN use Linux as well, and some services serve even more traffic than Netflix ( Youtube ). BSD faster than Linux is a myth, the fact that 99% of those run on Linux means more people worked on those problems means it's most likely always faster. The funny thing is the rest of Netflix runs on Ubuntu, only those edge CDN runs on BSD.
- ratorx 4y agoDisclaimer: SRE at Google on a team vaguely related to video CDN stuff, but have no inside knowledge I don’t think you can dismiss BSD faster than Linux (or make any claims about the relative speed of different OSes) just because big companies run Linux. There are other costs involved and optimisations that can be shared if your edge serving stack is as similar as possible to the non-edge serving stack (that you have many more engineers developing for). All you can conclude is that with enough optimisation, Linux can be made to perform well enough for it to not be worth replacing (yet). Because replacing Linux would require replicating all the custom software and optimisations made to it for whatever other platform you pick.
- rwaksmunski 4y agoYou'd be surprised how many businesses run FreeBSD and keep it a secret as a competitive advantage.
- jeffbee 4y agoI still don't get the NUMA obsession here. It seems like they could have saved a lot of effort and a huge number of powerpoint slides by building a box with half of these resources and no NUMA: one CPU socket with all the memory and one PCIe root complex and all the disks and NICs attached thereto. It would be half the size, draw half the power, and be way easier to program.
- muststopmyths 4y agoCan you buy non NUMA mainstream CPUs though? Honest question because I’d love to be rid of that BS too
- jeffbee 4y agoNUMA is an outcome of system configuration. You can make a non-NUMA platform using any CPU. You just limit yourself to 1 CPU socket. Here's a Facebook engineering blog post about how they left NUMA behind. https://engineering.fb.com/2016/03/09/data-center-engineering/facebook-s-new-front-end-server-design-delivers-on-performance-without-sucking-up-power/ https://engineering.fb.com/2016/03/09/data-center-engineerin...
- Dylan16807 4y ago> You can make a non-NUMA platform using any CPU. You just limit yourself to 1 CPU socket. Well, not on Epyc generation 1. Those have four NUMA segments in each socket. Also those Xeon Platinum 9200 processors Intel made as an attention grab.
- jeffbee 4y agoEPYC Naples wasn't good for much of anything though, so I am trying to forget it.
- drewg123 4y agoThis is a testbed to see what breaks at higher speed. Our normal production platforms are indeed single socket and run at 1/2 this speed. I've identified all kinds of unexpected bottlenecks on this testbed, so it has been worth it. We invested in NUMA back when Intel was the only game in town, and they refused to give enough IO and memory bandwidth per-socket to scale to 200Gb/s. Then AMD EPYC came along. And even though Naples was single-socket, you had to treat it as NUMA to get performance out of it. With Rome and Milan, you can run them in 1NPS mode and still get good performance, so NUMA is used mainly for forward looking performance testbeds.
- ksec 4y agoIs NUMA a solved issue on Linux? Correct me if I am wrong but I was under the impression it may be better handled under certain conditions, but NUMA, the problem in itself is hardly solved.
- alberth 4y agoMaybe Brendan Gregg can further enlighten his new coworkers at Intel why Netflix chose both AMD & FreeBSD.