10 ms·
NUMA Siloing in the FreeBSD Network Stack [pdf]
- mikece 7y agoIs it still accurate that for network bound servers that FreeBSD still outperforms Linux?
- monocasa 7y agoStock, probably not. FreeBSD is a nicer platform to modify for your use case though.
- tieze 7y agoAs I understood Linux made a lot of progress in the last couple of years with say zero-copy patches, etc. I thought they reached some kind of parity.
- whalesalad 7y agoAFAIK Linux is actually more performant these days, network-wise.
- GhettoMaestro 7y agoI think once you get to a certain traffic level you are forced to do kernel-bypass stuff like DPDK. Regardless of Linux or FreeBSD being the kernel.
- the8472 7y agoIf it's just serving files you don't necessarily need DPDK/XDP. For server-grade hardware there now is P2P-DMA and TLS accelerators which can offload everything to peripherials while still using normal socket APIs. You get NVMe -(PCIe)-> crypto accelerator -(PCIe)-> ethernet for the bulk of the data. Neither CPU nor main memory see any of the network packets as long as they stay on the happy path. Only connection setup, DMA orchestration and occasional TLS renegotiation have to be handled.
- drewg123 7y agoIn the work we've done, the TLS crypto of bulk data is handled entirely in the CPU by the kernel via ktls. Please see the slides, specifically the data flow diagrams.
- cthalupa 7y agoI think he's talking about the general case (and since XDP is in the discussion, probably the general case for Linux) and not trying to speak specifically to what Netflix is doing for their CDN.
- floatboth 7y agoI know Chelsio has crypto directly on the NIC, but are dedicated crypto accelerator cards a thing and are they ever worth it? Why leave the CPU idle when the CPU itself is a good crypto accelerator (AES-NI, ARMv8 crypto)? AMD Ryzen has a built-in crypto "decelerator" — a FreeBSD driver was written for the crypto engine, but it's disabled by default because it made everything slower than AES-NI. (Though I guess it would be funny to use it to mine bitcoin, since it supports SHA256. AMD — Advanced Mining Devices!)
- loeg 7y agoIntel has a product line called QAT ("QuickAssisT"?) that does crypto acceleration, as well as compression. I don't know how performant it is. There are definitely several older crypto accelerators that were faster than CPUs of the time; I don't know if any of them (outside of QAT) is still relevant. The AMD Zen1 Crypto Co-Processor is indeed slower than AESNI; I think it's mostly used by stuff like SecureBoot, TPM, etc, and also used internally by the CPU to generate RDRAND/RDSEED data. It was probably never intended to be used by OS drivers and certainly not intended to be any kind of accelerator.
- shaklee3 7y agoSupposedly QAT is built into the chipsets of Skylake and above now. I've never seen anyone try it, though.
- toast0 7y agoNetflix is not bypassing the FreeBSD kernel. They bypass userland for bulk data though --- with sendfile, and kTLS, their webserver (nginx) has to negotiate TLS and parse requests and open files, but then it's pretty simple.
- drewg123 7y agoAuthor here.. We're doing ~200Gb/s through the kernel.
- GhettoMaestro 7y agoThat's why I carefully inserted my weasel words of "I think" ;) Thank you for your contributions. Great information.
- drewg123 7y agoThanks! I often do the same :)
- big_chungus 7y agoIt depends on how exactly each is tuned. Cache and buffer sizes, drivers, TCP congestion control algorithms, and more can affect this hugely, and are different. The defaults on linux (and probably on freebsd) also change reasonably often, so you can expect linux 4.x to have singificant differences from 5.x.
- drewg123 7y agoAuthor here.. It depends heavily on the workload. We have man years of optimizations for our CDN workload that we have been upstreaming to FreeBSD. See, for example, the other talk I co-presented regarding Kernel TLS on FreeBSD. https://people.freebsd.org/~gallatin/talks/euro2019-ktls.pdf https://people.freebsd.org/~gallatin/talks/euro2019-ktls.pdf
- continuations 7y agoWhat made Netflix pick FreeBSD over Linux in the first place?
- o-__-o 7y agoBSDs unwavering commitment to reliability. A Linux kernel crashes. A lot. FreeBSD ran cdrom.com for 365+ days without rebooting. Just saying
- yjftsjthsd-h 7y agoWhat's "a lot" here, and are you sure that's still true? Because my suspicion is that it might have been true before FAANG sunk billons of R&D into Linux, but I would be shocked if Linux kernels cash at any rare worth noticing today. Although it's splitting hairs, really; when is the last time you saw any major kernel crash?
- drewg123 7y agoI crash FreeBSD every day :)
- o-__-o 7y agoyou're making me want to start building some BSD boxes
- throw0101a 7y agoEvery year I still see a Linux system or two live-lock. When I was still doing Solaris and FreeBSD admining that never happened. However most systems aren't pushed to extremes, so these corner cases aren't seen very often, regardless of OS.
- loeg 7y agoI think it's safe to say any kind of sweeping generalization like this is unlikely to be broadly true (in either direction). Both work quite well.
- toast0 7y agoI wonder how this would compare with Receive Side Scaling (RSS), which you can use to pin NIC queues to cpus, and then pin the rest of the handling to the same cpu, avoiding a significant amount of interprocess communication. NUMA concerns may be more important than CPU pining within a domain though.
- cthalupa 7y agoRSS is more about spreading a workload over multiple cores to not overwhelm any single core, particularly in high PPS situations. This is about PCI-E devices being hooked up to a NUMA node to avoid saturating the link between nodes. There's a limited amount of bandwidth, and crossing nodes saturates this and increases latency, both of which will have limiting effects on your total possible throughput. RSS configuration doesn't need you to set up your hardware in any specific way, with this you need to ensure that the set of disks and nic are hooked up to coupled domains - e.g. if you place the two NICs on the same NUMA node, then no software configuration is going to fix that, and you'd have to go and physically rearrange things to fix it. You might still use RSS to distributed the workload across multiple cores within that NUMA domain when using this setup.
- mosselman 7y agoIsn't it great that we work in an industry where a company like Netflix, which has increasing competition lately, just shares potential competitive advantages like this? I get that, in part, this is a way to get people excited to work for Netflix, however, the people working on this are, probably, pretty proud of what they achieved and thus like to share it with us, their colleagues at other places of work and they have the freedom to do so because of our industry's great track record of knowledge sharing.
- i_cant_speel 7y agoAgreed. I believe technology as a whole is light-years ahead of where it would be if companies treated all of their findings and system designs as closely-guarded secrets. I am relatively early in my career and I get to open-source a project that I have been working on for the first time and it feels great.
- um_ya 7y agoJust imagine how far ahead the other industries would be if we handled patent and trademark law like we do in the software industry.
- bscphil 7y ago> I get that, in part, this is a way to get people excited to work for Netflix It's even more amazing to me that this is a successful tactic in light of the fact that there's no shortage of people willing to work for ad-tech firms. If working for a company with good values was such a draw, you'd think it would give them more of a competitive advantage. Maybe the real reason is that Netflix's "good values" are genuine, not just a draw for employees. I'm really not sure.
- gigatexal 7y agoThe whole ad tech space as a data nerd is super enticing. I could work on some really cool data warehouse / database issues but at the end of the day I’d be helping an ad tech company... Worse yet is I’ve met some really cool people in the space but it is still ad tech
- byefruit 7y agoI assumed these were cache servers that went into ISPs, does anyone know what they mean by "Increases AWS cloud management overhead" on a couple of the slides?
- tiernano 7y agoGuessing that they have managed services in aws for maintaining all the nodes and where they are. If each box has 2x the ips, they would need more mgmt resources to keep track of what is on that box, if it's online, stats, etc.
- drewg123 7y agoYes, exactly. When we first started the project, I was told that multiple IPs were not an option for that reason.
- bluedino 7y agoWas that a technical or administrative restriction? I'm interested in how an organization makes a decision like that, and then what channels are used to reverse it.
- drewg123 7y agoAuthor here: The talk will be on Youtube eventually, and a lot of points are explained in more detail in the actual talk. I was just going to bed in advance of traveling back to the states tomorrow, but I'll try to answer any question in the morning.
- DSingularity 7y agoDo you think these kinds of results can be achieved through improved schedulers and allocators?
- floatboth 7y agoWhy is the hw.model on EPYC redacted, when the chips are listed in the slides? Secret engineering sample? :)
- cperciva 7y agoHe said in the talk that yes, the chips used in these measurements were not publicly released. IIRC he said that they were slightly lower clock speed than the publicly available chips.
- deleted 7y ago[deleted]
- tiffanyh 7y agoIt’s my understanding that many organizations prefer *BSDs over Linux not because BSD is more performant ... but because you’re way more likely to get your organizations patch accepted into the upstream than Linux.
- bsder 7y agoI doubt that is the case. The FreeBSD guys can be just as hardass about submissions as the Linux folks. However, there are considerations: 1) GPL vs BSD licenses. Companies like BSD licenses much more than GPL licenses. GPL adherents can whine all they want, but this is simply true. 2) BSD has a long history of having very good networking stacks--albeit on specific hardware. Linux supported everything initially--including really cheap crap--and consequently its networking stack was a lot more ad hoc. FreeBSD chose specific hardware for stability--but then supported that much more completely. 3) FreeBSD has a long history of being the servers in Internet infrastructure. There are specific architectural choices in the kernel because of this. There is probably still some inertia, too, in that the kind of old guard people who REALLY grok networking are still more comfortable on FreeBSD machines. Consequently, it is hardly surprising that an advanced networking development would take place on FreeBSD.
- throw0101a 7y agoRe: 1 and 2. Given that there are vendors that use FreeBSD for their appliances, they really don't want to have to send out techs to customers sites to fix things. So when the appliance makers choose hardware, they talk to component vendors about quality. It's no surprise that you see commits from Intel and Chelsio employees in the FreeBSD logs: companies like Netflix, Isilon, NetApp, and Juniper partner with them to make sure things aren't buggy. These collaborations lead to point 3.
- pekoribj 7y agoGood
- bluedino 7y agoNice to see AMD replacing Intel, they've gone with EPYC 7551 & 7502P, from 2x Intel “Skylake” / “Cascade Lake” Xeon edit: it looks like they hit 200GB/s with both Intel and AMD
- morning_gelato 7y agoIt's disappointing to hear that AMD has not provided adequate performance monitoring tools though. I would have thought by the second generation of Zen that this would be an area that AMD had given attention to.
- greglindahl 7y agoAMD's engineering team is much smaller, and they've always been behind on performance tools. HPC people have been complaining ever since Opteron. It's probably not going to change.
- drewg123 7y agoYes, we haven't gone with anything. These are proof of concept prototype systems. They serve real customer traffic in my testing, but are unicorns at this point.
- thomasjudge 7y agoThis may be a dumb question, but why would streaming video need to be encrypted? Is this just part of the "encrypt everything" best practice these days? Is there metadata accompanying the video data that shouldn't be unencrypted? Is it just so it's not possible to eavesdrop on the fact that I'm watching "Marvelous Mrs. Maisel"?
- 0xdeadb00f 7y agoThe main reason is probably DRM.
- deleted 7y ago[deleted]
- crispyporkbites 7y agoIt’s for privacy. That’s all you need to know.
- 7y ago
- amelius 7y agoIf they are just serving/streaming static data from a single server, wouldn't an entirely different architecture make more sense? For example, why even use a CPU? They have the financial resources to build their own hardware, e.g. encryption ASICs, and they can do interesting things like bundle the pipeline for multiple viewers watching the same movie at the same timepoint.
- virtuallynathan 7y agoCommodity server gear is really cheap, and is more or less "fast enough". The wins from serving say, >1Tbps from a 1U box with an ASIC or FPGA is negated by the downsides of having such a large failure domain, and the cost of development for what would be a pretty low volume part. We talk about this stuff every once in a while, but it doesn't really make sense to do right now. (Disclaimer: I work at Netflix)
- betaby 7y agoAre those changes in 12.1? If not, what would be a time frame?
- loeg 7y agoNot all of them are landed in CURRENT yet, much less stable/12. They'll be in 13.0; I can't speak to any future stable 12.2. On the other hand, CURRENT is pretty solid. A lot of folks, Netflix included, just run CURRENT.
- yjftsjthsd-h 7y agoWait, they run CURRENT in production? Is that... safe?
- jlgaddis 7y agoIn this particular (Netflix) case, what's the worst that could happen? One of their OCAs panics and falls over... and all customer traffic gets shifted to another OCA in the same or another location? Sure, they might run into some bug in -CURRENT, but what's the chance of a bunch of their OCAs all hitting it at the same time?
- morning_gelato 7y agoThere's at least one video online [1] that talks about Netflix's process for internal FreeBSD releases. They do 5 weeks of development and then 5 weeks of testing. [1] https://www.youtube.com/watch?v=vcyQBup-Gto https://www.youtube.com/watch?v=vcyQBup-Gto (about the 12 minute mark)
- branzo 7y agoI've been running CURRENT in production myself for about 2 years now. No problems so far.
- loeg 7y agoYes.
- 7y ago
- bt848 7y agoSo why is it important to have a multisocket NUMA machine? Why not just save yourself a lot of hassle by having one socket? I know that the previous generation AMD machine had unavoidable NUMA but the new one doesn't.
- o-__-o 7y agoSomething something performant concurrency
- baolongtrann 7y agoIt's money. Having a two-socket machine instead 2 one-socket machines, even when performance is only 80%, is saving both OPex and CAPex.
- toast0 7y agoOne bigger host meams less management overhead. (see other comments about being u willing to run multiple ips and what not) It may be a bit early to have a presentation from the latest Epyc processors. Most of the work was likely done with previous processors, but their slides said their AMD boxes are single socket.
- kylek 7y agoDensity/power/cooling are real constraints when you aren't 100% bought-in to the cloud.
- loeg 7y agoThis talk is about Zen+ Epyc, not Zen2 (which is where the non-cache memory gets uniform). I don't know if they have release quality Epyc 7003 (Zen2) samples available yet, and if they do, NFLX probably isn't allowed to publish benchmarks about them. There's almost certainly still some value in their existing NUMA work even on Zen2, as things like L1/L2/L3 cache have locality even if memory and PCIe does not. Pretty sure Intel single socket of this generation is totally non-viable for this workload due to lack of PCIe lanes. Maybe viable when Intel gets gen4 PCIe.
- bt848 7y ago
- jdsully 7y agoThe Linux network stack has been the bane of my existence trying to squeeze more performance out of KeyDB. I really hope it gets this kind of love in the future. On Linux there’s a spinlock in do soft_irq that blocks even in non-blocking IO.
- easytiger 7y agoHave you tried optimising and benchmarking on a 2.6 kernel? I found the more recent kennels in rhel7 that the amount of kernel work being done very hard to control. The first core on any numa node is especially problematic and induce horrific jitter
- shaklee3 7y agoWhy not use DPDK? All of these problems go away, and people have reported hitting 1Tbps on a single node.
- jdsully 7y agoWe have! https://docs.keydb.dev/blog/2019/06/17/blog-post/ https://docs.keydb.dev/blog/2019/06/17/blog-post/ Still though I hate seeing needless waste. There’s no reason the active thread needs to block on a soft_irq when there are unused cores to process them.
- morning_gelato 7y agoHave you tried reaching out to the linux netdev mailing list about this? They may at least be able to explain their design decision.
- shaklee3 7y agoCan you move all softirqs into different cores? That's usually one of the dpdk tuning steps.
- jdsully 7y agoMy reading of the source code is there is no way to prevent this. The spinlock is guarding the check which determines if there is work to do. Its been a few months and I had intended to go back and try using a try_lock(). But I’m not normally a kernel dev.
- pnako 7y agoLucky you, still living in the 90s. In our timeline, things are not looking that great. There was a big terrorist attack on US soil in 2001 and a series of wars in the middle east. We have computers that fit in your pocket. Scientists discovered something called the Higgs Boson and landed a probe on a comet. Microsoft released a lot of their code as open source. And Linux has achieved almost total supremacy over the server, mobile and supercomputer markets.
- alinspired 7y agoWhat a progress! In 2015 they were serving only ~9gb/sec https per server: https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf