13 ms·
These are the slides from my EuroBSDCon presentation. AMA
by drewg123 5y ago
These are the slides from my EuroBSDCon presentation. AMA
- GrigoriyMikh 5y agoTo be clear, was FreeBSD used because of historical reasons or because similar performance can't be/harder to achieve on Linux?
- smoldesu 5y agoI'd imagine scaling, licensing and overhead all had something to do with it, too.
- GrigoriyMikh 5y agoInteresting. Are there any benchmarks that you would recommend to look at, regarding FreeBSD vs Linux networking performance?
- GrigoriyMikh 5y agoFor anyone interested, here is benchs from late 2018, comparing Fedora and FreeBSD performance: https://matteocroce.medium.com/linux-and-freebsd-networking-cbadcdb15ddd https://matteocroce.medium.com/linux-and-freebsd-networking-...
- trasz 5y agoIt’s probably worth noting that there have been huge scalability improvements - including introduction of epochs (+/- RCU) - in FreeBSD over the last few years, for both networking and VFS.
- lostmsu 5y agoWhy does the author put so much effort into testing VMs? Bare metal installations aren't even tried, so the article won't represent more typical setup (unless you want to run in cloud, in which case it would make sense to test in cloud).
- genewitch 5y agoIf given the choice I'd never run anything on bare metal again. Let's say we have some service we want to run on a bare metal server. For not very much more hardware money, amortized, I can set up two or three VMs to do the same server, duplicated. Then if any subset of that metal goes bad, a replica/duplicate is already ready to go. There's no network overhead, etc. I've been doing this for stuff like SMTPE authority servers and ntpd and things that absolutely cannot go down, for over a decade.
- smoldesu 5y agoThat doesn't really matter for benchmarking purposes. I think the parent comment was emphasizing that syscall benchmarks don't make sense when you're running through a hypervisor, since you're running tacitly different instructions than would be run on a bare-metal or provisioned server.
- teknoraver 5y agoBecause the cards were in PCI passthrough, so the performance was exactly the same of a physical system
- lostmsu 5y agoAuthor mentions as much as to say that there's was some indirectness at least in the interrupts. There are also VirtIO drivers involved, and according to the article, they had effect too.
- crest 5y agoNginx and OpenSSL are open source. Give it a try and reproduce their results with Linux ;-).
- betaby 5y agosendfile + kTLS. I'm unaware of the in-kernel TLS implementation for Linux. Is there any around?
- merb 5y agohttps://www.kernel.org/doc/html/latest/networking/tls-offload.html https://www.kernel.org/doc/html/latest/networking/tls-offloa... https://docs.mellanox.com/display/OFEDv521040/Kernel+Transport+Layer+Security+%28kTLS%29+Offloads https://docs.mellanox.com/display/OFEDv521040/Kernel+Transpo...
- drewg123 5y agoYes, Linux has kTLS. When I tried to use it, it was horribly broken, so my fear is that its not well used/tested, but it exists. Mellanox, for example, developed their inline hardware kTLS offload on linux.
- Matthias247 5y ago> When I tried to use it, it was horribly broken, so my fear is that its not well used/tested, but it exists. Do you have any additional references around this? I'm aware that most rarely used functionality is often broken and therefore usually don't recommend people to use it, but would like to learn about kTLS in particular. I think for Linux OpenSSL 3 now added support for it in userspace. But there's also the kernel components as well as drivers - all of them could have their set of issues.
- drewg123 5y agoI recall that simple transmits from offset 0..N in a file worked. But range requests of the form N..N+2MB lead to corrupt data. Its been 2+ years, and I heard it was later fixed in Linux.
- Thaxll 5y agoI mean most CDN and FANG run on Linux, I think in that case it's kTLS that makes a big difference the rest not much.
- drewg123 5y agoAsync sendfile is also an advantage for FreeBSD. It is specific to FreeBSD. It allows an nginx worker to send from a file that's cold on disk without blocking, and without resorting to threadpools using a thread-per-file. The gist is that the sendfile() call stages the pages waiting to be read in the socket buffer, and marks the mbufs with M_NOTREADY (so they cannot be sent by TCP). When the disk read completes, a sendfile callback happens in the context of the disk ithread. This clears the M_NOTREADY flag and tells TCP they are ready to be sent. See https://www.nginx.com/blog/nginx-and-netflix-contribute-new-sendfile2-to-freebsd/ https://www.nginx.com/blog/nginx-and-netflix-contribute-new-...
- Thaxll 5y agosendfile() with splice and io_uring is similar? I know that this is very experimental on Linux. The overall idea is to copy bytes from disk to the socket with almost no allocation and not blocking, this is the idea right?
- Matthias247 5y agoMaybe. A few things in io-uring are implemented by letting a kernel task/thread block on doing the actual work. Which calls that are seems to change in every version, and might be tricky to find out without reading the kernel code.
- polskibus 5y agoWas FreeBSD your first choice? Or did you try with Linux first? What were the numbers for Linux-based solution, if there was one?
- drewg123 5y agoFreeBSD was selected at the outset of the Open Connect CDN (~2012 or so). We did a bake off a few years ago, and FOR THIS WORKLOAD FreeBSD outperformed Linux. I don't want to get into an OS war, that's not productive.
- jenny91 5y agoWas licensing also a contributing factor, or was that irrelevant for you?
- drewg123 5y agoMy understanding is that licensing did factor into the decision. However, I didn't join Netflix until after the decision had been made.
- tatersolid 5y agoI recall reading that Netflix chose FreeBSD a decade ago due to asynchronous disk IO was (and still is?) broken and/or limited to fixed block offsets. So nginx just works better on FreeBSD versus Linux for serving static files from spinning rust or SSD.
- otterley 5y agoHow would you characterize the economics of this vs. alternative solutions? Achieving 400Gb/s is certainly a remarkable achievement, but is it the lowest price-per-Gb/s solution compared to alternatives? (Even multiple servers.)
- pastrami_panda 5y agoThis, can someone add to this it's crucial.
- drewg123 5y agoI'm not on the hardware team, so I don't have the cost breakdown. But my understanding is that flash storage is the most expensive line item, and it matters little if its consolidated into one box, or spread over a rack, you still have to pay for it. By serving more from fewer boxes, you can reduce component duplication (cases, mobos, ram, PSUs) and more importantly, power & cooling required. The real risk is that we introduce a huge blast radius if one of these machines goes down.
- deleted 5y ago[deleted]
- sroussey 5y agoOuch. I guess there is always a trade off to be made…
- throwdbaaway 5y agoI almost fell for the hype of pcie gen4 after reading https://news.ycombinator.com/item?id=25956670 https://news.ycombinator.com/item?id=25956670, and it is quite interesting that pcie gen3 nvme drives can still do the job here. What would be the worst case disk I/O throughput while serving 400Gb/s?
- kofejnik 5y agothank you for this link, superuseful for my next project
- betaby 5y agoHow exactly 'Constrained to use 1 IP address per host' helps eliminate cross-NUMA transfers?
- drewg123 5y agoIf we could use more than 1 IP, then we could treat 1 400Gb box as 4 100Gb boxes. That could lead to the "perfect case" every time, since connections would always stay local to the numa node where content is present.
- betaby 5y agoThere is no kTLS for IPv6? IPv6 space is abundant and most mobiles is USA/Canada have IPv6. Won't that solve the problem?
- drewg123 5y agoIPv4 and IPv6 can both use kTLS. We offer service via V6, but most clients connect via IPv4. It differs by region, and even time of day, but IPv4 is still the vast majority of traffic.
- chungy 5y agoI've had to blackhole Netflix IPv6 ranges on my router because Netflix would identify my IPv6 connection as being a "VPN" even though it's not.
- drewg123 5y agoIf you could email me your ranges, I can try to look into it internally. Use my last name at gmail.com (or at freebsd.org)
- genewitch 5y agoI'm not GP; Hurricane Electric ipv6 always had Netflix think I was on a VPN, but now I have real ipv6 through the same ISP, I just pay more money so Netflix doesn't complain anymore.
- 404mm 5y agoAs an infrastructure engineer, these numbers are absolutely mind blowing to me! Not sure if it’s ok to ask.. how many servers like this one does it take to serve the US clients?
- drewg123 5y agoI don't have the answer, and even if I did, I'm not sure I'd be allowed to tell you :) But note that these are flash servers; they serve the most popular content we have. We also have "storage" servers with huge numbers of spinning drives that serve the longer tail. They are constrained by spinning rust speeds, and can't serve this fast.
- gct 5y agoI found somewhere that Netflix has ~74 million US/canada subscribers. If we guesstimate half of those might be on at peak time, that's 37 million users. At 400k connections/server that's only 85 servers to serve the connections, so I think the determining factor is the distribution of content people are watching.
- HalcyonicStorm 5y agoIs there anyway we can see a video of the presentation? I'm extremely interested
- allanjude 5y agoThe videos should appear on the conference's youtube channel in a few weeks: https://www.youtube.com/eurobsdcon https://www.youtube.com/eurobsdcon
- codetrotter 5y agoIs there something I as a person can contribute to make the videos available sooner? I have video editing skills and have also done some past-time subtitling of videos. I have all of the software necessary to perform both of these tasks and would be willing to do so free of charge.
- dragontamer 5y agoI eventually figured it out. But I would suggest maybe giving a brief 1-slide thingy on "lagg", and link aggregation? (Maybe it was clear from the presentation, but I only see the slides here soooooo...) I'm not the best at network infrastructure, though I'm more familiar with NUMA stuff. So I was trying to figure out how you only got 1-IP address on each box despite 4-ports across 2-nics. I assume some Linux / Windows devops people are just not as familiar with FreeBSD tools like that! EDIT: Now that I think of it: maybe a few slides on how link-aggregation across NICs / NUMA could be elaborated upon further? I'm frankly not sure if my personal understanding is correct. I'm imagining how TCP-connections are fragmented into IP-packets, and how those packets may traverse your network, and how they get to which NUMA node... and it seems really more complex to me than your slides indicate? Maybe this subject will take more than just one slide?
- zamadatix 5y agoLACP/link aggregation are IEEE Ethernet standards concepts supported by nearly every hardware or software network stack. https://en.wikipedia.org/wiki/Link_aggregation https://en.wikipedia.org/wiki/Link_aggregation
- drewg123 5y agoThanks for the feeback. I was hoping that the other words on the slide (LACP and bonding) would give enough context. I'm afraid that my presentation didn't really have room to dive much into LACP. I briefly said something like the following when giving that slide: Basically, the LACP link partner (router) hashes traffic consistently across the multiple links in the LACP bundle, using a hash of its choosing (typically an N-tuple, involving IP address and TCP port). Once it has selected a link for that connection, that connection will always land on that link on ingress (unless the LACP bundle changes in terms of links coming and going). We're free to choose whatever egress NIC we want (it does not need to be the same NIC the connection entered on). The issue is that there is no way for us to tell the router to move the TCP connection from one NIC to another (well, there is in theory, but our routers can't do it) I hope that helps
- dragontamer 5y ago
- minimax 5y agoWhat lead you to investigate PCIe relaxed ordering? Can you suggest a book or other resource to learn more about PCIe performance?
- drewg123 5y agoTo be honest, it was mostly the suggestion from AMD. At the time, AMD was the only Gen4 PCIe available, and it was hard to determine if the Mellanox NIC or the AMD PCIe root was the limiting factor. When AMD suggested Relaxed Ordering, that brought its importance to mind.
- gautamcgoel 5y ago1. Could GPU acceleration help at all? 2. When serving video, do you use floating point operations at all? Could this workload run on a hypothetical CPU with no floating point units? 3. How many of these hardware platforms do you guys own? 10k?100k?
- sroussey 5y agoThe videos are precomputed. So no GPU required to stream
- drewg123 5y ago1) No. Well, potentially as a crypto accelerator, but QAT and Chelsio T6 are less power hungry and more available. GPUs are so expensive/unavailable now that leveraging them in creative ways makes less sense than just using a NIC like the CX6-Dx, which has crypto as a low cost feature. 2) These are just static files, all encoding is done before it hits the CDN.
- toast0 5y agoI wonder if one creative (and probably stupid) way to leverage GPUs might be just as an additional RAM buffer to get more RAM bandwidth. Rather than DMA from storage to system RAM, and then from system RAM to the NIC, you could conceivably DMA to GPU RAM and then to the NIC for a subset of sends. Not all of the sends, cause of PCIe bandwidth limits. OTOH, DDR5 is coming soon and is supposed to bring double the bandwidth and double the fun.
- kvathupo 5y agoThis may be a naive question, but data is sent at 400Gb/s to the NIC, right? If so, is it fair to assume to assume that data is actually sent/received at a similar rate? I ask since I was curious why you guys opted not to bypass sendfile(2). I suppose it wouldn't matter in the event that the client is some viewer, as opposed to another internal machine.
- Matthias247 5y agoWell it's not 1 client. It's thousands of viewers and streams. An individual stream will have whatever the maximum 4k bandwidth for Netflix is.
- drewg123 5y agoWe actually try really, really hard not to blast 400Gb/s at a single client. The 400Gb/s is in aggregate. Our transport team is working on packet pacing, or really packet spreading, so that any bursts we send are small enough to avoid being dropped by the client, or an intermediary (cable modem, router, etc).
- kierank 5y agoHave you done any work to see whether the NIC hardware packet pacing mechanisms could improve QoE by reducing bursts?
- Matthias247 5y agoThe architecture slides don't show any in-memory read caching of data? I guess there is at least some, but would it be at the disk side or the NIC side? I guess sendfile without direct IO would read from a cache.
- drewg123 5y agoCaching is left off for simplicity. We keep track of popular titles, and try to cache them in RAM, using the normal page cache LRU mechanism. Other titles are marked with SF_NOCACHE and are discarded from RAM ASAP.
- markjdb 5y agoHow much data ends up being served from RAM? I had the impression that it was negligible and that the page cache was mostly used for file metadata and infrequently accessed data.
- drewg123 5y agoIt depends. Normally about 10-ish percent. I've seen well over that in the past for super popular titles on their release date.
- Matthias247 5y agoin which node would that page cache be allocated? In the one where the disk is attached, or where the data is used? Or is this more or less undefined or up to the OS?
- drewg123 5y agoThis is gone over in the talk. We allocate the page locally to where the data is used. The idea is that we'd prefer the NVME drive to eat any latency for the NUMA bus transfer, and not have the CPU (SW TLS) or NIC (inline HW TLS) stall waiting for a transfer.
- Aissen 5y agoHow do you benchmark this ? Do you use real-life traffic, or have a fleet of TLS clients ? If you have a custom testsuite, are the clients homogeneous ? How many machines do you need ? Do the clients use KTLS ?
- drewg123 5y agoWe test on production traffic. We don't have a testbench. This is problematic, because sometimes results are not reproducible. Eg, if I test on the day of a new release of a popular title, we might be serving a lot of it cached from RAM, so that cuts down memory bandwidth requirements and leads to an overly rosy picture of performance. I try to account for this in my testing.
- Aissen 5y agoThanks, it's an interesting set of tradeoffs.
- cnst 5y agoTest in prod FTW! However, how do you test the saturation point when dealing with production traffic? Won't you have to run your resources underprovisioned in order to achieve saturation? Doesn't that degrade the quality of service? Or are these special non-ISP Netflix Open Connect instances that are specifically meant to be used for saturation testing, with the rest of the load spilling back to EC2?
- drewg123 5y agoWe have servers in IX locations where there is a lot of traffic. Its not my area of expertise (being a kernel hacker, not a network architect), but our CDN load does not spill back to EC2. The biggest impact I have to QoE is when I crash a box, but clients are architected to be resilient against that.
- jareklupinski 5y ago> Mellanox ConnectX-6 Dx - Support for NIC kTLS offload Wild, didn't know nVidia was side-eyeing such far-apart but still parallel channels for their ?GPUs?. Was this all achievable using nVidia's APIs out-of-the-box, or did the firmware/driver require some in-house engineering :)
- fragmede 5y agoMellanox was bought by nVidia 2 years ago, so while it's technically accurate to say it's an nVidia card, that elides their history. Mellanox has been selling networking cards to the supercomputing market since 1999. Netflix absolutely had to do some tuning of various counters/queues/other settings in order to optimize for their workload and get the level of performance they're reporting here, but Mellanox sells NICs with firmware/drivers that work out-of-the-box.
- zamadatix 5y agoHey Drew, thanks for taking the time. What would you rate the relative complexity of working with the NIC offloading vs the more traditional optimizations in the rest of the deck? Have you compared other NIC vendors before or has Mellanox been the go to that's always done what you've needed?
- drewg123 5y agoI wish the video was online... We tried another vendor's NIC (don't want to name and shame). That NIC did kTLS offload before Mellanox. However, they could not retain TLS crypto state in the middle of a record. That meant we had to coerce TCP into trying really hard to send at TLS record boundaries. Doing this caused really poor QoE metrics (increase rebuffers, etc), and we were unable to move forward with them.
- LeifCarrotson 5y agoThis level of architecture management on big server CPUs is amazing! I occasionally handle problems like this on a small scale, like minimizing wake time and peripheral power management on an 8 bit microcontroller, but there the entire scope is digestible once you get into it, and the kernel is custom-designed for the application. However, in my case, and I expect in yours, requirements engineering is the place where you can make the greatest improvements. For example, I can save a few cycles and a few microwatts by sequencing my interrupts optimally or moving some of the algorithm to a look-up table, but if I can, say, establish that an LED indicator flash that might need to be 2x as bright but only lasts for a couple milliseconds every second is as visible as a 500ms LED on/off blink cycle, that's a 100x power savings that I can't hope to reach with micro-optimizations. What are your application-level teams doing to reduce the data requirements? General-purpose NUMA fabrics are needed to move data in arbitrary ways between disc/memory/NICs, but your needs aren't arbitrary - you basically only require a pipeline from disc to memory to the NIC. Do you, for example, keep the first few seconds of all your content cached in memory, because users usually start at the start of a stream rather than a few minutes in? Alternatively, if 1000 people all start the same episode of Stranger Things within the same minute, can you add queues at the external endpoints or time shift them all together so it only requires one disk read for those thousand users?
- fragmede 5y ago> Alternatively, if 1000 people all start the same episode of Stranger Things within the same minute It would be fascinating to hear from Netflix on some serious details of the usage patterns they see and particular optimizations that they do for that, but I doubt there's so much they can do given the size of the streams, the 'randomness' of what people watch and when they watch, and for the fact that the linked slides say the servers have 18x2TB NVME drives per-server and 256GB. I wouldn't be surprised if the Netflix logo opener exists once on disk instead of being the first N seconds of every file though.
- allanjude 5y agoIn previous talks Netflix has mentioned that due to serving so many 1000s of people from each box, that they basically do 0 caching in memory, all of the system memory is needed for buffers that are enroute to users, and they purposely avoid keeping any buffer cache beyond what is needed for sendfile()
- swozey 5y agoI build this stuff so it's so cool to read this, I can't really be public about my stuff. Are you using completely custom firmwares on your Mellanoxes? Do you have plans for nvmeOF? I've had so many issues with kernels/firmware scaling this stuff that we've got a team of kernel devs now. Also how stable are these servers? Do you feel like they're teetering at the edge of reliability? I think once we've ripped out all the OEM firmware we'll be in a much better place. Are you running anything custom in your Mellanoxes? dpdk stuff
- drewg123 5y agoNo, nothing custom, we don't have access to the source. We like to run each server as an independent entity, so we don't run NVMEof. They're pretty much rock solid. If you'd like to discuss more, ping me via email. Use my last name at gmail . com.
- caskstrength 5y agoIt is hard to decipher your message, but just to clarify, firmware doesn't process packets (dataplane) it only manages and configures hardware (control plane). And no, you definitely won't be in "much better place" by "ripping it off" because modern NICs have very complex firmware with hundreds (if not thousands) man/years spent implementing and optimizing it.
- swozey 5y agoBud, I work on this stuff, I know all about Cavium and Mellanox firmware and it's issues, specifically with things like the math/integers used to determine packetflow which is used to charge customers having issues on specific versions that that have been internally patched. Do you think I just randomly typed this? It's even worse now that a huge chunk of their firmware teams have been lost in the shuffle of all of the competitors being bought and brought under one umbrella. Do you recall a bug on AWS years ago where they were incorrectly charging customers bandwidth usage? Everyone uses these types of nics, tons of PAAS/SAAS corps had that problem. What an obnoxious, pedantic and naive response. "And no, you definitely won't be in "much better place" by "ripping it off" because modern NICs have very complex firmware with hundreds (if not thousands) man/years spent implementing and optimizing it." What do you think I just explained? I literally write this stuff. Modern nic firmware is still written in C and still has bugs. Do you seriously think drivers/firmware aren't going to have bugs? You just said yourself that they're high complexity. I can't believe I'm even needing to explain this. You've clearly never been a network engineer, do you have any idea how many bugs are in Juniper, Cisco, Palo, etc? If you don't work on bare metal architecture/distributed systems, move along. This isn't sysadmin talk. Almost nothing used is stock, even k8s gets forked due to bugs. You can't resell PAAS with bugs that conflict with customer billing. NFLX isn't reselling bandwidth so they likely don't encounter these issues, they're using something like Cedexis to force their CDN providers at the edge to compete with one another down to the lowest cost/reliability and they're liable for things like this and can be sued/loss based on their contract. They (CDN customers) are acutely aware of when billing doesn't match up with realized BW usage. They'll drop the traffic to your CDN and split more of it over to a "better" CDN until those issues are mitigated - and while you're not getting that traffic you're not getting that customers exoected monthly payment... because now they're buying less bandwith from you because Cedexis tells them that you're less performant/reliable. ANY large customer buying bandwidth from a CDN does this, none of this is specific to NFLX whom I know nothing about beyond.. "this is how reselling PAAS works." I bet the next response is "no way NFLX uses a CDN".. LOL. They all do, my friend, HBO, paramount, disney, etc. They aren't in the biz of edge caching.
- codetrotter 5y agoWhat architectures are you guys running FreeBSD on? Would these techniques be applicable on arm64 and/or riscv64?
- wh313 5y agoDoes FreeBSD `sendfile` avoid a context switch from userspace to kernelspace as well, or is it only zero-copy? I've worked with 100Gbps NICs and had to end up using both a userspace network stack and a userspace storage driver on Linux to avoid the context switch and ensure zero-copy. Also, have you looked into offloading more of the processing to an FPGA card instead?
- drewg123 5y agoThere is no context switch, like most system calls, sendfile runs in the thread context of the thread making the syscall. FreeBSD has "async sendfile", which means that it does not block waiting for the data to be read from disk. Rather, the pages that have been allocated to hold the data are staged in the socket buffer and attached to mbufs marked "not ready". When the data arrives, the disk interrupt thread makes a callback which marks the mbufs "ready", and pokes the TCP stack to tell them they are ready to send. This avoids the need to have many threads parked, waiting on disk io to complete.
- altilunium 5y agoSlide 5. What is the difference of "mem bw" and "networking units"?
- denizgenc 5y agoNetworking tends to use bits per second instead of bytes per second, so in order to more easily compare the memory bandwidth to the rest of the values used in the presentation, the presenter multiplied the B/s values by 8 to get the corresponding b/s values.
- altilunium 5y agoOh.. Networking unit is using "bit" instead of "byte".