19 ms·
Serving Netflix Video at 400Gb/s on FreeBSD [pdf]
- msdrigg 5y agoI think its funny that these servers use FreeBSD but if you zoom in on the CPU graphic in the slides it is reading a book called ‘Programming Linux’
- speedgoose 5y agoGood catch.
- drewg123 5y agoWow, I never zoomed in on that. I've been using that as clip art in my slides for years now. You should work for CSI
- cucumb3rrelish 5y agothis guy enhances
- tyingq 5y agoThat is funny. The title is pretty damn small. Here's what appears to be the source SVG: https://openclipart.org/download/28107/klaasvangend-processor-idle.svg https://openclipart.org/download/28107/klaasvangend-processo... And the text is in a <tspan>, so it could be appropriately re-titled if desired :)
- jhgb 5y ago> so it could be appropriately re-titled if desired ...to something like "How to not be too curious for your own good"?
- deleted 5y ago[deleted]
- drewg123 5y agoThese are the slides from my EuroBSDCon presentation. AMA
- GrigoriyMikh 5y agoTo be clear, was FreeBSD used because of historical reasons or because similar performance can't be/harder to achieve on Linux?
- smoldesu 5y agoI'd imagine scaling, licensing and overhead all had something to do with it, too.
- GrigoriyMikh 5y agoInteresting. Are there any benchmarks that you would recommend to look at, regarding FreeBSD vs Linux networking performance?
- GrigoriyMikh 5y agoFor anyone interested, here is benchs from late 2018, comparing Fedora and FreeBSD performance: https://matteocroce.medium.com/linux-and-freebsd-networking-cbadcdb15ddd https://matteocroce.medium.com/linux-and-freebsd-networking-...
- trasz 5y agoIt’s probably worth noting that there have been huge scalability improvements - including introduction of epochs (+/- RCU) - in FreeBSD over the last few years, for both networking and VFS.
- lostmsu 5y agoWhy does the author put so much effort into testing VMs? Bare metal installations aren't even tried, so the article won't represent more typical setup (unless you want to run in cloud, in which case it would make sense to test in cloud).
- betaby 5y agoAnd yet a lot of folks think Netflix is on AWS. I would think AWS gives a huge discount to Netflix and ask 'just run at least something on aws'. Those promotional videos have nothing useful about netflix on aws https://aws.amazon.com/solutions/case-studies/netflix-case-study/ https://aws.amazon.com/solutions/case-studies/netflix-case-s... and targeted to managers and unaware of the scale engineers.
- dangrossman 5y agoNetflix is on AWS -- the website and app backends, not the video streams. * Netflix.com resolves to ec2-52-3-144-142.compute-1.amazonaws.com from my computer at this moment. Some more domains their apps reach out to include: * nflxvideo.net: ec2-46-137-171-215.eu-west-1.compute.amazonaws.com. * appboot.netflix.com: ec2-34-199-156-113.compute-1.amazonaws.com. * uiboot.netflix.com: ec2-52-203-145-205.compute-1.amazonaws.com.
- betaby 5y agoThat exactly what I mean. AWS would represent single digit of the infrastructure and traffic.
- dangrossman 5y agoMaybe of their traffic, but not their infrastructure. Netflix rents 300,000 CPUs on AWS just for video encoding. Encoding one TV series produces over 1,000 files that sit on S3, ready for those PoPs to pull them in and cache them. They also do more data processing, analytics and ML than most people realize and that's all happening on AWS. For example, not only are each user's recommendations personalized, but all of the artwork is personalized too. If you like comedy, the cover art for Good Will Hunting may be a photo of Robin Williams. If you like romance, the cover art for the same movie may be a photo of Matt Damon and Minnie Driver going in for a kiss. They're also doing predictive analytics on what each person will want to watch tomorrow in particular. Every evening, all those non-AWS servers go ask AWS what video files they should download from S3 to proactively cache the next day of streaming.
- cleverpebble 5y agoIf this is one server, I can't imagine how much bandwidth Netflix pushes out of a single PoP - does anyone have those numbers, or at least estimates?
- wmf 5y agoI saw a photo somewhere of a Netflix POP that showed dozens of servers and a huge chassis router with hundreds of 100G links, so that's terabits.
- tgtweak 5y agoCapacity isn't indicative of bandwidth peak - but I'm sure peaks are well past the terabit measure of each server is being designed for 100-400Gbps.
- tgtweak 5y agoThey colocate these servers inside ISPs typically. Would be interesting to know, for the traffic that does go to a Netflix primary pop - how much the hottest one consumes. It feels like a luxury that they can fit their entire content library in all encoded formats into a single server (even though it's a massive server).
- sroussey 5y agoIt’s not the entire content library. It’s a cache of the most watched at that time.
- dylan604 5y agoThere were some crazy number like 80% of internet traffic is Netflix. I have no idea of the validity of that, but without seeing actual numbers, that sounds like a lot.
- kingosticks 5y agoThere are some numbers at https://www.lightreading.com/carrier-security/security-strategies/cybersecurity-expert-sees-opportunity-in-internet-growth-consolidation/a/d-id/752196 https://www.lightreading.com/carrier-security/security-strat...
- lprd 5y agoWill the presentation be available online anywhere? This topic is very interesting to me as I maintain a Plex server for my family and have always been curious how Netflix does streaming at scale.
- antman 5y agoHow large is your family?
- lokimedes 5y agoJust the closest /16 and a few friends.
- lprd 5y agoHaha, not large at all! I've just built a couple of servers and their main responsibility is content storage and then content serving (transcoding). My main bottleneck right now is that I don't have 10Gbps switches (both servers have dual 10Gbps intel NICs ready to go). I have to do some throttling so that users don't experience any buffering issues.
- bduhan 5y agoHave you tried using a crossover cable and static IPs to link the two servers directly?
- stepbeek 5y agoFYI if using modern hardware then you might not need a specific cable to try this thanks to Auto MDI-X: https://en.wikipedia.org/wiki/Medium-dependent_interface#Auto_MDI-X https://en.wikipedia.org/wiki/Medium-dependent_interface#Aut...
- erk__ 5y agoAt some point the presentations will come up here https://www.youtube.com/c/EuroBSDcon https://www.youtube.com/c/EuroBSDcon
- tester756 5y ago>Serving Netflix Video at 400Gb/s [pdf] Why not 50GB/s? We don't say 50000 m/h instead of 50km/h
- deleted 5y ago[deleted]
- nix23 5y agoWow you have much to learn buddy.
- drewg123 5y agoBecause its network bandwidth, and network units are in bits for historical reasons. I actually griped about this in my talk .. it makes it a pain to talk about memory bw vs network bw without mental gymnastics to move around that factor of 8
- tester756 5y agothank you
- jakeinspace 5y agoPersonally, I think that keeping the bit as the default network bandwidth unit is a good idea. When expressing physical transmission limitations, the equations of information theory are all in terms of bits (being the smallest unit of classical information). Obviously, this means that somewhere on the spectrum from the Shannon Limit to storage on disk, we've gotta shift from bits to bytes. The NIC is as good a border as any.
- kierank 5y agoHow do you think this will need to be re-architected for HTTP3? Will you do kernel bypass with a userspace stack?
- drewg123 5y agoWe've looked at http3 / quic, and have not yet seen any advantages for our workload. Just a large loss of CPU efficiency.
- anonuser123456 5y agoOut of curiosity, why TLS encrypt when it's already DRM? To prevent snooping on viewer habits?
- numlock86 5y agohttps://doesmysiteneedhttps.com/ https://doesmysiteneedhttps.com/ Maybe they should add "But I already have DRM!" to the list. DRM solves a complete different problem.
- boardwaalk 5y agoMore generically, “But my content is already encrypted!” I’m surprised it isn’t already there.
- drewg123 5y agoYes. See a tech blog from last year: https://netflixtechblog.com/how-netflix-brings-safer-and-faster-streaming-experience-to-the-living-room-on-crowded-networks-78b8de7f758c https://netflixtechblog.com/how-netflix-brings-safer-and-fas...
- tgtweak 5y agoCool but how much does one of those servers cost... 120k? Pretty crazy stuff though, would love to see something similar from cloudflare eng. since those workloads are extremely broad vs serving static bits from disk.
- dragontamer 5y agohttps://www.thinkmate.com/system/a+-server-1114s-wtrt https://www.thinkmate.com/system/a+-server-1114s-wtrt For a EPYC 7502P 32-core / 256GB RAM / 2x Mellanox Connect-X 6 dual-nic, I'm seeing $10,000. Then comes the 18x SSD drives, lol, plus the cards that can actually hold all of that. So the bulk of the price is SSD+associated hardware (HBA??). The CPU/RAM/Interface is actually really cheap, based on just some price-shopping I did.
- zamadatix 5y agoThe WD SN720 is an NVMe SSD so there are no HBAs involved it just plugs into the (many) PCIe lanes the Epyc CPU provides. CDW gives an instant anonymous buyer web price of ~$11,000.00 for all 18.
- kristjansson 5y agoSurprisingly less it seems? Those NICs are only like a kilobuck each, the drives are like 0.5k, CPU is like 3k-5k. So maybe 15k-20k all in, with the flash comprising about half that? Seems surprisingly cheap, but I’m not sure if that’s just great cost engineering on Netflix part or a poor prior on my part … I’ll chose to blame Nvidia’s pricing in other domains for biasing me up
- virtuallynathan 5y agoThat’s in the right ballpark, I believe.
- daper 5y agoI think the described machines were designed for a very specific workloads even in CDNs world. As I guess they serve exclusivelly fragments of on-demand videos, no dynamic content. So high bytes to number of requests ratio, nearly 100% hit ratio, very long cache TTL (probably without revalidation, almost no purge requests), small / easy to interpret requests, very effective HTTP keep alive, relatively small number of new TLS sessions (TLS handshake is not accelerated in hardware), low number of objects due to their size (much fewer stat()/open() calls that would block nginx process). Not talking about other CDN functionality like workers, WAF, page rules, rewrites, custom cache keys, no or very little logging of requests etc. That really simplifies things a lot compared to Cloudflare or Akamai.
- morty_s 5y agoEnjoying the slides! Is there a small typo on slide 17? >Strategy: Keep as much of our 200GB/sec of bulk data off the NUMA fabric [as] possible
- drewg123 5y agoNo, 400Gb/s == 50 GB/s With software kTLS, the data is moved an and out of memory 4x, so that's 200GB/s of bandwidth needed to serve 50GB/s (400Gb/s) of data.
- justin_ 5y agoThere is no video for this available yet AFAIK, but for those interested there is a 2019 EuroBSDcon presentation online from the same speaker that focuses on the TLS offloading: https://www.youtube.com/watch?v=p9fbofDUUr4 https://www.youtube.com/watch?v=p9fbofDUUr4 The slides here look great. I'm looking forward to watching the recording.
- _huayra_ 5y agoDrew also gave this talk about NUMA optimization that I found enjoyable (from the same conf): https://www.youtube.com/watch?v=8NSzkYSX5nY https://www.youtube.com/watch?v=8NSzkYSX5nY
- alberth 5y ago@drewg123 1. Is there any Likelihood Netflix needs will migrate to ARM in thr next few years? (I see you’re right up at end of the deck, curious if you’re seeing more advancements with ARM than x86 and as such, project ARM to surprise x86 for your needs in the foreseeable future) 2. Can you comment more on thr 800 Gbps reference at end of deck
- drewg123 5y agoWe're open to migrating to the best technology for the use case. Arm beat Intel to Gen4 PCIe, which is one of the reasons we tried them. Our stack runs just fine on arm64, thanks to the core FreeBSD support for arm64 The 800Gb is just a science experiment.. a dual-socket milan with a relatively equal number of PCIe lanes from each socket. It should have been delivered ages ago, but supply chain shortages impacted the delivery dates of a one-off prototype like this.
- azinman2 5y agoWhat’s amazing to me is how much data they’re pumping out — their bandwidth bills must be insane. Or they did some pretty amazing negotiating contracts considering Netflix is like what, $10-15/mo? And there are many who “binge” many shows likely consuming gigabytes per day (in addition to all the other costs, not least of which is actually making the programming)?
- dragontamer 5y agoOne of the funny truths about the business of networks, is that the one who "controls" the most bandwidth has the #1 negotiating spot at the table. Take for example, if say Verizon decided to charge more for bandwidth to Netflix... if Netflix said "no" and went with another provider, then Verizon's customer's would suffer from worse access times to Netflix. Verizon has the advantage in that they have a huge customer base that no one wants to piss off Verizon. So it cuts both ways. Bandwidth becomes not a cost at this scale, but instead a moat.
- harikb 5y agoMy hypothesis is that if Netflix/Youtube hadn't stressed out our bandwidth and forced the ISPs of the world upgrade for the last decade, the world wouldn't have been ready for WFH of the covid world. ISPs would have been more than happy to show the middle finger to the WFH engineers, but not to the binge-watching masses.
- elithrar 5y ago> My hypothesis is that if Netflix/Youtube hadn't stressed out our bandwidth and forced the ISPs of the world upgrade for the last decade, the world wouldn't have been ready for WFH of the covid world. Couldn't agree more. We see the opposite when it comes to broadband monopolies: "barely good enough" DSL infrastructure, congested HFC, and adversarial relationships w.r.t. subscriber privacy and experience. When it became worthwhile to invest in not just peering but also last-mile because poor Netflix/YouTube/Disney+/etc performance was a reason for users to churn away, they invested. This isn't to say that this is all "perfect" for consumers either, but this tension has only been good for consumers vs. what we had in the 90's and early-mid 00's.
- ramshanker 5y agoMeanwhile my internet connection is 40mbps
- Sunspark 5y agoIt's disappointing to me that Netflix keeps reducing the bitrate on their videos.. they're not the only offender but the most prominent.
- deleted 5y ago[deleted]
- woah 5y agoDumb question, but what, specifically, does "fabric" mean in this context?
- creativenolo 5y agoI'd love to know how many users could be served off one of these boxes
- puppet-master 5y agoYou can do reasonable 1080p at 3Mbit with h.264, if these were live streams you could fit around 130k sessions on the machine. But this workload is prerecorded bulk data, I imagine the traffic is considerably spikier than in the live streaming case, perhaps reducing the max sessions (without performance loss) by 2-3x.
- arnaudsm 5y agoSlightly unrelated, but did they ever consider P2P? It scales really well and saves Terabytes of bandwidth, so I wonder what the cons are, except that's it's associated with illegal distribution.
- fortuna86 5y agoMaybe dumb question, but does p-2-p work on native TV apps, chromecast, etc ? I know it does if you run a client app on Windows or Mac
- zamadatix 5y agoThat's not P2P in quite the same sense of the above was talking about. Chromecast is client/server it's just your device can become the server or instruct the Chromecast to connect to a certain server.
- fortuna86 5y agoRight, but can that realistically be done?
- wmf 5y agoIf Netflix had made P2P the standard ten years ago then TVs and Chromecasts would support P2P today. But they didn't so they don't.
- deleted 5y ago[deleted]
- zamadatix 5y agoP2P doesn't offer as many experience consistency guarantees and in general adds a lot of complexity (=cost) that doesn't exist in a client server model. Even if you went full bore on P2P you still have to maintain a significant portion of the centralized infrastructure both to seed new content as well as for clients that aren't good P2P candidates (poor connections, limited asymmetric connections, heavily limited data connections). Once you got through all of those technical issues even if you found you could overall save cost by reducing the core infrastructure... it rides on the premise customers are going to be fine with their devices using their upload (both active bandwidth and total data) at the same price as when it was a hosted service. But yes I imagine licensing is definitely an issue too. Right now only certain shows can be saved offline to a device and only for very restricted periods of time for the same reason. It's also worth noting in many cases Netflix doesn't pay for bandwidth.
- segfaultbuserr 5y ago> Run kTLS workers, RACK / BBR TCP pacers with domain affinity TCP BBR is in the FreeBSD upstream now? Cool.
- zamadatix 5y agoAs of 13.0
- gigatexal 5y agoHilarious to see all the attempts at “but why not Linux”. I wonder how many will read this and consider trying out FreeBSD. It’s a rather dope OS but I am a bit biased.
- 5faulker 5y agoSeems like an absolute overkill for now.
- lvh 5y agoI don't see why it's "absolute overkill" or even "for now"? It seems reasonable for Netflix to want to optimize that pipeline and minimize the amount of hardware they need to deploy, especially for dense areas? Top quality video from Netflix can be 2MB/s, so back-of-the-envelope that's only 20k streams. I'd expect needing that capacity on any random evening in Logan Square (a Chicago neighborhood).
- mrlonglong 5y agoAt 400Gb/s, how long would it take to read through the entire catalogue?
- fabioyy 5y agohave you considered dpdk?
- TedShiller 5y agoFor those wondering about NUMA (Non Uniform Memory Architecture) here is an explanation: https://m.youtube.com/watch?v=KmtzQCSh6xk https://m.youtube.com/watch?v=KmtzQCSh6xk
- NeckBeardPrince 5y agoI hate and respect you at the same time.
- oliv__ 5y agoAssociate network connections with NUMA nodes ● Allocate local memory to back media files when they are DMA’ed from disk ● Allocate local memory for TLS crypto destination buffers & do SW crypto locally ● Run kTLS workers, RACK / BBR TCP pacers with domain affinity ● Choose local lagg(4) egress port All of this is upstream! This sounds like some alien language
- pts_ 5y agoWelcome to devops.
- qwertox 5y agoThis is not devops. This is plain "high performance video delivery".
- hagbarddenstore 5y agoWelcome to system operations (sysop). We've been doing this before devops was even a word.
- stingraycharles 5y agoBasically, everything is about “pinning” the hardware on a low level. You pin the network adapters to be handled by a specific set of cores. You pin the filesystem to handle specific sets of files on specific cores. You then ensure the router in the rack to distribute the http requests to the network ports exactly in the way that they always arrive on the network adapters that have those files pinned. It’s not much different from partitioning / sharding and “smart” load balancing a cluster of servers, it’s just on a lower level of abstraction.
- oliv__ 5y agoThere's something amazing in the complexity of this document and the engineering described but also incredibly depressing when you start to think that people so smart they can achieve this are working on increasing computer bandwidth for entertainment and yet we still haven't solved our own environmental footprint.
- scrubs 5y agoFor those with know how, where can I get a technical paper which describes this numa architecture. A server with multiple nics with a peferred core association is of interest. Presumably this needs Linux support too.