19 ms·
Serving Netflix Video Traffic at 800Gb/s and Beyond [pdf]
- acd 4y agoPossibly speed increase: Switch from TCP to UDP protocol for even faster video delivery send speed? Why use TCP for video content? Packet loss but then one would use http3 like UDP tech?
- amelius 4y agoHow many customers does that serve?
- Aissen 4y agoAt 15Mb/s for a start-quality 4k stream (5 times higher than the average ISP speed measured by Netflix), that serves 53k simultaneous customers. In the US, the fastest ISP for Netflix usage seems to be Comcast (https://ispspeedindex.netflix.net/country/us https://ispspeedindex.netflix.net/country/us ), with an average speed of 3.6Mbps. That would serve an average of 222k simultaneous customers on a single server.
- samcrawford 4y agoThat 15Mb/s figure for 4K is out of date by a couple of years. They previously targeted a fixed average bitrate of 15.6Mb/s. They now target a quality level, as scored by VMAF. This makes their average bitrate for 4K variable, but they say it has an upper bound of about 8Mb/s. See https://netflixtechblog.com/optimized-shot-based-encodes-for-4k-now-streaming-47b516b10bbb https://netflixtechblog.com/optimized-shot-based-encodes-for...
- Aissen 4y agoYep, that's correct. It looks like Netflix forgot to update their support pages for this: https://help.netflix.com/en/node/306 https://help.netflix.com/en/node/306 .
- Gigachad 4y agoMore likely that they just wanted to keep it at that as a kind of worst case scenario. If you meet their recommended spec, there should be no way you will have issues.
- School-Cotton 4y agoWhat does start-quality mean?
- Aissen 4y agoNot much, see sibling comment. It used to be the minimum quality for enjoyable 4k. (4k Blu-Ray discs have much higher bitrates with HEVC). But since, Netflix heavily optimized their encoding, greatly reducing the bandwidth needs.
- danielheath 4y agoVideo formats require more data for the first frame of each scene - subsequent frames can be encoded as transformations of the previous frame.
- deleted 4y ago[deleted]
- ksec 4y agoThat depends on the content's bitrate. Netflix serves their video with bitrate anywhere from 2 - 18Mbps. Say if average were 10Mbps, that is roughly 80K customer per box.
- ksec 4y agoAnd this is still on ConnectX-6 Dx, with PCI-Gen 5 and ConnectX-7, Netflix should be able to push for 1.6Tbps per box. This will hopefully keep drewg123 and his team busy for another year :P
- dragontamer 4y agoAt that point, RAM itself would likely be the bottleneck. But maybe DDR5 will come out by then and get this team busy again lol.
- wmf 4y agoGenoa does indeed have roughly double the memory bandwidth.
- mschuster91 4y agoThey... serve 800 gigabytes a second on one single content server, do I get that right?
- Aissen 4y agoHow do you deal with the higher power density of these servers that needs to be put at the ISP locations ? Don't they have some constraints for the open connect machines ?
- kkielhofner 4y agoThe Dell R7525 chassis is available with dual 800w power supplies. General thinking for power supplies is that each power supply is connected to completely separate power distribution - independent cabling, battery backup, generators, and points of entry to the facility. In many cases it's also two different power grids. This is so that if one power source fails anywhere the load can move over to the other power supply without exceeding the power that can be delivered through a single power supply or trip a breaker anywhere. Under normal operating conditions each power supply is doing half the load. Additionally, the National Electric Code in the US specifies that continuous load should not exceed 80% of given circuit/breaker capacity. So with dual 800 power supplies at "max" 80% load that's "only" 640 watts for one of these 2U servers. For 208V power that's only 3 amps. High density (for sure) compared to the old days but not as ridiculous as it may seem.
- Aissen 4y agoYou're right, it's not that much for 2U. But for this config, I think they'd probably go for the 1400W power supplies: - 16 x 25W SSDs - 2 x 225W CPUs - On top of that, add RAM, cooling, etc. Honestly, it's still manageable. I doubt they'd put 10 of those in a single rack (you'd need an ISP that would want serve 2.2M subscribers in peak from a single location, not necessarily desirable on their side); but if the site is getting full, you'd feel the (power) pressure (slowly) mounting.
- kkielhofner 4y agoI didn't dig into the CPU config, etc but you're right they'd probably go for the 1400W power supplies which is 5.4 amps max at 208V. It's for an older config (based on the other specs) but the current Netflix OpenConnect docs call for 750w[0] which is more reasonable even for this hardware configuration because no one really wants to consistently run their power supplies at 80% (even in branch loss) for obvious reasons. They absolutely wouldn't want to concentrate them. The entire purpose is to reduce ISP network load and get as close to the customer eyeballs as possible. I don't have any experience with these but I imagine ISPs would install them at their peering "hubs" in major cities - in my experience the usual suspects like Chicago, NYC, Miami, etc. [0] - https://openconnect.zendesk.com/hc/en-us/articles/360034538352 https://openconnect.zendesk.com/hc/en-us/articles/3600345383...
- paxys 4y agoI love technical content like this. Not only is it incredibly interesting and informative, it also serves as a perfect counterpoint to the popular "why does Netflix need X thousand engineers, I could build it in a weekend" sentiment that is frequently brought up on forums like this one. Building software and productionalizing/scaling it are two very different problems, and the latter is far more difficult. Running a successful company always requires an unlimited number of very smart people who are willing to get their hands dirty optimizing every aspect of the product and business. Too many people today think that programming starts and ends at pulling a dozen popular libraries and making some API calls.
- fennecfoxy 4y agoLmao. _But it does_. At least for the most part. It's such a previous generation thing to be angry at the way that modern development is done. Of _course_ dev is now web heavy and people are pulling in all sorts of libraries to make their lives easier. But pretending that those same devs wouldn't also be capable of development on lower/less abstracted levels (well sure, maybe not _all_ of them) is insulting.
- le-mark 4y ago> Too many people today think that programming starts and ends at pulling a dozen popular libraries and making some API calls. The needle keeps moving doesn’t it? A tremendous breadth of difficult problems can be effectively addressed by pulling together libraries and calling APIs today that weren’t possible before. Today’s hard problems are yesterday impossibilities. The challenge for those seeking to make an impact is to dream big enough.
- ezconnect 4y agoThe basic problem is the same, pushing the hardware to its limits.
- whatshisface 4y agoThe basic problem is delivering value to someone.
- theideaofcoffee 4y agoThis is a remarkable technical achievement that builds on all of its past work, as are the other updates from Netflix in the past with serving ever more traffic from a single box. That said, I still find it terrifying that so many users would be affected by a single machine going down, that blast radius is so huge! Do we know if the rates that these hosts serve actually make it into production? Or do they derate the amount they serve from a single host and add others?
- lanstin 4y agoI think they buffer and if the stream has issues the client connects to another host. They have been doing chaos monkey for a long long time.
- drewg123 4y agoAs I said in a parallel comment, this is a testbed platform to see what problems we'll encounter running at these speeds. Production hosts are single socket, and can run at roughly 1/2 this speed. I regret that I've crashed boxes doing hundreds of Gb/s. Thankfully our stack is resilient enough that customers didn't notice.
- wistlo 4y agoThis could be an answer as to why Netflix comes up reliably when all the other streaming services in my experience (Hulu, Disney, HBO Max, Amazon Prime) can take many multiples of time to initialize and deliver a stable stream.
- OJFord 4y agoOf those I only have Prime, and really agree. It was never as good I don't think, but lately in particular it's been so slow to start (and then it's an advert! It'll do it again for the actual content once I click 'skip'!) and occasionally pauses to buffer mid-stream. I don't get that with Netflix, I've occasionally had it crash out 'sorry this could not be played right now' (which is a weird bug itself - because it always loads fast & fine when I immediately press play on it again) but never such slow loading or pausing.
- drewg123 4y agoTo be honest, this has much more to do with Randall Stewart's RACK TCP, and his team's obsession with improving our member's QoE. Ironically, this costs a lot of CPU as compared to normal TCP (since it is doing pacing, limiting TSO burst sizes, etc). https://github.com/freebsd/freebsd-src/blob/main/sys/netinet/tcp_stacks/rack.c https://github.com/freebsd/freebsd-src/blob/main/sys/netinet...
- frankjr 4y agoHere's the accompanying video: https://cdnapisec.kaltura.com/index.php/extwidget/preview/partner_id/2539171/uiconf_id/44037791/entry_id/1_q714dkb3/embed/dynamic https://cdnapisec.kaltura.com/index.php/extwidget/preview/pa...
- antonio-ramadas 4y agoI found the same video on the website of the summit: https://nabstreamingsummit.com/videos/2022vegas/ https://nabstreamingsummit.com/videos/2022vegas/ I’m on mobile and there does not seem to exist a direct link. Search for: “Case Study: Serving Netflix Video Traffic at 400Gb/s and Beyond”
- deleted 4y ago[deleted]
- paxys 4y agoDoes anyone know where these servers are hosted? Certainly not AWS I imagine?
- kkielhofner 4y agoAs close to the eyeballs as possible. With OpenConnect[0] they are located in ISP facilities and/or carrier-neutral facilities with access to a peering fabric (kind of the same thing as the OpenConnect Appliance is "hosted" by the ISP). It's a win-win. ISPs don't have to use their peering and/or transit bandwidth to upstream peers and users get a much better experience with lower latency, higher reliability, less opportunity for packet loss, etc. [0] - https://openconnect.netflix.com/en/ https://openconnect.netflix.com/en/
- Moral_ 4y agoA lot of the reasons they've had to build most of this stuff themselfs is because they decided for some reason to use freeBSD. The NUMA work they did, I remember being in a meeting with them as a Linux Developer at Intel at the time. They bought NVMe drives or were saying they were going to buy NVMe drives from Intel which got them access to "on the ground" kernel developers and CPU people from Intel. Instead of talking about NVMe they spent the entire meeting asking us about howt the Linux kernel handles NUMA and corner cases around memory and scheudling. If I recall correctly I think they asked if we could help them upstream BSD code for NVMe and NUMA. I think in that meeting there was even some L9 or super high up NUMA CPU guy from Hillsborough they some how convinced to join. The conversation and technical discussion was quite fun, but it was sort of funny to us at the time they were having to do all this work on the BSD kernel that was solved years ago for linux. Technical debt I guess.
- cperciva 4y agoNetflix tried Linux. FreeBSD worked better.
- dboreham 4y agoBy some definition of better.
- trasz 4y agoIt worked faster. It's a common misconception among newbies that "Linux has NUMA" automatically means it will use NUMA properly in a given workload. What it actually means is you _should_ be able to use existing functionality. Sometimes you'll only need to configure it, sometimes you'll need to reimplement it from the scratch, and doing that in FreeBSD is easier because there's less bloat.
- throw0101c 4y ago*At the time when they created the OCA project. If someone was going to do a similar comparison now the results could be different.
- 4y ago
- la64710 4y agoGreat engineering but how does this 800Gb/s throughput achieved translate downstream all the way to the consumers? I suspect there may be switches and routers from ISPs and others that Netflix do not control in between that will reduce the effective throughout to the end user.
- Cyph0n 4y agoOn the contrary, even older routers can handle this load with no sweat. Service provider-grade routers can handle 10 to 200 Tbps depending on size.
- Aeolun 4y agoBut then it gets to my home and it’s trashed down to 100Mbit/s
- kkielhofner 4y agoISP routers have been more-or-less indistinguishable from switches for decades at this point. They're all "line rate" which is to say that regardless of features, packet size, etc they'll push traffic between interfaces at whatever the physical link is capable of without breaking a sweat. In the case of Netflix it is in the ISPs best interest to let them push as much traffic to their customer eyeballs as possible. After all, it's much "easier" and cheaper to build out your internal fabric and network (which you have to do anyway for the traffic) than it is to buy and/or acquire transit to "the internet" for this level of traffic.
- jwmoz 4y agoI've wondered how they achieve it and it's so far beyond my knowledge and skills, truly astounding. The level of expertise and costs must be so high.
- Aeolun 4y agoSpend a few years just thinking of how to optimize video delivery and you’d be a lot closer to understanding :)
- yrgulation 4y agoThis is innovation and proper engineering. They choose freebsd. Shows they are not afraid of solving actual hard problems that yield impressive results. These are the types of engineers i’d hire in a heart beat - if i ever was to own a successful company. Simply following trends and doing what everyone else does leads to mediocre results and the assembly line type of work that most software development has become.
- mrbonner 4y agoMy work requires to deal with political crap day long to get promoted to a staff role. I miss this kind of work.
- ryanianian 4y ago> deal with political crap day long to get promoted to a staff role That's largely what many staff+ engineers have to do, even in otherwise healthy organizations. "Staff" isn't a glorified, autonomous, and stress-free version of senior at most companies. There's nothing wrong with staying at the senior level indefinitely provided (1) the pay and other factors are keeping up with your contributions and (2) the staff+ and management folks are being effective umbrellas for the politics and messy uninteresting details behind interesting problems like this.
- drewg123 4y agoAuthor here. AMA
- ManWith2Plans 4y agoDo you have a link to video or audio for this presentation? I'm probably don't speak for just myself when I say I would love to see it.
- quux 4y agoSomeone else linked the video here: https://news.ycombinator.com/item?id=32520750 https://news.ycombinator.com/item?id=32520750
- Bluecobra 4y agoWhat kind of tuning is done in the BIOS? Is that profile available to view to everyone? Are you using a custom BIOS from Dell?
- drewg123 4y agoNot much tuning needed to be done. The little that was is mentioned in the talk, and was basically to set NPS=1 and to disable DLWM, in order to be able to access the full xgmi interconnect bandwidth at all times, even when the CPU is not heavily loaded.
- throw0101c 4y agoYou're using ConnectX-6 Dx here. Any technical reason for that particular NIC, or just haven't gotten around to ConnectX-7s yet? Have you examined other NIC vendors? (Chelsio?)
- drewg123 4y agoThis talk was given roughly 4 months ago. CX7 was not available yet. I'm looking forward to testing on them when we get some. We looked at Chelsio (as T6 was available well before CX6-DX). However, the CX6-DX offers a killer feature not available on T6. The CX6-DX can remember the crypto state of any in-order stream, while the T6 cannot. That means that the TCP stack can send, say, 4K of a TLS record, wait for acks, and come back 40ms later and send the next 4K and DMA just the requested 4K from the host. The T6 cannot remember the state, and would need to DMA the first 4K (which was already sent) in order to re-establish the crypto state, and then DMA the requested 4K. This could run the PCIe bus out of bandwidth. The alternative is to make TCP always chunk sends at the TLS record size, but this was horrible for streaming quality.
- daper 4y agoI have some experience serving static content and working with CDNs. Here is what I find interesting / unique here: - They are not using OS page cache or any memory caching for that, every request is served directly from disks. This seems possible only when requests are spread between may NVMe disks since single high-end NVMe like Micron 9300 PRO has max 3.5GB/s read speed (or 28Gbps) - far less than 800Gbps. Looks like it works ok for long-tail content but what about new hot content everybody wants to watch at the day of release? Do they spread the same content over multiple disks for this purpose? - Async I/O resolves issues with nginx process stalling because of disk read operation but only after you've already opened the file. Depending on FS / number of files / other FS activities, directory structure opening the file can block for significant time and there is no async open() AFAIK. How they resolve that? Are we assuming i-node cache contains all i-nodes and open() time is insignificant? Or are they configuring nginx() with large open file cache? - TLS for streamed media was necessary because browsers started to complain about non-TLS content. But that makes things sooo complicated as we see in the presentation (kTLS is 50% of CPU usage before moving to encryption offloaded by NIC). One has to remember that the content is most probably already encrypted (DRM), we just add another layer of encryption / authentication. TLS for media segments make so little sens IMO. - When you relay on encryption or TCP offloading by NIC you are stuck with that is possible with your NIC. I guess no HTTP/3 over UDP or fancy congestion control optimization in TCP until the vendor somehow implements it in the hardware.
- eru 4y agoDoes the encryption in DRM protect the metadata?
- daper 4y agoAFAIK no. The point of DRM is to prevent recording / playing the media on a device without decryption key (authorization). So the goal is different than TLS that is used by the client to ensure the content is authentic, unaltered during transmission and not readable by a man-in-the-middle. But do we really need such protection for a TV show? "Metadata" in HLS / DASH is a separate HTTP request which can be served over HTTPS if you wish. Then it can refer to media segments served over HTTP (unless your browser / client doesn't like "mixed content").
- _gabe_ 4y agoThis is incredible. I really like how you're able to trace the evolution of the systems as well. It makes me wonder what the next hardware revolution will be. It seems like most resource intense applications are bottlenecking at transferring memory. UE5's nanite tech hinges on the ability to transfer memory directly from disk to GPU, Netflix built specific hardware to avoid copying memory between userspace and hardware, and I wonder how much other performance we're missing out on because we can't transfer memory fast enough. How much faster could AI training be if we could get memory directly from disk to the GPU and avoid the CPU orchestrating it all? What about video streaming? I have a feeling these processes already use some clever tricks to avoid unnecessary trips through the CPU, but it will be interesting to see which direction hardware goes with this in mind.
- aftbit 4y agoThis is definitely the direction that things are going. In the GPU space, see things like GPUDirect[1]. In networking and storage, especially for hyperscale stuff, see the rise of DPUs[2] replacing CPUs. 1: https://developer.nvidia.com/gpudirect https://developer.nvidia.com/gpudirect 2: https://www.servethehome.com/what-is-a-dpu-a-data-processing-unit-quick-primer/ https://www.servethehome.com/what-is-a-dpu-a-data-processing...
- throw0101c 4y agoPrevious discussions on 400Gb/s: * https://papers.freebsd.org/2021/eurobsdcon/gallatin-netflix-freebsd-400gbps/ https://papers.freebsd.org/2021/eurobsdcon/gallatin-netflix-... * https://news.ycombinator.com/item?id=28584738 https://news.ycombinator.com/item?id=28584738
- 0x500x79 4y agoAwesome feats of engineering here taking hardware and software into account when designing the system for a holistic approach to serving content as quickly as possible! The slide deck background though: At least half of the products in the slide deck template are no longer on Netflix...
- Joel_Mckay 4y agoHow would this compare with 42 server slots running 100Gbps DRBD in RAID 0? If I recall, it can pre-shard the data based on a round-robin balancer. ;)
- drewg123 4y agoWe don't consider solutions like DRBD that introduce inter-dependencies between servers. Any CDN server has to be able to fail and not take down our service.
- Joel_Mckay 4y agoMost CDNs just use DNS to route user traffic, and can redirect if latency spikes even at ISP NOCs (the CDN traffic controller often doesn't care why a given service is under performing). Have a wonderful day. =)
- ajross 4y agoSo... the driver and device level seems happy here, but is anyone else creeped out by "asynchronous sendfile()"? I mean, how do you even specify that? You have a giant file you want dumped down the pipe, so you call it, and... just walk away? How do you report errors? What happens to all the data buffered if the other side resets the connection? What happens if the connection just stalls? In synchronous IO paradigms, this is all managed by the application with whatever logic the app author wants to implement. You can report the errors, ping monitoring, whatever. But with this async thing, what's the API for that? Do you have to write kernel code that lives above the driver to implement devops logic? How would one even document that? +1 for the technical wizardry, but seems like it's going to be a long road from here to an "OS" feature that can be documented.
- drewg123 4y agoThis has been a feature upstream in FreeBSD for roughly 6 years. If there is a connection RST, then the data buffered in the kernel is released (either freed immediately, or put into the page cache, depending on SF_NOCACHE). sendfile_iodone() is called for completion. If there is no error, it marks the mbufs on the socket buffer holding the pages that were recently brought in as ready, and pokes the TCP stack to send them. If there was an error, it calls TCP's pru_abort() function to tear down the connection and release what's sitting on the socket buffer. See https://github.com/freebsd/freebsd-src/blob/main/sys/kern/kern_sendfile.c#L289 https://github.com/freebsd/freebsd-src/blob/main/sys/kern/ke...
- zackmorris 4y agoThis was my thought too. I've been struggling with the concept of "if you don't have anything nice to say, don't say anything at all" lately, because I've been programming too long and just see poison pills and better alternatives everywhere I look. But I believe that async is an anti-pattern. From the article: * When an nginx worker is blocked, it cannot service other requests * Solutions to prevent nginx from blocking like aio or thread pools scale poorly Nothing against nginx (I use it all the time, it's great) but I probably would have used a synchronous blocking approach. The bottleneck there would be artificial limits on stuff like I/O and the number of available sockets or processes. So.. why isn't anyone addressing these contrived limits of sync blocking I/O at a fundamental level? We pretend that context switching overhead is real, but it's not. It's an artifact of poorly written kernels from 30+ years ago (especially in Windows) where too many registers and too much thread state must be saved while swapping threads. We're basically all working around the fact that the big players have traditionally dragged their feet on refactoring that latency. And that some of the more performant approaches like atomic operations using compare and swap (CAS) on thread-safe queues beat locks/mutexes/semaphores. And that content-addressable memory with multiple busses or even network storage beats vertical scaling optimizations. So I dunno, once again this feels like kind of a drink-the-kool-aid article. If we had a better sync blocking foundation, then a simple blocking shell script could serve video and this whole PDF basically goes away. Rinse, repeat with most web programming too, where miles-long async code becomes a single deterministic blocking function that anyone can understand. I'm kind of reaching the point where I expect more from big companies to fix the actual root causes that force these async workarounds. I kind of gave up on stuff like that over the last 10 years, so am behind the times on improvements to sync blocking kernel code. I'd love to hear if anyone knows of an OS that excels at that.
- phantomathkg 4y ago> Serve only static media files This part is weird to me. My understanding is DRM lock the file at a per user level, so the DRM encrypted chunk would be different from yours. Unless for all bitrate, and for all streaming format, Netflix has already pre-computed everything. Otherwise, there must be some sort of pre-computation before it can be served over TLS.
- wmf 4y agoThat's not how DRM works. The content is encrypted once and that key is sent to the client. The content key is probably wrapped in some per-session key (which may be wrapped in a per-user key wrapped in a per-device key or something).
- csmpltn 4y agoAt this point, they should've just gone for an in-house bare-bones operating system that supports the bare minimum: reading chunks from disk, encrypting them, and forwarding them to the NIC. Besides that, it sees like all of the heavy lifting here is done by Mellanox hardware...
- wmf 4y agoFreeBSD is their "in-house" operating system since they modify it to do what they want.
- csmpltn 4y agoBut do they really need an entire operating system for what amounts to simply copying around chunks of data? I think they could've gone for some slim RTOS-ish solution instead: no user-mode, no drivers, bare minimum.
- wmf 4y agoThey're using the FreeBSD filesystem and network stack, both of which are significant amounts of code. I guess they could have tried the rump kernel concept but it sounds like a lot of work.
- drewg123 4y agoI worked on an OS like that once. The problem is with "all the other stuff" that you need to support that's outside the core mission of your OS. You wind up bogged down on each additional feature that you need to implement from scratch (or port from another OS with a compatible license). With FreeBSD, all this comes for free. We chose to use FreeBSD, and have contributed our code back to the FreeBSD upstream to make the world a better place for everyone.
- drewg123 4y agoIts only doing the crypto. The VM system and the TCP stack are doing most of the heavy lifting, and are both stock FreeBSD.
- skynetv2 4y agoThis is amazing work. I cant help but state that we have been doing these in HPC environments for at least 15 years - User space networking, offloads, NUMA domains aware scheduling, jitter reduction ... great to see it being put to good use in more mainstream workloads. Goes to show - software is eating the world.
- drewg123 4y agoI worked in HPC as well, and I have to point out emphatically that this IS NOT USERSPACE NETWORKING. The TCP stack resides in the kernel. The vast majority of our CPU time is system + interrupt time.
- haunter 4y agoAs a total outsider it looks like FreeBSD is the “silent” OS behind lot of the big money projects. Not just this but recently learned it’s the base OS of the Playstation 4 and 5 system too. Is there a reason why FreeBSD is so popular? Just general reliability? And why not the other BSD projects? Also one, like me, would assume Linux is behind all of these but alas not.
- Thev00d00 4y agoSony used it because they got an entire OS for free with no obligation to release the source.
- keewee7 4y agoThe sysadmin experience on FreeBSD used to be more opinionated than on Linux. This was before most Linux distros adopted systemd. The reason companies like Sony and Apple pick FreeBSD is because they get an open source POSIX-compliant OS they can drastically modify down to the kernel level without having to open source their modifications.
- naikrovek 4y agoBlame the GPL for this. The GPL is directly responsible for the livelihood of many/all BSD developers, and i could not be happier about that. Linux is overrated in a lot of ways.
- robocat 4y agoHowever the GPL is irrelevant in this case because the NetFlix Open Connect appliance is not sold to ISPs. The GPL is only relevant if you are distributing GPL software e.g. Sony PlayStation.
- naikrovek 4y agoNot directly relevant perhaps. I would not go as far as to say it is necessarily completely irrelevant; there are many ways usage of FreeBSD in one portion of a company, or by employees hired from another company, can affect decisions made by the company. Who knows what plans Netflix had in mind when this decision was made? Maybe they simply commit to FreeBSD because they do not want to release their code under the GPL. I certainly never do, and I will never ever make any improvements to any GPL code I use. I think it is an awful license. Maybe Netflix does, too? That would be directly relevant to the license of the OS even if they never release their customized OS, if they have a customized version at all.
- pclmulqdq 4y agoThis is amazing work from the Netflix team. I'm looking forward to 1.6 Tb/s in 4 years. It is interesting that this work is happening on FreeBSD, and potentially with diverging implementations than Linux. Linux programs seem to be moving towards userspace getting more power, with things like io_uring and increasing use of frameworks like DPDK/SPDK. This work is all about getting userspace out of the way, with things like async sendfile and kernel TLS. That's pretty neat!
- the8472 4y agokTLS has been added to linux too including offload. It also has p2p-dma, so in principle you can shovel the file directly from NVMe to the NIC and have the NIC encrypt it, so it'll never touch the CPU or main memory. But that only works on specific hardware.
- deleted 4y ago[deleted]
- robocat 4y agoMemory is the cache for popular content. You couldn’t serve fast enough directly from NVMe. “~200GB/sec of memory bandwidth is needed to serve 800Gb/s” and “16x Intel Gen4 x4 14TB NVME”. So each NVMe drive would need to serve 12.5GB/s which is more than the 8GB/s limit for PCIe 4.0 x4. Also popular content would need to be on every drive, drastically lowering the total content stored. Also see drewg’s comment on this for a different reason: https://news.ycombinator.com/item?id=32523509 https://news.ycombinator.com/item?id=32523509
- pclmulqdq 4y agoWith HBM2 sapphire rapids chips, I assume you can actually get there. There is probably an insane price premium for them, though, so I wouldn't hold my breath.
- mgerdts 4y agoPCIe Gen 5 drives look poised for wide availability next year and NVIDIA has been demoing CX7 [1] which is also PCIe Gen 5. Intel already has some Gen 5 chips and AMD looks like they will follow soon [2]. Surely there will be other bumps, but I bet they pull it off in way less than 4 years. 1. https://www.servethehome.com/nvidia-connectx-7-shown-at-isc-2022-supermicro/ https://www.servethehome.com/nvidia-connectx-7-shown-at-isc-... 2. https://wccftech.com/amd-epyc-7004-genoa-32-zen-4-core-cpu-sample-spotted-more-cache-4-6-ghz-clocks/ https://wccftech.com/amd-epyc-7004-genoa-32-zen-4-core-cpu-s...
- RektBoy 4y ago
- faizshah 4y agoI have a bit of a naive question. If TLS has this much overhead how do HFT and other finance firms secure their connections? I know they use a number of techniques like kernel bypass to get the lowest latency possible, but maybe they have explored some solution to this problem as well.
- shiftpgdn 4y agoMellanox cards or private links
- wyager 4y agoTLS doesn't really add latency on top of TCP after you make the initial connection - it mostly adds a bit of extra processing overhead for encryption. HFT firms aren't usually encryption-bandwidth-constrained. I'm not actually sure if most exchange FIX connections or whatever actually run over TLS, but that would be reasonable.
- drewg123 4y agoTLS has the highest overhead when you're serving data at rest, like static files that are not already in the CPU cache. For serving dynamic data that is in the CPU cache, TLS offload matters a lot less. Our workload is basically the showcase for TLS offload.
- naikrovek 4y agoi love (love) how everyone else who answered this question alongside you made what appears to be a complete stab in the dark guess while only you knew the answer. never be afraid to admit that you don’t know something. guessing wrong is a much worse look than not answering at all.
- AtlasBarfed 4y agoHFT needs to be outlawed. No exchange should allow trades to complete, I would argue in any time less than 15 minutes, and each trade should have a random 1-15 minute delay pad on top of that. The HFT access only serve the larger financial firms, and are used to do frontloading and other basically-illegal tricks. It provides anti-competitive advantage to large firms for markets that are supposed to open access/fair trading. And of course it leads to AI autotrading madness. I get that it keeps a lot of tech people very well compensated, but it is either in the service of unregulated fraud at worst and unfair advantage at best.
- trunnell 4y agoThe OpenConnect team at Netflix is truly amazing and lots of fun to work with. My team at Netflix partnered closely with them for many years. Incidentally, I saw some of their job posts yesterday. If you think this presentation was cool, and you want to work with some competent yet humble colleagues, check these out: CDN Site Reliability Engineer https://jobs.netflix.com/jobs/223403454 https://jobs.netflix.com/jobs/223403454 Senior Software Engineer - Low Latency Transport Design https://jobs.netflix.com/jobs/196504134 https://jobs.netflix.com/jobs/196504134 The client side team is hiring, too! (This is my old team.) Again, it's full of amazing people, fascinating problems, and huge impact: Senior Software Engineer, Streaming Algorithms https://jobs.netflix.com/jobs/224538050 https://jobs.netflix.com/jobs/224538050 That last job post has a link to another very deep-dive tech talk showing the client side perspective.
- totony 4y agoI see netflix has an office in Toronto, but the jobs are "remote, US". Any idea if remote is also an option for canadians?
- NikhilVerma 4y agoI am curious why they manually split the video to compress individual clips with different bit rates. Encoders usually have a variable-bit-rate option that does the same?
- trunnell 4y agoThat's the essence of adaptive streaming - the player continuously selects a video bitrate that can be downloaded quickly enough given the network conditions. That requires multiple video bitrates being available, and each one must be encoded such that the player can switch between them at known cut points.
- ksec 4y ago>Senior Software Engineer - Low Latency Transport Design I am not a Netflix subscriber but I dont think Netflix does live streaming for anything much if at all. The Ultra Low Latency seems to suggest Netflix is exploring this idea. Which could be Live Sport or some other shows.
- gmm1990 4y agoWhy use/make processors with numa if you have to go to all this trouble not to use it?
- 0x457 4y agoWell, the point of NUMA is to allow you to do things like in slides rather than - everyone suffers equally talking to North Bridge. Fabric between NUMA nodes isn't the selling point - fast and direct connection between CPU and other components is. Plus, not every workload is: read from disk -> encrypt -> send to nic
- jalino23 4y agoI was specifically looking for whats their tech stack for playback? they pretty much have to use HLS for ios safari right? where do those manifest server fit in? what about non ios browser playback?
- alpb 4y agoWhat's the benefit of going from 100Gb/s to 800Gb/s through kernel/hardware optimizations as opposed to adding more machines to meet the same throughput in this case? I'd be curious at what point returns on the engineering effort is diminishing in this problem.
- seabrookmx 4y agoIIRC a lot of these boxes are deployed at actual ISP's so they're closer to customers. I'd imagine the rackspace is therefore limited and the more you can push from a single machine, the better.
- quotehelp1829 4y agoI think it's quite obvious that instead of 8 machines you then only need 1. This results in reduced costs for machinery, storage (as each machine would have its own storage) and probably power consumption too. Also, same room of servers can throughput 8 times more content. Edit: Whoops, apparently this tab has been open for four hours and of course someone already had responded to you, lol.
- qwertox 4y agoDiscussion of the same presentation 11 months ago, when the title was 400GB/s. https://news.ycombinator.com/item?id=28584738 https://news.ycombinator.com/item?id=28584738 This was the video which was posted back then alongside the slides: https://www.youtube.com/watch?v=_o-HcG8QxPc https://www.youtube.com/watch?v=_o-HcG8QxPc
- ndom91 4y agoVideo of this presentation available here: https://cdnapisec.kaltura.com/index.php/extwidget/preview/partner_id/2539171/uiconf_id/44037791/entry_id/1_q714dkb3/embed/dynamic https://cdnapisec.kaltura.com/index.php/extwidget/preview/pa...
- forgot_old_user 4y agothank you!
- touisteur 4y agoWill be fun to see what can be done with pcie5 stuff and new 400g NICs. Really amazed by the recent increase in bandwidth. Sfp56 recently becoming 'mainstream' in datacenters with 200G controlers at <1500 each, you can stuff 8 or 10 of those in your server. And you get immediate x2 with next gen. If you can offload some of the heavy work to one (or several) GPUs or these FPGA accelerator boards (Alveo or more niche but also crazy ReflexCES with eth-800G capability) you're really starting to get a 'datacenter in a box' system. If compacity is important, the next years are going to be very interesting.
- wly_cdgr 4y ago
- lprd 4y agoI wonder if they are using something like truenas or just interfacing directly with OpenZFS (assuming they use ZFS).
- drewg123 4y agoWe use UFS for content. The ZFS caching layer (ARC) does not integrate with the kernel's page cache, and so sendfile() requires a copy to send data residing on ZFS. I believe that ZFS also does not implement the async methods required for the async portion of sendfile.
- BonoboIO 4y agoIt amazes me, that Netflix is capable of such top of the line engineering things, but is for the love of god unable to stream HD Content to my iPhone. Tried everything gigabit wifi, cellular ... It is better for me to pirate their content, play it with plex and be happy. I pay for Netflix and it is absurd. I think the best years are over for Netflix. The hard awakening is here to make content that the users want and they are a movie/tv content company, not primarily a „tech company“.
- Infinitesimus 4y agoSome ISPs throttle Netflix. Not sure your background but it might be helpful to have more details about the type phone (I'd expect a difference in 13 pro max vs an old 7) and ISP to see if others have similar problems.
- BonoboIO 4y agoiPhone XS iOS 15.4 Netflix App Updated A1 Telekom both mobile and wifi. It does not even work at work with a different ISP. Netflix has all the bandwidth data and metrics, but this is not working since ages. Maybe a more basic setup on their end would bring better results. Focus more and delivery, not 10 different UI versions, AB Tests, Batch Job Workflows and so on. They Post on their engineering blog how they test multiple TVs, multiple Profiles in encoding, great things, but if the basics don’t work ... well what is it good for. I think they lost their focus.
- Mindwipe 4y agoI think it is more likely there is a specific issue with your device or connectivity.
- BonoboIO 4y agoSupport is not helpful, so ... I don’t know anything to do else. I m not angry, but maybe I m not the only one and sooner or later they have to focus on delivering a good experience. Even Prime delivers a better experience, and they know nothing about video :D Their interface is so basic, but to their credit ... the quality is there.
- a-dub 4y agoi haven't looked yet but i'm going to guess: edge caching running on custom hardware with smart predictions and congestion control algorithms for determining what gets cached where and when.
- bagels 4y agoSlide says "we don’t transcode on the server" Surely they transcode on some server? Maybe they just mean they don't do it on the same server that is serving bits to customers?
- naikrovek 4y agoIt seemed clear to me: they don't transcode on the server that is sending data to the viewer. Transcoding is done once per piece of media and target format combination, instead of on the fly as it is viewed.
- walrus01 4y agoEverything old is new again: Anyone remember seeing a 32-bit/33 MHz PCI (not pci-x, not pci-e) card for SSL acceleration in the late 1990s? It was totally a thing at one point in time when your typical 1U rackmount server was single-core CPUs and quite weak in overall math processing power. OpenBSD had support for them like 22 years ago. https://www.google.com/search?client=firefox-b-d&q=SSL+accelerator+pci+card+openbsd https://www.google.com/search?client=firefox-b-d&q=SSL+accel... Now we have TLS1.2/TLS1.3 offload getting built into the PCI-E 4.0 100/200/400GbE (whatever speed) NIC.
- drewg123 4y agoThe cards you reference are not really the same thing. They are doing lookaside acceleration, where the card has to transfer the data from host memory, encrypt it, and transfer it back. The Mellanox (and Chelsio) NICs do in-line crypto. They DMA down the network packets as they would have to do anyway in order to send them. Then they encrypt them on their way out onto the wire. This reduces memory bandwidth requirements by roughly 50%.
- zoltar 4y agoIsn't this a $50k-$100k box? Feels like this isn't the cheapest way to do this by an order of magnitude. What am I missing?
- detaro 4y agoSo how would you serve that amount of traffic for an order of magnitude less? I'm not too deep into this stuff, but it doesn't seem that far out?
- zoltar 4y agoI guess you simply can't in the density required where this would go. It was just a knee-jerk reaction to the napkin-math for the cost. I've never wanted to build a single box pulling that much traffic when 20 would do.
- mannyv 4y agoFunny, I thought Netflix was all AWS. NM, this is for the hosted solution.
- ksec 4y agoHere is an interesting thought experiment. With PCI-7.0 coming in 2025/2026, product launch by 2028. And assuming no further bottleneck, we assume even a linear increase of memory bandwidth would be handled with HBM3 or HBM4. We should be able to hit 6.4Tbps by 2030. In a single 1U unit. With next generation codec VVC, or even a generation beyond that. Serving an average 10Mbps files, a single box is capable of serving 640K customers. A single Rack could serve 25 Million Customer. i.e 10 Rack would be able to serve all of the current Netflix subscribers world wide, simultaneously. Although I would assume the CPU being the bottleneck with 8x the required processing power.
- ksec 4y agoThis got me thinking, at what point would the benefits of these optimisation are no longer meaningful. At 225Million subscriber, assuming somehow, it has a peak concurrent user of 70% total, around 160M. This figure should already be way above the actual number because they are spread across the Globe in different Time Zone. At 10Mbps Netflix would need a peak bandwidth of 1600M Mbps, or 1600Tbps. At 1.6Tbps that is only 1000 box required worldwide. Or 2000 if you include redundancy. Once you factor in you need a box with every ISP, Local Peer Exchange or Regional ISP worldwide, and there are hundreds of those. Shrinking it further wouldn't make much of a different. Hopefully that means Netflix now has the incentive to bump bitrate to 20Mbps or higher.
- shukantpal 4y agoI’m wondering if for these extreme use cases a bare metal program that doesn’t sit on top of a OS / kernel would be substantially more efficient.