6 ms·
Just some napkin maths. ( Correct me if I am wrong ) Looking at the 800Gbps Config, Dell R7525 with Dual 64C / 128T and 4x Connect-DX 800Gbps in 2U. With Zen
by ksec 4y ago
Just some napkin maths. ( Correct me if I am wrong )
Looking at the 800Gbps Config, Dell R7525 with Dual 64C / 128T and 4x Connect-DX 800Gbps in 2U.
With Zen 4C, 128C and PCI-E 5.0, Connect-7, two node could fit into 2U. i.e doubling to 1.6Tbps per 2U.
That is going from 16Tbps to 32Tbps per Rack. ( Using 40U only )
To things in perspective, if every user were to use 20Mbps Stream at the same time, ( not going to happen due to time zone difference ), the 250M Netflix subscribers worldwide would need 5000M Mbps or 5000 Tbps. That is less than 200 Racks to serve every single of their customer on planet earth. ( Ignoring Storage. ) You could ship a Rack to every Region, State, Nation, Jurisdiction or Local ISP and Exchange and be done with it.
I hope Lisa Su sent drewg123 and his team at Netflix with Zen 4C ASAP to play, cough, I mean help them test it.
Note: We have PCI-E 6.0 ( and 7.0 ), DDR6 on Roadmap. The 200 Racks could be down to 50 Racks by the end of this decade. Assuming Netflix is still streaming at the same bitrate.
- alberth 4y ago> That is going from 16Tbps to 32Tbps per Rack ... only need 200 racks I doubt ISP's give an entire rack to Netflix. I wouldn't be surprised if they only get like 4U total (hence why throughput per server is so important to Netflix).
- BonoboIO 4y agoThe minimum requirements https://openconnect.zendesk.com/hc/en-us/articles/360034538352-Requirements-for-deploying-embedded-appliances https://openconnect.zendesk.com/hc/en-us/articles/3600345383... I think it depends on the size of isp, probably a rack would be too much even for the biggest isps, but one 4u too less.
- TFortunato 4y agoLooking at the banner pic on their main page, they seem to have at least one ISP install of multiple racks in the wild. Also, doing a little reading on how "fill" of the devices works, they talk about doing peer-to-peer filling of appliances located at the same site, which leads me to believe, even if not deploying a full rack, deploying multiple appliances to an ISP site is a relatively normal occurance https://openconnect.netflix.com/en/peering/ https://openconnect.netflix.com/en/peering/
- meltedcapacitor 4y agoWhy not? It's top bandwidth consumer for a retail ISP and surely any reasonable amount of rack space is worth the savings in interconnect bandwidth.
- loeg 4y agoThey are frequently rack space constrained, hence these super dense hardware.
- jedberg 4y agoSome ISPs give a full rack, some don't. It depends on how much traffic they have and how willing they are. But a lot of the racks sit at internet exchange points, where Netflix rents one or more racks at a time.
- reaperducer 4y agoThat is less than 200 Racks to serve every single of their customer on planet earth. ( Ignoring Storage. ) If you're going to ignore storage, Netflix could just ship a low-end video server to every one of its customers and be done with it. Every problem is an easy problem if your pretend the hard parts don't exist.
- OliverGuy 4y agoHow much storage does Netflix actually need for its whole library? It's got about 17,000 titles globally [1]. If they have copies in SD, 720p, HD and 4k that would be 68,000 versions (plus some extra audio tracks for stuff dubbed in multiple languages, but I suspect this is fairly minimal in terms of storage though) Let's assume that the resolutions have the bitrates at 5, 10, 15 and 20 mbps. The average length of a Netflix original movie is ~90mins [2] So that would require about 575TB in storage if I have done my maths correctly. You would need about 20x30TB Kioxia CD6 SSDs for all that. Very expensive but definitely technically possible. I could totally see it being possible to fit those drives in a single node to push the 800gbps required, not increasing the over rack requirement at all. Not sure if the bandwidth from that many drives is enough, might have to cache some of the most watched stuff to ram) Not gonna see any in home boxes with all the titles pre loaded any time soon though. As a hard drive array that's still 30x20TB drives. [1] https://www.comparitech.com/blog/vpn-privacy/netflix-statistics-facts-figures/#:~:text=Netflix%20has%20over%2017%2C000%20titles,libraries%20as%20of%20October%202022 https://www.comparitech.com/blog/vpn-privacy/netflix-statist... [2] https://stephenfollows.com/netflix-original-movies-shows/#:~:text=The%20average%20Netflix%20Original%20fiction,two%20shorter%20films%20mentioned%20above https://stephenfollows.com/netflix-original-movies-shows/#:~...)
- AdrianB1 4y agoDo they keep on every server the global library? I guess they partition it geographically.
- tecleandor 4y agoIn their OpenConnect network they keep the most demanded titles and the latest releases. And IIRC that refreshes nightly (with new releases and whatever is hot that day) https://openconnect.zendesk.com/hc/en-us/articles/360035618071 https://openconnect.zendesk.com/hc/en-us/articles/3600356180...
- loeg 4y agoNetflix is more likely to use a single box of this kind of throughout at any given POP than a rack of them. For bigger installations they can use cheaper, less throughput-dense hardware (although I don’t know if they do).
- lillecarl 4y agoTake a look at the hardware, it isn't particularly expensive stuff.
- loeg 4y agoAside from GPUs, I'm not sure how you would increase the cost density much. Those NICs doing hundreds of Gbps and TLS aren't cheap, nor are the fast SSDs needed to sustain the load, nor is RAM or top end AMD server CPUs. Of course, the cost is absolutely worth it to Netflix!
- lillecarl 4y agoYes, but it's still just one box, if you're building a cluster of cheaper characteristics you need more of everything. A high-end server VS a cluster of 10 machines, 10 machines wouldn't be cheaper to get to the same throughput, it's not alien specialized supertech, it's just top of the line commodity hardware. (10 is just an example number here).
- loeg 4y agoI mean, I guess I disagree with your stipulation that you couldn't lower total costs somewhat using slightly more slightly lower end hardware, if rack space was cheap. > top of the line commodity hardware Yeah -- cost in commodity hardware scales super-linearly with performance.
- lillecarl 4y agoOn one box you don't have to buy a 15000$ (or two) network switch(es), that's significant, you don't have to pay for N chassis, motherboards, nics, NVMe. Considering they're really eeking every last bit of juice out of the box, I'm doubtful that distributing it would be cheaper. Also they're not maxxing the CPU out, they just need memory bandwidth and the Mellanox nics. Storage would be more expensive on more boxes since they can't distribute the storage, they have to use local storage to reach the performance we're after.
- aeyes 4y agoYou are probably overestimating Netflix traffic by a lot. IX.br peak traffic is 20Tb/s, DE-CIX peak traffic is 14Tb/s, AMS-IX is around 11Tb/s. The 800Gpbs machine is probably enough for a country. Netflix traffic stats at PIT Chile, this is their only peering connection in Chile: https://www.pitchile.cl/wp/graficos-con-indicadores/streaming-1-ocupacion-de-enlace/ https://www.pitchile.cl/wp/graficos-con-indicadores/streamin...
- lostlogin 4y agoI notice people streaming in very low resolution without realising it, and sometimes intervene when the pain gets too great. I’d be vey surprised if the average bitrate was anywhere near the appropriation. However that wasn’t the point of the calculation, it was looking for a maximum.
- orangepurple 4y agoAgree and 20 mbps is a reasonable rate for modern codecs for resolutions up to 4k for the 99% of viewers
- KyleSanderson 4y agoYou still need the premium package for this. It maxes at 1080p with no HDR on the standard package.
- lostlogin 4y agoNetflix 4K is pretty low bitrate. I haven’t checked recently but was not seeing anything approaching 20mbps last time I looked. That said, it looked pretty good.
- srmn 4y agoThis assumption misses out on all the private interconnect links and deployed OpenConnect appliances within ISP networks - a majority of Netflix's traffic today. IXes are only a small portion of overall internet traffic.
- recuter 4y agoThere is the rather intriguing prospect of NVM Express over Fabrics (NVMe-oF): https://en.wikipedia.org/wiki/NVM_Express#NVMe-oF https://en.wikipedia.org/wiki/NVM_Express#NVMe-oF Marvel Octeon 10 DPU (with an integrated 1 Terabit switch): https://www.marvell.com/content/dam/marvell/en/company/media-kit/octeon-10/marvell-octeon-10-media-deck.pdf https://www.marvell.com/content/dam/marvell/en/company/media... Probably pretty soon you'll be able to chuck in a few hot swappable 100 TB Nimbus exadrives (https://nimbusdata.com/products/exadrive/ https://nimbusdata.com/products/exadrive/) in there and call it a day. 1T in 1U. :)
- Melatonic 4y agoInteresting to see that Infiniband is still kicking
- coherentpony 4y agoNot really. Ethernet and Infiniband are both perfectly capable from a bandwidth perspective. Streaming video isn't remotely close to latency-bound, which is where Infiniband would be better suited.
- bombcar 4y agoStreaming video is about the perfect thing to send as you can cache it for days. If only it could start playing faster.
- virtuallynathan 4y agoBack of the Napkin Zen4 / Genoa gets you to ~500GB/s PCIe and ~500GB/s of DRAM bandwidth -- nearly 4Tbps! Zen3/Rome is ~300GB/s PCIe and ~300GB/s DRAM -- about 2.4Tbps. A single 2U box with Genoa might scale to 1.25Tbps+ of useful Netflix traffic. We'll have to see what magic Drew can pull :)
- Melatonic 4y agoThe people doing this might also be doing infra as code for the virtualization layer on the hardware itself - which this might not be able to satisfy. At minimum they surely have a ton of this stuff deployed already so changing hardware specs big time might not be worth the cost. Also are you taking into account encryption for those specs?
- Matthias247 4y ago> You could ship a Rack to every Region, State, Nation, Jurisdiction or Local ISP and Exchange and be done with it. And when a single rack is down the whole region might be impacted negatively. The blast radius of such a setup would be huge! As a CDN you really want to avoid this. Even for regular operations you need to be able to afford some servers going out of service for software update and other maintenance - without impacting availability, latency for users (the latter would happen if you route them to a complete different region), or infrastructure costs (which increase if you can't serve data from as close to the user as possible anymore, and have to pay for additional networking fees). All major CDN providers have hundreds of regions (all with multiple hosts), and you can't really avoid the former. You could run less hosts per region, but whether 1 is the feasible will depend a lot on other parts of your system.