7 ms·
Speed, scale and reliability: 25 years of Google datacenter networking evolution
- alex_young 2y agoLike most discussions of the last 25 years, this one starts 9 years ago. Good times.
- deleted 2y ago[deleted]
- eru 2y agoThe Further Resources section goes a bit further back.
- jerzmacow 2y agoWow and it doesn't open with a picture of their lego server? Wasn't that their first one, 25 years ago?
- teractiveodular 2y agoIt's a marketing piece, they don't particularly want to emphasize the hacky early days for an audience of Serious Enterprise Customers.
- deleted 2y ago[deleted]
- dangsux 2y ago[dead]
- DeathArrow 2y agoIt seems all cutting edge datacenters like x.ai Colossus are using Nvidia networking. Now Google is upgrading to Nvidia networking, too. Since Nvidia owns most of the Gpgpu products, they have top notch networking and interconnect, I wonder if they don't have a plan to own all datacenter hardware in the future. Maybe they plan to also release CPUs, motherboards, storage and whatever else is needed.
- timenova 2y agoThat was their plan with trying to buy ARM...
- Kab1r 2y agoGrace Hopper already includes Arm based CPUs (and reference motherboards)
- danpalmer 2y agoI read this slightly differently, that specific machine types with Nvidia GPU hardware also have Nvidia networking for tying together those GPUs. Google has its own TPUs and don’t really use GPUs except to sell them to end customers on cloud I think. So using Nvidia networking for Nvidia GPUs across many machines on cloud is really just a reflection of what external customers want to buy. Disclaimer, I work at Google but have no non public info about this.
- adrian_b 2y agoNvidia networking is what used to be called Mellanox networking, which was already dominant in datacenters.
- immibis 2y agoOnly within supercomputers (including the smaller GPU ones used to train AI). Normal data centers use Cisco or Juniper or similarly.well known Ethernet equipment, and they still do. The Mellanox/Nvidia Infiniband networks are specifically used for supercomputer-like clusters.
- huqedato 2y ago[flagged]
- encom 2y ago[flagged]
- cletus 2y agoThis mentions Jupiter generations, which I think is about 10-15 years old at this point. It doesn't really talk about what existed before so it's not really 25 years of history here. I want to say "Watchtower" was before Jupiter? but honestly it's been about a decade since I read anything about it. Google's DC networking is interesting because of how deeply integrated it is into the entire software stack. Click on some of the links and you'll see it mentions SDN (Software Defined Network). This is so Borg instances can talk to each other within the same service at high throughput and low latency. 8-10 years ago this was (IIRC) 40Gbps connections. It's probably 100Gbps now but that's just a guess. But the networking is also integrated into global services like traffic management to handle, say, DDoS attacks. Anyway, from reading this it doesn't sound like Google is abandoning their custom TPU silicon (ie it talks about the upcoming A3 Ultra and Trillium). So where does NVidia ConnectX fit in? AFAICT that's just the NIC they're plugging into Jupiter. That's probably what enables (or will enable) 100Gbps connections between servers. Yes, 100GbE optical NICs have existed for a long time. I would assume that NVidia produce better ones in terms of price, performance, size, power usage and/or heat produced. Disclaimer: Xoogler. I didn't work in networking though.
- virtuallynathan 2y agoThis latest revision of Jupiter is apparently 400G, as is the ConnectX-7, A3 Ultra will have 8 of them!
- neomantra 2y agoNvidia got ConnectX from their Mellanox acquisition -- they were experts in RMDA, particularly with Infiniband but eventually pushing Ethernet (RoCE). These NICs have hardware-acceleration of RDMA. Over the RDMA fabric, GPUs can communicate with each other without much CPU usage (the "GPU-to-GPU" mentioned in the article). [I know nothing about Jupiter, and little about RDMA in practice, but used ConnectX for VMA, its hardware-accelerated, kernel-bypass tech.]
- ceph_ 2y agoFrom memory: Firehose > Watchtower > WCC > SCC > Jupiter v1
- 2y ago
- maz1b 2y agoPretty crazy. Supporting 1.5mbps video calls for each human on earth? Did I read that right? Just goes to show how drastic and extraordinary levels of scale can be.
- sethammons 2y agoScale means different things to different people
- deleted 2y ago[deleted]
- 486sx33 2y agoThe most amazing surveillance machine ever …
- belter 2y agoAwesome Google... Now learn what an availability zone is and stop creating them with firewalls across the same data center. Oh and make your data centers smaller. Not so big they can be seen in Google Maps. Because otherwise, you will be unable to move those whale sized workloads to an alternative. https://youtu.be/mDNHK-SzXEM?t=564 https://youtu.be/mDNHK-SzXEM?t=564 https://news.ycombinator.com/item?id=35713001 https://news.ycombinator.com/item?id=35713001 "Unmasking Google Cloud: How to Determine if a Region Supports Physical Zone Separation" - https://cagataygurturk.medium.com/unmasking-google-cloud-how-to-determine-if-a-region-supports-physical-zone-separation-1a7bfbb3d280 https://cagataygurturk.medium.com/unmasking-google-cloud-how...
- tecleandor 2y agoMaking a datacenter not visible from Google Maps, at least on most big cities where Google zones are deployed, would mean making them smaller than a car. Or even smaller than a dishwasher. If I check London (where europe-west2 is kinda located) on Google Maps right now, I can easily discern manhole covers or people. If I check Jakarta (Asia-southeast2) things smaller than a car get confusing, but you can definitely see them.
- belter 2y agoYour comment does not address the essence of the point I was trying to make. If you have a monstrous data-center, instead of many smaller, in relative size, you are putting too many eggs on a giant basket.
- joshuamorton 2y agoWhat if you have dozens of big data centers?
- jiggawatts 2y agoTo reinforce your point: The scale of cloud data centres reflects the scale of their customer base, not the size of the basket for each individual customer. Larger data centres actually improve availability through several mechanisms: more power components such as generators means the failure of any one is just a few percent instead of a total blackout. You can also partition core infrastructure like routers and power rails into more fault domains and update domains. Some large clouds have two update domains and five fault domains on top of three zones that are more than 10km apart. You can’t beat ~30 individual partitions with your data centres at a reasonable cost!
- darkmx0z 2y ago[flagged]
- dangoodmanUT 2y agoDoes gcp have the worst networking for gpu training though?
- dweekly 2y agoFor TPU pods they use 3D torus topology with multi-terabit cross connects. For GPU, A3 Ultra instances offer "non-blocking 3.2 Tbps per server of GPU-to-GPU traffic over RoCE". Is that the worst for training? Namely: do superior solutions exist?
- ksec 2y agoThey managed to double from 6 Petabit per second in 2022 to 13 Pbps in 2023. I assume with ConnectX-8 this could be 26 Pbps in 2025/26. The ConnextX-8 is PCI-e 6 so I assume we could get 1.6Tbps ConnextX-9 with PCI-e 7.0 which is not far away. Cant wait to see the FreeBSD Netflix version of that post. This also goes back to how increasing throughput is relatively easy and has a very strong roadmap. While increasing storage is difficult. I notice YouTube has been serving higher bitrate video in recent years with H.264. Instead of storing yet another copy of video files in VP9 or AV1 unless they are 2K+.
- reaperducer 2y agoSpeed, scale and reliability Choose any two.
- teractiveodular 2y agoWhich of those is Google's network missing?
- nl 2y agoIn any decision making matrix you need a constraint that get consumed (economics, size, etc) to force a "choose any two" type situation. You absolutely can have speed, scale and reliability. You can't have speed, scale, reliability and low cost.
- deleted 2y ago[deleted]