5 ms·

Jupiter Rising: A Decade of Google’s Datacenter Network [pdf]

97 points by packetslave 11y ago

packetslave 11y ago
Originally discussed at https://news.ycombinator.com/item?id=9734305 https://news.ycombinator.com/item?id=9734305 but this is the actual paper, which was just published.
ariwilson 11y ago
Amazing network infrastructure has always been one of my favorite hidden technical wonders behind Google. It's amazing how they handle so much data.
- thrownaway2424 11y ago
  When you actually have to use it what's amazing is how much congestion and packet loss there is. On paper it looks like a zero-impedance source of data but in reality it barely keeps up.
  - raldi 11y ago
    What QoS are you referring to?
throwawayplanet 11y ago
I did a brief stint replacing burnt out parts in a Google datacenter. This paper is weird--like maybe they switched around the names of some things and didn't mention others. Just an FYI.
- packetslave 11y ago
  I used to work in cluster networking, before I became an SRE. Near as I can tell, no names have been switched around in this paper, and it covers all of the major generations of Google cluster networks up to a fairly recent point in time.
- mvgoogler 11y ago
  I used to work in platforms (not networking, but we worked pretty closely with them) and the information in the paper was accurate and surprisingly complete.
- deelowe 11y ago
  Been working on this stuff since the beginning. Looks right to me.
mik3y 11y ago
Finally! Time to fly the flannel.. I built the first 3 generations described here starting in '04; it's awesome now to have something to point to, especially as FB has lead the charge on opening their own gear. And I'm sure it's going to make collaboration and hiring that much easier. The secrecy was at first a universally-agreed necessity; in simplistic terms, we didn't want MSFT knowing how much money to throw at the problem, or even what the problem was. This was true (and surely still true to an extent) for all of platforms: it was always amusing to see public photo shoots at "google datacenters", which in reality were little more than a stack of google search appliances at a corp location. The level of detail here is great and really sums up 10 years of a lot of hard-earned findings. I'm thrilled pictures of the hardware are even included, that team is just top notch. For anyone who isn't a Google or Facebook but has large or growing colo/dc network needs, check out Cumulus Networks. It applies a lot of the SDN ideas (think ssh/puppet-driven config mgmt for your switches as just the start) and topology possibilities seen here; doesn't hurt that JR did a brief stint on firehose :)
- kijiki 11y ago
  Thanks for the kind words for Cumulus! - nolan
- deelowe 11y ago
  Hey Mikey! The first generation certainly was a blast! Amazing how far things have come. :-)
- ChuckMcM 11y ago
  I agree, its nice they have finally come out with some of the stuff they are/were doing. The Jupiter stuff sounds especially tasty for east-west heavy workloads.
mrbill 11y ago
Neat to see where the "Pluto" mystery switch might fit in, now. http://www.wired.com/2012/09/pluto-switch/ http://www.wired.com/2012/09/pluto-switch/
MattCruikshank 11y ago
I used to work at Google as well, and frankly I was so abstracted from all of this that it's absurd. It's like finally reading how my car is an "intern-al comb-usti-on eng-ine" whatever that means.
hackuser 11y ago
How much opportunity is there for this level of datacenter design to reduce power consumption? What are the trade-offs?
- wmf 11y ago
  In general a faster network allows more flexible scheduling, which should give more freedom to tune the scheduler for energy savings. Check out http://web.stanford.edu/~davidlo/resources/2015.thesis.pdf http://web.stanford.edu/~davidlo/resources/2015.thesis.pdf
- mik3y 11y ago
  The real power savings come not from the switches themsleves, but from the application and scheduling architectures it enabled. Having full cross-sectional bandwidth between any pair of hosts means the bin packing problem is a lot easier. You don't need to (say) make sure your map reduce job is scheduled with one shard per rack because racks only have so much bandwidth. Any host on any rack will do. You can forget racks even exist. This makes overall utilization of clusters more efficient (tighter bin packing), and the corollary is you don't need as many clusters and machines. (Not that has ever stopped Google from building more :)

97 points by packetslave 11y ago

Originally discussed at https://news.ycombinator.com/item?id=9734305 https://news.ycombinator.com/item?id=9734305 but this is the actual paper, which was just published.

ariwilson 11y ago

Amazing network infrastructure has always been one of my favorite hidden technical wonders behind Google. It's amazing how they handle so much data.

thrownaway2424 11y ago
When you actually have to use it what's amazing is how much congestion and packet loss there is. On paper it looks like a zero-impedance source of data but in reality it barely keeps up.
- raldi 11y ago
  What QoS are you referring to?

throwawayplanet 11y ago

I did a brief stint replacing burnt out parts in a Google datacenter. This paper is weird--like maybe they switched around the names of some things and didn't mention others. Just an FYI.

packetslave 11y ago
I used to work in cluster networking, before I became an SRE. Near as I can tell, no names have been switched around in this paper, and it covers all of the major generations of Google cluster networks up to a fairly recent point in time.
mvgoogler 11y ago
I used to work in platforms (not networking, but we worked pretty closely with them) and the information in the paper was accurate and surprisingly complete.
deelowe 11y ago
Been working on this stuff since the beginning. Looks right to me.

mik3y 11y ago

Finally! Time to fly the flannel.. I built the first 3 generations described here starting in '04; it's awesome now to have something to point to, especially as FB has lead the charge on opening their own gear. And I'm sure it's going to make collaboration and hiring that much easier. The secrecy was at first a universally-agreed necessity; in simplistic terms, we didn't want MSFT knowing how much money to throw at the problem, or even what the problem was. This was true (and surely still true to an extent) for all of platforms: it was always amusing to see public photo shoots at "google datacenters", which in reality were little more than a stack of google search appliances at a corp location. The level of detail here is great and really sums up 10 years of a lot of hard-earned findings. I'm thrilled pictures of the hardware are even included, that team is just top notch. For anyone who isn't a Google or Facebook but has large or growing colo/dc network needs, check out Cumulus Networks. It applies a lot of the SDN ideas (think ssh/puppet-driven config mgmt for your switches as just the start) and topology possibilities seen here; doesn't hurt that JR did a brief stint on firehose :)

kijiki 11y ago
Thanks for the kind words for Cumulus! - nolan
deelowe 11y ago
Hey Mikey! The first generation certainly was a blast! Amazing how far things have come. :-)
ChuckMcM 11y ago
I agree, its nice they have finally come out with some of the stuff they are/were doing. The Jupiter stuff sounds especially tasty for east-west heavy workloads.

mrbill 11y ago

Neat to see where the "Pluto" mystery switch might fit in, now. http://www.wired.com/2012/09/pluto-switch/ http://www.wired.com/2012/09/pluto-switch/

MattCruikshank 11y ago

I used to work at Google as well, and frankly I was so abstracted from all of this that it's absurd. It's like finally reading how my car is an "intern-al comb-usti-on eng-ine" whatever that means.

hackuser 11y ago

How much opportunity is there for this level of datacenter design to reduce power consumption? What are the trade-offs?

wmf 11y ago
In general a faster network allows more flexible scheduling, which should give more freedom to tune the scheduler for energy savings. Check out http://web.stanford.edu/~davidlo/resources/2015.thesis.pdf http://web.stanford.edu/~davidlo/resources/2015.thesis.pdf
mik3y 11y ago
The real power savings come not from the switches themsleves, but from the application and scheduling architectures it enabled. Having full cross-sectional bandwidth between any pair of hosts means the bin packing problem is a lot easier. You don't need to (say) make sure your map reduce job is scheduled with one shard per rack because racks only have so much bandwidth. Any host on any rack will do. You can forget racks even exist. This makes overall utilization of clusters more efficient (tighter bin packing), and the corollary is you don't need as many clusters and machines. (Not that has ever stopped Google from building more :)