4 ms·

A Look inside Google’s Data Center Networks

164 points by cjdulberger 11y ago

Splendor 11y ago
I'm excited to see this type of information from Google but this post seems more like an announcemant that they released information and less like actual information.
- wmf 11y ago
  Hopefully the actual talk included more information. It should appear at https://www.youtube.com/user/OpenNetSummit/videos https://www.youtube.com/user/OpenNetSummit/videos at some point.
  - sciurus 11y ago
    The abstract is at http://opennetsummit.org/conference/agenda/wednesday/#keynotes-att-microsoft-google http://opennetsummit.org/conference/agenda/wednesday/#keynot... "Google’s data centers power the most demanding interactive, storage, and cloud services in the Internet, all requiring the highest levels of availability. Bandwidth and scale demands are growing exponentially, doubling approximately every year. Thus, Google’s data center network architecture must deliver cost-effective networking operating at the granularity of tens of thousands of servers, all while maintaining operational simplicity. We present a first look into Google’s data center network design and implementation, focusing on the data, control, and management plane principles underpinning five generations of our network architecture."
  - jhalstead 11y ago
    Looks like it's up @ https://www.youtube.com/watch?v=FaAZAII2x0w https://www.youtube.com/watch?v=FaAZAII2x0w
- packetslave 11y ago
  There's also a paper on the same material scheduled to be published at SIGCOMM in August: http://conferences.sigcomm.org/sigcomm/2015/papers.php http://conferences.sigcomm.org/sigcomm/2015/papers.php "Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network"
- jldugger 11y ago
  The Practice of Cloud System Administration (2015) contained a reference or two to Clos networking topologies and a vague hint that vendors might adopt it eventually. Given the author is a former Googler, I guess we now know why that aside block was put in.
  - YesThatTom2 11y ago
    OMG! I'm ecstatic that someone noticed! That made my day! http://the-cloud-book.com http://the-cloud-book.com
- sneak 11y ago
  "to organize the world’s information and make it universally accessible and useful" apparently just as serious as "don't be evil".
_spoonman 11y ago
I clicked the article, scrolled down a bit and said, "...that's it?"
- hga 11y ago
  First comment ends saying "(For full details you'll have to wait for a paper we'll publish at SIGCOMM 2015 in August.)"
lstamour 11y ago
See also: http://www.wired.com/2015/06/google-reveals-secret-gear-connects-online-empire/ http://www.wired.com/2015/06/google-reveals-secret-gear-conn... (again, not much info, but puts it in context for a less technical audience...)
- jamescun 11y ago
  For those curious about the erroneously mailed Pluto switch forum post referenced, from the archive with pictures: https://web.archive.org/web/20140410024606/http://www.networking-forum.com/viewtopic.php?f=46&t=29803 https://web.archive.org/web/20140410024606/http://www.networ...
fs111 11y ago
This would be a way better article if they kept the "ZOMG WE ARE SO AWESOME" and "WOW, WE INVENTED IT ALL" tone out..
- bitmapbrother 11y ago
  I didn't get that tone. Just curious, do you have an issue with Google?
  - fs111 11y ago
    no, just allergic to blog posts full of fluffy tech-marketing speak. Why not, "hey we made something that works for us, come check it out." instead of "that hot new thing you have over there? we had that for a decade b/c we are sooo smart."
    - bitmapbrother 11y ago
      Perhaps the 2015 ONS talk, on which the blog is based upon, will help. https://www.youtube.com/watch?v=xLFYuCre97Y&t=1h57m40s https://www.youtube.com/watch?v=xLFYuCre97Y&t=1h57m40s
  - hueving 11y ago
    They explicitly spell out how impressive their own stats were and how they were doing SDN before anyone else. How did you not get that tone?
hueving 11y ago
If any other company than Google posted a fluff piece like this, it would never see the front page! Are we really excited by an announcement about a future publication sprinkled with allusions to how much better they are than everyone?
- magicalist 11y ago
  > Please don't submit comments complaining that a submission is inappropriate for the site. If you think a story is spam or off-topic, flag it by clicking on its "flag" link. &c. It's a blog post announcing they're going to be presenting some new information on their network topologies at a conference this week, so not sure what you'd expected that's getting you so bent out of shape. Moreover it's a post with some impressive-sounding numbers and links to some legitimately interesting (albeit year or more old) publications. Seems like that puts it easily above 1/3 of front page submissions just right there.
SafdarIqbal 11y ago
These kinds of herculean efforts do make sense at Google's scale (at least from my reading of their papers & blog posts), but are there advantages for data centers operating at smaller scales to adopt some of the approaches used here e.g. custom-built commodity hardware-based network switches, SDN-based central controllers etc.? EDIT: Maybe a better question is, at what point do you think about eschewing the traditional Cisco/Juniper gear and look towards these techniques?
- sangnoir 11y ago
  > Maybe a better question is, at what point do you think about eschewing the traditional Cisco/Juniper gear and look towards these techniques? Answer 1: when it's cheaper. Answer 2: when traditional gear gets in the way. My guess is it's mostly 1 and rarely 2 for most organizations. Might be a combination of both.
  - eitally 11y ago
    The trap ... which ironically only affects bleeding edge corporations ... is that it's usually cheaper superficially, but doing anything novel in the infrastructure space leads to enormous issues of supportability downstream. For all the complaints programmers have about noobs not knowing anything, or training new hires on company coding standards & practices, it's FAR worse on the infrastructure side. I'd argue that traditional gear gets in the way a lot more than most people know / would think, but because you can pull any John Q Public off the street to support a Cisco or Juniper environment, "no one was ever fired for buying IBM" applies. Even investments in the common non-core networking gear is usually limited to just a handful of vendors so diversity isn't really a problem in most cases.
mark_l_watson 11y ago
I think that Google is starting a marketing blitz to compete with Amazon and Microsoft for cloud services. Inside Google, their technology is awesome but I think they need to get very solid in customer support to compete. You can quickly get tech help on the phone from Microsoft and Amazon, and Google needs to match that. That said, I have never signed up for their premium support so I might not be totally fair in this criticism.
- nulltype 11y ago
  Even their lowest level of premium support is pretty good. I think the higher levels have phone support.
  - vgt 11y ago
    At Google Cloud there are two ideas: - Things like BigQuery require less support in general (managed aspect of it = economies of scale of support. Think gmail support req's versus exchange) - Higher levels of support come with dedicated Google engineers, and they're not bad
Oculus 11y ago
Here's the talk itself https://www.youtube.com/watch?v=FaAZAII2x0w https://www.youtube.com/watch?v=FaAZAII2x0w

164 points by cjdulberger 11y ago

I'm excited to see this type of information from Google but this post seems more like an announcemant that they released information and less like actual information.

wmf 11y ago
Hopefully the actual talk included more information. It should appear at https://www.youtube.com/user/OpenNetSummit/videos https://www.youtube.com/user/OpenNetSummit/videos at some point.
- sciurus 11y ago
  The abstract is at http://opennetsummit.org/conference/agenda/wednesday/#keynotes-att-microsoft-google http://opennetsummit.org/conference/agenda/wednesday/#keynot... "Google’s data centers power the most demanding interactive, storage, and cloud services in the Internet, all requiring the highest levels of availability. Bandwidth and scale demands are growing exponentially, doubling approximately every year. Thus, Google’s data center network architecture must deliver cost-effective networking operating at the granularity of tens of thousands of servers, all while maintaining operational simplicity. We present a first look into Google’s data center network design and implementation, focusing on the data, control, and management plane principles underpinning five generations of our network architecture."
- jhalstead 11y ago
  Looks like it's up @ https://www.youtube.com/watch?v=FaAZAII2x0w https://www.youtube.com/watch?v=FaAZAII2x0w
packetslave 11y ago
There's also a paper on the same material scheduled to be published at SIGCOMM in August: http://conferences.sigcomm.org/sigcomm/2015/papers.php http://conferences.sigcomm.org/sigcomm/2015/papers.php "Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network"
jldugger 11y ago
The Practice of Cloud System Administration (2015) contained a reference or two to Clos networking topologies and a vague hint that vendors might adopt it eventually. Given the author is a former Googler, I guess we now know why that aside block was put in.
- YesThatTom2 11y ago
  OMG! I'm ecstatic that someone noticed! That made my day! http://the-cloud-book.com http://the-cloud-book.com
sneak 11y ago
"to organize the world’s information and make it universally accessible and useful" apparently just as serious as "don't be evil".

_spoonman 11y ago

I clicked the article, scrolled down a bit and said, "...that's it?"

hga 11y ago
First comment ends saying "(For full details you'll have to wait for a paper we'll publish at SIGCOMM 2015 in August.)"

lstamour 11y ago

See also: http://www.wired.com/2015/06/google-reveals-secret-gear-connects-online-empire/ http://www.wired.com/2015/06/google-reveals-secret-gear-conn... (again, not much info, but puts it in context for a less technical audience...)

jamescun 11y ago
For those curious about the erroneously mailed Pluto switch forum post referenced, from the archive with pictures: https://web.archive.org/web/20140410024606/http://www.networking-forum.com/viewtopic.php?f=46&t=29803 https://web.archive.org/web/20140410024606/http://www.networ...

fs111 11y ago

This would be a way better article if they kept the "ZOMG WE ARE SO AWESOME" and "WOW, WE INVENTED IT ALL" tone out..

bitmapbrother 11y ago
I didn't get that tone. Just curious, do you have an issue with Google?
- fs111 11y ago
  no, just allergic to blog posts full of fluffy tech-marketing speak. Why not, "hey we made something that works for us, come check it out." instead of "that hot new thing you have over there? we had that for a decade b/c we are sooo smart."
  - bitmapbrother 11y ago
    Perhaps the 2015 ONS talk, on which the blog is based upon, will help. https://www.youtube.com/watch?v=xLFYuCre97Y&t=1h57m40s https://www.youtube.com/watch?v=xLFYuCre97Y&t=1h57m40s
- hueving 11y ago
  They explicitly spell out how impressive their own stats were and how they were doing SDN before anyone else. How did you not get that tone?

hueving 11y ago

If any other company than Google posted a fluff piece like this, it would never see the front page! Are we really excited by an announcement about a future publication sprinkled with allusions to how much better they are than everyone?

magicalist 11y ago
> Please don't submit comments complaining that a submission is inappropriate for the site. If you think a story is spam or off-topic, flag it by clicking on its "flag" link. &c. It's a blog post announcing they're going to be presenting some new information on their network topologies at a conference this week, so not sure what you'd expected that's getting you so bent out of shape. Moreover it's a post with some impressive-sounding numbers and links to some legitimately interesting (albeit year or more old) publications. Seems like that puts it easily above 1/3 of front page submissions just right there.

SafdarIqbal 11y ago

These kinds of herculean efforts do make sense at Google's scale (at least from my reading of their papers & blog posts), but are there advantages for data centers operating at smaller scales to adopt some of the approaches used here e.g. custom-built commodity hardware-based network switches, SDN-based central controllers etc.? EDIT: Maybe a better question is, at what point do you think about eschewing the traditional Cisco/Juniper gear and look towards these techniques?

sangnoir 11y ago
> Maybe a better question is, at what point do you think about eschewing the traditional Cisco/Juniper gear and look towards these techniques? Answer 1: when it's cheaper. Answer 2: when traditional gear gets in the way. My guess is it's mostly 1 and rarely 2 for most organizations. Might be a combination of both.
- eitally 11y ago
  The trap ... which ironically only affects bleeding edge corporations ... is that it's usually cheaper superficially, but doing anything novel in the infrastructure space leads to enormous issues of supportability downstream. For all the complaints programmers have about noobs not knowing anything, or training new hires on company coding standards & practices, it's FAR worse on the infrastructure side. I'd argue that traditional gear gets in the way a lot more than most people know / would think, but because you can pull any John Q Public off the street to support a Cisco or Juniper environment, "no one was ever fired for buying IBM" applies. Even investments in the common non-core networking gear is usually limited to just a handful of vendors so diversity isn't really a problem in most cases.

mark_l_watson 11y ago

I think that Google is starting a marketing blitz to compete with Amazon and Microsoft for cloud services. Inside Google, their technology is awesome but I think they need to get very solid in customer support to compete. You can quickly get tech help on the phone from Microsoft and Amazon, and Google needs to match that. That said, I have never signed up for their premium support so I might not be totally fair in this criticism.

nulltype 11y ago
Even their lowest level of premium support is pretty good. I think the higher levels have phone support.
- vgt 11y ago
  At Google Cloud there are two ideas: - Things like BigQuery require less support in general (managed aspect of it = economies of scale of support. Think gmail support req's versus exchange) - Higher levels of support come with dedicated Google engineers, and they're not bad

Oculus 11y ago

Here's the talk itself https://www.youtube.com/watch?v=FaAZAII2x0w https://www.youtube.com/watch?v=FaAZAII2x0w