5 ms·

"We defined and employed SDN principles to build Jupiter, a datacenter interconnect capable of supporting more than 100,000 servers and 1 Pb/s of total bandwidt

"We defined and employed SDN principles to build Jupiter, a datacenter interconnect capable of supporting more than 100,000 servers and 1 Pb/s of total bandwidth to host our services." This type of scales boggle my mind. Though I have found I can no longer keep up with all the terminologies popping up every day. Posts like these are my only connection to learning the massive scaling of things to make the modern networks work. "We leverage our large-scale computing infrastructure and signals from the application itself to learn how individual flows are performing, as determined by the end user’s perception of quality." Is this implying they are using Machine Learning to improve their own version of content delivery network?

sroussey 10y ago
The google network is gold plated, lacks the jitter inherent in the internet at large or inside other competitors' networks. It makes it tempting to ignore some aspects of distributed computing, if only for a moment.
- felipemnoa 10y ago
  Could you expand a bit more on you comment? I feel I'm missing some context. Specifically, what do you mean by gold plated? Why is it tempting to ignore some aspects of distributed computing? I'm missing a lot of context that you are implicitly implying so could you elaborate?
  - geeio 10y ago
    I assume they are talking about things like the truetime clocks used in spanner, which are not available on commodity hardware
    - wbl 10y ago
      Depends on commodity. You can just buy GPS slaved rubidium clocks with PTP output.
  - jemfinch 10y ago
    He's saying that it's so reliable and the throughput is so high that sometimes you have to convince yourself that your computers are halfway across the planet.
    - daenney 10y ago
      I'm not sure they're saying that, they're just claiming Google has really good and well run networks. But even Google hasn't solved the speed of light issue, packets can only travel so fast. If your computer is halfway across the planet, you'll notice no matter how fancy your network is.
  - dalailambda 10y ago
    Building distributed systems makes you aware of how unreliable things are at the large scale, e.g. the network. The parent comment implies that Google's network is so fast and reliable that it becomes tempting to ignore best practices and work as if it's a non distributed system.
  - walrus01 10y ago
    It's gold plated because they basically built their own ISP by acquiring either: a: dark fiber IRUs between cities/metro areas b: N x 10 and 100 Gbps wavelengths as L2 transport services from city to city, from a major carrier such as level3 or zayo c: some combination of A and B and they use that to build backbone links between their own network equipment that they have full control over. Google is its own AS and operates its own transport network around the US 48 states and around the world. the exact design of what they're doing within their own AS at layers 1 and 2 is pretty opaque unless you happen to be a carrier partner that is willing to violate a whole raft of NDAs. But basically they've built their own backbone to a very massive scale yet without the huge capital expense of actually laying their own fiber between cities. their network has incredibly low jitter because they don't run their links to saturation, and know EXACTLY what the latency is supposed to be from router interface to router interface between the pairs of core routers that are installed in each major city. Down to five decimal places, most likely. When you have your own dark fiber IRUs and operate your own WDM transport platforms you are in possession of things like OTDR traces for your dark fiber that tells you down to four decimal places the km length of your fiber path. It also helps that the sort of people who have 'enable' on the AS15169 routers and core network gear are recruited from the top tier of network engineers and appropriately compensated. If they weren't working for Google they would be working for another major global player like NTT, DT, France Telecom/Orange, SingTel or Softbank.
    - puzzle 10y ago
      Where do you get the crazy idea that Google doesn't run its links to saturation? It's crazy because it would cost an enormous amount of money. The B4 paper states multiple times that Google runs links at almost 100% saturation, versus the standard 30-40%. That's accomplished through the use of SDN technology and, even before that, through strict application of QoS. https://web.stanford.edu/class/cs244/papers/b4-sigcomm2013.pdf https://web.stanford.edu/class/cs244/papers/b4-sigcomm2013.p... A few more details about strategies here: https://research.google.com/pubs/archive/45385.pdf https://research.google.com/pubs/archive/45385.pdf Then there's a whole bunch of other host-side optimizations, including the use of new congestion control algorithms. http://queue.acm.org/detail.cfm?id=3022184 http://queue.acm.org/detail.cfm?id=3022184 You might recognize the name of the last author...

sroussey 10y ago

The google network is gold plated, lacks the jitter inherent in the internet at large or inside other competitors' networks. It makes it tempting to ignore some aspects of distributed computing, if only for a moment.

felipemnoa 10y ago
Could you expand a bit more on you comment? I feel I'm missing some context. Specifically, what do you mean by gold plated? Why is it tempting to ignore some aspects of distributed computing? I'm missing a lot of context that you are implicitly implying so could you elaborate?
- geeio 10y ago
  I assume they are talking about things like the truetime clocks used in spanner, which are not available on commodity hardware
  - wbl 10y ago
    Depends on commodity. You can just buy GPS slaved rubidium clocks with PTP output.
- jemfinch 10y ago
  He's saying that it's so reliable and the throughput is so high that sometimes you have to convince yourself that your computers are halfway across the planet.
  - daenney 10y ago
    I'm not sure they're saying that, they're just claiming Google has really good and well run networks. But even Google hasn't solved the speed of light issue, packets can only travel so fast. If your computer is halfway across the planet, you'll notice no matter how fancy your network is.
- dalailambda 10y ago
  Building distributed systems makes you aware of how unreliable things are at the large scale, e.g. the network. The parent comment implies that Google's network is so fast and reliable that it becomes tempting to ignore best practices and work as if it's a non distributed system.
- walrus01 10y ago
  It's gold plated because they basically built their own ISP by acquiring either: a: dark fiber IRUs between cities/metro areas b: N x 10 and 100 Gbps wavelengths as L2 transport services from city to city, from a major carrier such as level3 or zayo c: some combination of A and B and they use that to build backbone links between their own network equipment that they have full control over. Google is its own AS and operates its own transport network around the US 48 states and around the world. the exact design of what they're doing within their own AS at layers 1 and 2 is pretty opaque unless you happen to be a carrier partner that is willing to violate a whole raft of NDAs. But basically they've built their own backbone to a very massive scale yet without the huge capital expense of actually laying their own fiber between cities. their network has incredibly low jitter because they don't run their links to saturation, and know EXACTLY what the latency is supposed to be from router interface to router interface between the pairs of core routers that are installed in each major city. Down to five decimal places, most likely. When you have your own dark fiber IRUs and operate your own WDM transport platforms you are in possession of things like OTDR traces for your dark fiber that tells you down to four decimal places the km length of your fiber path. It also helps that the sort of people who have 'enable' on the AS15169 routers and core network gear are recruited from the top tier of network engineers and appropriately compensated. If they weren't working for Google they would be working for another major global player like NTT, DT, France Telecom/Orange, SingTel or Softbank.
  - puzzle 10y ago
    Where do you get the crazy idea that Google doesn't run its links to saturation? It's crazy because it would cost an enormous amount of money. The B4 paper states multiple times that Google runs links at almost 100% saturation, versus the standard 30-40%. That's accomplished through the use of SDN technology and, even before that, through strict application of QoS. https://web.stanford.edu/class/cs244/papers/b4-sigcomm2013.pdf https://web.stanford.edu/class/cs244/papers/b4-sigcomm2013.p... A few more details about strategies here: https://research.google.com/pubs/archive/45385.pdf https://research.google.com/pubs/archive/45385.pdf Then there's a whole bunch of other host-side optimizations, including the use of new congestion control algorithms. http://queue.acm.org/detail.cfm?id=3022184 http://queue.acm.org/detail.cfm?id=3022184 You might recognize the name of the last author...