4 ms·

> A software company building hardware is Google, and it's Pixel phones show FWIW Google has put a lot of effort in the hardware of their datacenters, and in t

> A software company building hardware is Google, and it's Pixel phones show FWIW Google has put a lot of effort in the hardware of their datacenters, and in that sense they're a hardware giant as well. Compared to their datacenter projects, Pixel is a funny hobbby for Google.

aliasxneo 9y ago
As a datacenter technician for Google, this makes me happy to see other people recognize this.
- kuschku 9y ago
  It’s just sad that we’ll never see any of that openly available, or described.
  - j-walker 9y ago
    There's a lot of openly available networking hardware on the Open Compute Project.
    - kuschku 9y ago
      Sure, but that's not even close to the "every board and switch is custom designed" story.
      - pyvpx 9y ago
        there isn't as much exotic "stuff" as you might think. they have the same ODMs build switches and line cards to their specifications, with chips from the same vendors (Broadcom, Cavium, etc) only they can easily hire a team and sign the NDA and write/optimize/perfect the forwarding plane and control plane implementations. hell, the IS-IS implementation in Quagga is usable because of Google :)
  - jankey 9y ago
    It's not a lot, but there are a bunch of pictures on google.com/about/datacenters, including a 'street view' inside our DC in North Carolina. There are also a bunch of papers on the network architecture, https://research.google.com/pubs/Networking.html https://research.google.com/pubs/Networking.html . But yes, a lot is closed for the obvious reason that this is really core to the business Google is running. (I'm a datacenter engineer at Google)
  - gcardone_ 9y ago
    Disclaimer: I'm an SRE at Google, my opinions are my own. We actually release a decent amount of information, see [0]. I particularly recommend "Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google's Datacenter Network" [1]. [0] https://research.google.com/pubs/Networking.html https://research.google.com/pubs/Networking.html [1] https://research.google.com/pubs/archive/46286.pdf https://research.google.com/pubs/archive/46286.pdf
- winslow 9y ago
  Could you shed some light on why google moved away from their shipping container design to a more traditional yet still custom data center approach? Thanks!
  - pavs 9y ago
    Not working for Google, but I think it's for multiple reasons. - hardware has gotten a lot faster, which helps the second reason. - Google giving away free GGC nodes to ISPs (if you meet their requirements). The distribution of these nodes has gotten a lot wider since container days. - Google is building more smaller scale (a dozen racks or so) traditional data centers closer to ISPs. I don't think shipping container design was never a very good idea, to begin with - it was an interesting experiment that didn't scale as well as they thought it would.
nunez 9y ago
Absolutely. The servers in their production racks are nearly 100% custom down to the silicon and involve serious amounts of engineering to make happen.
- mandeepj 9y ago
  Just curious - How are their servers different than FB's who also modified the heck out of them?
- agoodthrowaway 9y ago
  Sorry but I have to disagree. They are custom but that kind of customization is not all that difficult. Essentially it's using same HW as everyone else in a way that supports their scale better. That's it.
  - jchw 9y ago
    Now I really don't know anything about this, but I remember quite a while ago when configuring a Linux kernel build that there was an entire section dedicated to Google's server hardware. If it's really all that normal, why would it be Google-specific/why wouldn't there be other large vendors present as well? Talking about the GOOGLE_FIRMWARE menu.
- heeen2 9y ago
  But not custom enough for Intel to remove Management Engine from their CPUs
mandeepj 9y ago
> Pixel is a funny hobbby for Google Seems like it but unfortunately it is their product and they are selling it.

aliasxneo 9y ago

As a datacenter technician for Google, this makes me happy to see other people recognize this.

kuschku 9y ago
It’s just sad that we’ll never see any of that openly available, or described.
- j-walker 9y ago
  There's a lot of openly available networking hardware on the Open Compute Project.
  - kuschku 9y ago
    Sure, but that's not even close to the "every board and switch is custom designed" story.
    - pyvpx 9y ago
      there isn't as much exotic "stuff" as you might think. they have the same ODMs build switches and line cards to their specifications, with chips from the same vendors (Broadcom, Cavium, etc) only they can easily hire a team and sign the NDA and write/optimize/perfect the forwarding plane and control plane implementations. hell, the IS-IS implementation in Quagga is usable because of Google :)
- jankey 9y ago
  It's not a lot, but there are a bunch of pictures on google.com/about/datacenters, including a 'street view' inside our DC in North Carolina. There are also a bunch of papers on the network architecture, https://research.google.com/pubs/Networking.html https://research.google.com/pubs/Networking.html . But yes, a lot is closed for the obvious reason that this is really core to the business Google is running. (I'm a datacenter engineer at Google)
- gcardone_ 9y ago
  Disclaimer: I'm an SRE at Google, my opinions are my own. We actually release a decent amount of information, see [0]. I particularly recommend "Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google's Datacenter Network" [1]. [0] https://research.google.com/pubs/Networking.html https://research.google.com/pubs/Networking.html [1] https://research.google.com/pubs/archive/46286.pdf https://research.google.com/pubs/archive/46286.pdf
winslow 9y ago
Could you shed some light on why google moved away from their shipping container design to a more traditional yet still custom data center approach? Thanks!
- pavs 9y ago
  Not working for Google, but I think it's for multiple reasons. - hardware has gotten a lot faster, which helps the second reason. - Google giving away free GGC nodes to ISPs (if you meet their requirements). The distribution of these nodes has gotten a lot wider since container days. - Google is building more smaller scale (a dozen racks or so) traditional data centers closer to ISPs. I don't think shipping container design was never a very good idea, to begin with - it was an interesting experiment that didn't scale as well as they thought it would.

nunez 9y ago

Absolutely. The servers in their production racks are nearly 100% custom down to the silicon and involve serious amounts of engineering to make happen.

mandeepj 9y ago
Just curious - How are their servers different than FB's who also modified the heck out of them?
agoodthrowaway 9y ago
Sorry but I have to disagree. They are custom but that kind of customization is not all that difficult. Essentially it's using same HW as everyone else in a way that supports their scale better. That's it.
- jchw 9y ago
  Now I really don't know anything about this, but I remember quite a while ago when configuring a Linux kernel build that there was an entire section dedicated to Google's server hardware. If it's really all that normal, why would it be Google-specific/why wouldn't there be other large vendors present as well? Talking about the GOOGLE_FIRMWARE menu.
heeen2 9y ago
But not custom enough for Intel to remove Management Engine from their CPUs

mandeepj 9y ago

> Pixel is a funny hobbby for Google Seems like it but unfortunately it is their product and they are selling it.