7 ms·

Google joins Open Compute Project to drive standards in IT infrastructure

152 points by rey12rey 11y ago

wyldfire 11y ago
> energy efficient and more cost effective ... engaging the industry to identify better disk solutions for cloud based applications My pet issue w/IT infrastructure is the management modules. Finding a server w/a management module that works everytime is nigh impossible. Do google and Facebook design their own or do they somehow just work around their quirks?
- GauntletWizard 11y ago
  Google and Facebook don't bother with management modules; They're more likely on switched power for each machine, with significant work in automatic netboot reprovisioning when things go wrong on these systems. In the event that fails, they swap machines out for diagnosis... if they bother with the diagnosis part.
  - haneefmubarak 11y ago
    Ahh yes, the tried and true "turn it off and on again" philosophy... All kidding aside, if it's a random rare issue (some bit flipped somewhere), chances are that power cycling it may fix the issue (upon initialization). If it's anything more serious, the issue will likely persist, in which case taking the machine out of production is likely the best immediate course of action.
  - wmf 11y ago
    AFAIK Facebook tried reboot-on-LAN for one generation then went back to BMCs. I haven't seen any explanation about why, though.
- tyingq 11y ago
  Facebook apparently is just getting to this recently: https://code.facebook.com/posts/1471778586452119/openbmc-for-server-porting-and-supporting-new-features-for-yosemite-/ https://code.facebook.com/posts/1471778586452119/openbmc-for...
atomic77 11y ago
I haven't followed this project too closely, but, it seems interesting that it has taken this long for Google to join (and the conspicuous absence of Amazon). Anyone able to speculate on why now?
- payne92 11y ago
  Pure speculation: Google was the first to build very high density server farms. As such, they had various proprietary designs and approaches, and became heavily invested. And then it took them a while to decide that leveraging the open source ecosystem would be better than maintaining those various proprietary elements.
  - atomic77 11y ago
    This makes sense. Perhaps Amazon hasn't come far enough along in that process, or, they see their proprietary designs as competitive advantage.
  - ChuckMcM 11y ago
    I don't think it is speculation, Google has always considered the infrastructure things they did as a competitive advantage, super top secret stuff that even other employees don't know about kind of silos. And to your second point, it is interesting that OCP (some of it perhaps influenced by people who had worked at Google and then later worked at Facebook) appears to have minimized some of the advantages Google had, and now they find themselves possibly getting behind (one of the nice things, and bad things, about open projects like this are that you get a lot more resources applied to making things better than one company can muster). For similar reasons I wonder if ARM will displace x86 if only because there are probably 15 to 20 independent teams of smart people with ARM architecture licenses making better ARM processors, and perhaps 4 such teams at Intel making improvements in the x86 architecture. At some point it seems "open" seems to eventually overwhelms "walled garden." Although I'm totally open to counter examples where that hasn't been the case.
    - jldugger 11y ago
godzillabrennus 11y ago
This is great news and just another nail in the coffin of what Wired calls the Fucked By Cloud vendors: http://www.wired.com/2015/10/meet-walking-dead-hp-cisco-dell-emc-ibm-oracle/ http://www.wired.com/2015/10/meet-walking-dead-hp-cisco-dell...
- tw04 11y ago
  He fails to explain why "Amazon" is the future. He starts off by insinuating it's because traditional vendors are more expensive... I guess I'd ask to see the raw data he's using. I've run the numbers, AWS isn't cheaper AT ALL unless you're talking bursting workloads that run for less than a month, or a company that only needs one or two servers but still needs the reliability of a larger environment. Anything outside of that is cheaper to do on-prem 9 times out of 10. Either he's got his head in the sand, has bought into the "cloud is cheaper hype", or he has other justifications he fails to list.
  - tyingq 11y ago
    He implies that it's cheaper than, for example, buying HP+EMC+Oracle... And then, not directly, roughly saying that scaling up on cheap commodity type hardware is only possible because of the efforts of Amazon/Google/Facebook. He doesn't directly compare AWS versus, say Rackspace.
    - tw04 11y ago
      I'm not comparing it to Rackspace. I'm telling you I've run the numbers, and I can do it cheaper on prem with HP + EMC or Cisco + NetApp than I can in AWS. Unless I'm running compute at less than 50% capacity and storage at less than 60% capacity, it's far cheaper to do it on prem.
  - api 11y ago
    But cloud, because cloud and cloud cloud! We recently did the math on some cloud database solutions vs. warehousing data ourselves and using Glacier as a last-resort backup. Turns out it's orders of magnitude cheaper to DIY and would remain so unless we needed -- as you say -- a whole lot of burstable compute. In that case we could always upload (or ship) all the data to the cloud if we wanted. Cloud is cheaper for some work loads and use cases but not others. You have to analyze your own problem and see what works best. What cloud vendors have managed to do is put a ton of marketing hype out there to the point that cloud has captured the entire discourse. Today if it's not cloud it's not cloud and that means it's not cloud, because cloud. I've heard many stories recently of companies spending 2-4X what they currently spend on IT to put it in the cloud in order to save money. It's hilarious.
wilhil 11y ago
And, as not an employee of a multi billion pound company, how can I get involved?! I ask every time, and this project is amazing, but, it feels just for the big guys!
- pas 11y ago
  You can buy OCP compatible stuff, promote it, join the mailing list, blog about it, and so on. Otherwise, you can't. It's basically the club of kids who build this kind of hardware stuff (do R&D on infrastructure). When Facebook started OCP it seemed like an awkward initiative, after all, who would manufacture OCP-compatible stuff, when you can't find anything OCP-compatible at all? (It needs custom racks, for shame! Madness!) But slowly, it turned out, that there are multiple groups working on custom designs, trying to get out of the world of half-assed firmwares, crazy sales calls and useless support vectors associated with traditional vendors. So initially it only made sense if you had at least a few thousand servers and a team to work on that extra few percentages of efficiency. Nowadays, it seems a much more diverse club. And especially after the network stuff got opened up - the timing with SDN was right anyhow - they seem doomed to succeed.
  - skuhn 11y ago
    The equipment is available from various vendors, but the biggest hurdle is finding a datacenter facility that can handle OCP gear. Retail colocation is pretty much 19" racks only and 120V/208V power only. Even small scale wholesale datacenter deployments are tough to do if you want to go whole hog on OCP specs. You can still get OCP hardware in standard 19" form factors for these environments, but I don't personally see much point in it. You're seriously limiting your vendor and hardware selections, and if you aren't seeing the purported power efficiency gains then it's not worthwhile. The allure of eliminating the pain of classic vendors is still there, but there are paths through the Dell / HP / Supermicro maze that are tolerable without burning it down and going to OCP. There are other challenges with OCP's overall design in standard facilities beyond the server cabinet level too. Their (optional) battery cabinet design -- where you save on the facility cost by omitting the UPS bank in favor of battery cabinets for every triplet rack -- is explicitly forbidden at basically every top tier datacenter provider that I'm aware of. No batteries allowed on the floor because of the fire suppression system. You can definitely get a provider to build you a room where they'll let you have batteries on the floor, but that is a multi-megawatt sort of problem, not a 50kw sort of problem.
Animats 11y ago
Finally, racks go metric. The 19 inch rack is one of the oldest standards in computing. ENIAC used 19 inch racks. Open Compute, though, uses wider, and metric, racks. 19 inch rack gear can be mounted in Open Rack with an adapter. The Open Compute spec says that shelves of IT gear are provided with 12 VDC power. There's power conversion in the base of the rack. Facebook has standards for distribution to the racks at 54VDC, 277VAC 3Ø, and 230 VAC (Eurovolt). Apparently Google wants to add 48VDC, which was the old standard for telephone central offices. Facebook's choice of 54VDC distribution is strange. Anyone know why they picked that number?
- jauer 11y ago
  54VDC and 48VDC are roughly the same thing in telco land. E.g. I hooked up a Juniper MX960 last week with "-48 VDC Nominal" power supplies. The operating voltage range is -40 to -72VDC. A "48V" rectifier will output 54VDC to the power bus (feeding batteries and systems) since it's battery float voltage. If the rectifier goes offline (loss of AC power) the batteries will power the bus keeping systems online. I'm curious what the actual difference in practice is between Facebook's 54VDC and Google's 48VDC.
- wtbob 11y ago
  > Finally, racks go metric. How is that a benefit for anyone other than hardware makers?
- gumby 11y ago
  The historical telecom standard is -48 VDC. 48 is easy to understand (using 4 of the banks of 12 VDC lead-acid batteries in the basement of the CO building). The DC part is also easy to understand these days. But why a negative voltage? Think about it before you look it up! And why -54 VDC? Well typical lead acid chemistry actually yields 2.25 V cells -- 24 of those is 54, so it's another legacy artifact. Modern battery chemistry is different, but so much of the other tech is too. Here's a modern power supply I just found (http://www.tdipower.com/PDF/archive/10A_dcdc_conv.pdf http://www.tdipower.com/PDF/archive/10A_dcdc_conv.pdf) that supplies between -47 and -54 ... hmm, wonder why? Thankfully, telecom is fortunately one of the most backwards-compatible industries in the world, which is why the whole rickety edifice continues to work well (and support tons of innovation) even when people use it in ways never intended by its designers. BTW I'd be disappointed to learn if any reasonable 21st century EE education didn't still include this stuff.
spacecowboy_lon 11y ago
Interesting that it looks they are dumping 12v and going back to telecoms standard 48V :-)

152 points by rey12rey 11y ago

> energy efficient and more cost effective ... engaging the industry to identify better disk solutions for cloud based applications My pet issue w/IT infrastructure is the management modules. Finding a server w/a management module that works everytime is nigh impossible. Do google and Facebook design their own or do they somehow just work around their quirks?

GauntletWizard 11y ago
Google and Facebook don't bother with management modules; They're more likely on switched power for each machine, with significant work in automatic netboot reprovisioning when things go wrong on these systems. In the event that fails, they swap machines out for diagnosis... if they bother with the diagnosis part.
- haneefmubarak 11y ago
  Ahh yes, the tried and true "turn it off and on again" philosophy... All kidding aside, if it's a random rare issue (some bit flipped somewhere), chances are that power cycling it may fix the issue (upon initialization). If it's anything more serious, the issue will likely persist, in which case taking the machine out of production is likely the best immediate course of action.
- wmf 11y ago
  AFAIK Facebook tried reboot-on-LAN for one generation then went back to BMCs. I haven't seen any explanation about why, though.
tyingq 11y ago
Facebook apparently is just getting to this recently: https://code.facebook.com/posts/1471778586452119/openbmc-for-server-porting-and-supporting-new-features-for-yosemite-/ https://code.facebook.com/posts/1471778586452119/openbmc-for...

atomic77 11y ago

I haven't followed this project too closely, but, it seems interesting that it has taken this long for Google to join (and the conspicuous absence of Amazon). Anyone able to speculate on why now?

payne92 11y ago
Pure speculation: Google was the first to build very high density server farms. As such, they had various proprietary designs and approaches, and became heavily invested. And then it took them a while to decide that leveraging the open source ecosystem would be better than maintaining those various proprietary elements.
- atomic77 11y ago
  This makes sense. Perhaps Amazon hasn't come far enough along in that process, or, they see their proprietary designs as competitive advantage.
- ChuckMcM 11y ago
  I don't think it is speculation, Google has always considered the infrastructure things they did as a competitive advantage, super top secret stuff that even other employees don't know about kind of silos. And to your second point, it is interesting that OCP (some of it perhaps influenced by people who had worked at Google and then later worked at Facebook) appears to have minimized some of the advantages Google had, and now they find themselves possibly getting behind (one of the nice things, and bad things, about open projects like this are that you get a lot more resources applied to making things better than one company can muster). For similar reasons I wonder if ARM will displace x86 if only because there are probably 15 to 20 independent teams of smart people with ARM architecture licenses making better ARM processors, and perhaps 4 such teams at Intel making improvements in the x86 architecture. At some point it seems "open" seems to eventually overwhelms "walled garden." Although I'm totally open to counter examples where that hasn't been the case.
  - jldugger 11y ago

godzillabrennus 11y ago

This is great news and just another nail in the coffin of what Wired calls the Fucked By Cloud vendors: http://www.wired.com/2015/10/meet-walking-dead-hp-cisco-dell-emc-ibm-oracle/ http://www.wired.com/2015/10/meet-walking-dead-hp-cisco-dell...

tw04 11y ago
He fails to explain why "Amazon" is the future. He starts off by insinuating it's because traditional vendors are more expensive... I guess I'd ask to see the raw data he's using. I've run the numbers, AWS isn't cheaper AT ALL unless you're talking bursting workloads that run for less than a month, or a company that only needs one or two servers but still needs the reliability of a larger environment. Anything outside of that is cheaper to do on-prem 9 times out of 10. Either he's got his head in the sand, has bought into the "cloud is cheaper hype", or he has other justifications he fails to list.
- tyingq 11y ago
  He implies that it's cheaper than, for example, buying HP+EMC+Oracle... And then, not directly, roughly saying that scaling up on cheap commodity type hardware is only possible because of the efforts of Amazon/Google/Facebook. He doesn't directly compare AWS versus, say Rackspace.
  - tw04 11y ago
    I'm not comparing it to Rackspace. I'm telling you I've run the numbers, and I can do it cheaper on prem with HP + EMC or Cisco + NetApp than I can in AWS. Unless I'm running compute at less than 50% capacity and storage at less than 60% capacity, it's far cheaper to do it on prem.
- api 11y ago
  But cloud, because cloud and cloud cloud! We recently did the math on some cloud database solutions vs. warehousing data ourselves and using Glacier as a last-resort backup. Turns out it's orders of magnitude cheaper to DIY and would remain so unless we needed -- as you say -- a whole lot of burstable compute. In that case we could always upload (or ship) all the data to the cloud if we wanted. Cloud is cheaper for some work loads and use cases but not others. You have to analyze your own problem and see what works best. What cloud vendors have managed to do is put a ton of marketing hype out there to the point that cloud has captured the entire discourse. Today if it's not cloud it's not cloud and that means it's not cloud, because cloud. I've heard many stories recently of companies spending 2-4X what they currently spend on IT to put it in the cloud in order to save money. It's hilarious.

wilhil 11y ago

And, as not an employee of a multi billion pound company, how can I get involved?! I ask every time, and this project is amazing, but, it feels just for the big guys!

pas 11y ago
You can buy OCP compatible stuff, promote it, join the mailing list, blog about it, and so on. Otherwise, you can't. It's basically the club of kids who build this kind of hardware stuff (do R&D on infrastructure). When Facebook started OCP it seemed like an awkward initiative, after all, who would manufacture OCP-compatible stuff, when you can't find anything OCP-compatible at all? (It needs custom racks, for shame! Madness!) But slowly, it turned out, that there are multiple groups working on custom designs, trying to get out of the world of half-assed firmwares, crazy sales calls and useless support vectors associated with traditional vendors. So initially it only made sense if you had at least a few thousand servers and a team to work on that extra few percentages of efficiency. Nowadays, it seems a much more diverse club. And especially after the network stuff got opened up - the timing with SDN was right anyhow - they seem doomed to succeed.
- skuhn 11y ago
  The equipment is available from various vendors, but the biggest hurdle is finding a datacenter facility that can handle OCP gear. Retail colocation is pretty much 19" racks only and 120V/208V power only. Even small scale wholesale datacenter deployments are tough to do if you want to go whole hog on OCP specs. You can still get OCP hardware in standard 19" form factors for these environments, but I don't personally see much point in it. You're seriously limiting your vendor and hardware selections, and if you aren't seeing the purported power efficiency gains then it's not worthwhile. The allure of eliminating the pain of classic vendors is still there, but there are paths through the Dell / HP / Supermicro maze that are tolerable without burning it down and going to OCP. There are other challenges with OCP's overall design in standard facilities beyond the server cabinet level too. Their (optional) battery cabinet design -- where you save on the facility cost by omitting the UPS bank in favor of battery cabinets for every triplet rack -- is explicitly forbidden at basically every top tier datacenter provider that I'm aware of. No batteries allowed on the floor because of the fire suppression system. You can definitely get a provider to build you a room where they'll let you have batteries on the floor, but that is a multi-megawatt sort of problem, not a 50kw sort of problem.

Animats 11y ago

Finally, racks go metric. The 19 inch rack is one of the oldest standards in computing. ENIAC used 19 inch racks. Open Compute, though, uses wider, and metric, racks. 19 inch rack gear can be mounted in Open Rack with an adapter. The Open Compute spec says that shelves of IT gear are provided with 12 VDC power. There's power conversion in the base of the rack. Facebook has standards for distribution to the racks at 54VDC, 277VAC 3Ø, and 230 VAC (Eurovolt). Apparently Google wants to add 48VDC, which was the old standard for telephone central offices. Facebook's choice of 54VDC distribution is strange. Anyone know why they picked that number?

jauer 11y ago
54VDC and 48VDC are roughly the same thing in telco land. E.g. I hooked up a Juniper MX960 last week with "-48 VDC Nominal" power supplies. The operating voltage range is -40 to -72VDC. A "48V" rectifier will output 54VDC to the power bus (feeding batteries and systems) since it's battery float voltage. If the rectifier goes offline (loss of AC power) the batteries will power the bus keeping systems online. I'm curious what the actual difference in practice is between Facebook's 54VDC and Google's 48VDC.
wtbob 11y ago
> Finally, racks go metric. How is that a benefit for anyone other than hardware makers?
gumby 11y ago
The historical telecom standard is -48 VDC. 48 is easy to understand (using 4 of the banks of 12 VDC lead-acid batteries in the basement of the CO building). The DC part is also easy to understand these days. But why a negative voltage? Think about it before you look it up! And why -54 VDC? Well typical lead acid chemistry actually yields 2.25 V cells -- 24 of those is 54, so it's another legacy artifact. Modern battery chemistry is different, but so much of the other tech is too. Here's a modern power supply I just found (http://www.tdipower.com/PDF/archive/10A_dcdc_conv.pdf http://www.tdipower.com/PDF/archive/10A_dcdc_conv.pdf) that supplies between -47 and -54 ... hmm, wonder why? Thankfully, telecom is fortunately one of the most backwards-compatible industries in the world, which is why the whole rickety edifice continues to work well (and support tons of innovation) even when people use it in ways never intended by its designers. BTW I'd be disappointed to learn if any reasonable 21st century EE education didn't still include this stuff.

spacecowboy_lon 11y ago

Interesting that it looks they are dumping 12v and going back to telecoms standard 48V :-)