5 ms·

At Google Next in April, I attended a talk by the C-level in charge of Google's Datacenters, and they bragged about using machine learning to manage/anticipate

At Google Next in April, I attended a talk by the C-level in charge of Google's Datacenters, and they bragged about using machine learning to manage/anticipate their power draw, and the subsequent efficiencies those brought. They also bragged about how much they're trying to make these centers powered by renewables. When an audience member asked if they would be releasing their models that help them manage power, the speaker quickly changed the topic, which I felt told me all I needed to know about the sincerity of their motivations.

eitally 7y ago
The models in isolation aren't useful. The knowledge to create models like them is where the value lies, but it's only possible to generate value if you can do this in a way that is broadly applicable... which is where building systems management partnerships come into play. Yes, you can argue the point you made, but it's not particularly insightful because the specific IP employed to govern power mgmt at Google DCs is only truly applicable to Google DCs.
GhettoMaestro 7y ago
Release of source-code and licensing of said code is typically a very significant consideration for a company like Google. Even a "C-level" person in the singular sense most likely does not have the authority to approve such a thing. And if you don't have approval, you probably shouldn't talk or speculate about it in a public forum. Don't hate the player. Hate the legal-liability game.
Jyaif 7y ago
You think you can drop a model into a new data center with a different architecture, different censors, different workloads, and it will magically save energy?
- badrequest 7y ago
  That is a fair point, though I still think if your genuine motivation was the well-being of the Earth, then you'd, I dunno, _try?_
  - erikpukinskis 7y ago
    Presuming other people’s motivations from their behavior is a bit of a fool’s errand. It’s very easy to be wrong, and it’s the worst kind of wrong because you feel so justified. You have reasons.
  - Jyaif 7y ago
    That I agree with: they should have shared their methodology for saving energy.
    - BeatLeJuce 7y ago
      They did, the papers behind their tech are published and available free of charge (someone else here linked to https://deepmind.com/research/publications/safe-exploration-continuous-action-spaces https://deepmind.com/research/publications/safe-exploration-... )
- Barrin92 7y ago
  even if they didn't, seeing the limits of those models in different environments would certainly help the community overall, and if it saves a tenth of the energy it does for Google somewhere else it would be an enormous immediate reduction in energy consumption on a global scale, so it's hard to see what the downside would be. If one was truly concerned about 'democratizing AI', as companies like Google so often claim, then sharing access to the trained models would arguably be far more effective than just sharing research papers which many companies don't even know how to implement. In fact the large majority of companies that are responsible for energy consumption don't have any data on the scale that Google has, so I would go as far as call this a bad faith deflection from the beginning.
  - icebraining 7y ago
    It's exactly because the majority of companies don't know how to implement those models that it wouldn't do any good. This is the "give me teh codez" of AI. Google's models are specific to their environment, and they would be more likely to generate waste (at least in time and money required to implement them) than to reduce it.
modeless 7y ago
The trained models are likely not directly applicable to any other company's data centers. Instead, Google publishes machine learning research. Open access so that everyone can benefit. Here's a paper on Deepmind's reinforcement learning system for datacenter cooling: https://deepmind.com/research/publications/safe-exploration-continuous-action-spaces https://deepmind.com/research/publications/safe-exploration-...
mav3rick 7y ago
They could have saved Earth and you would find a fault. This is a great initiative, but you're criticizing about not "open sourcing" it. Google open sources plenty of things. The solution maybe heavily coupled to their infrastructure.
- dmix 7y ago
  And who knows it was probably out of the scope of the presentation and he simply didn't know if they had any plans to publish research or directly open source it. I doubt Google's being super protective of data center power optimization stuff. Demand optimization sounds like a relatively straightforward software problem for Amazon, Microsoft, etc to solve. Most of the utility would be for the smaller players.
bt848 7y ago
Google’s datacenter power management relies on software features that nobody outside Google is prepared to accept. Most operators expect to have thermal headroom for short CPU clock rate bursts and they don’t want some central daemon downclocking their machines. From what I’ve seen outside Google, most organizations would get much more benefit from focusing on utilization than trying to replicate Google fan speed control techniques. If you still have UPSs and dual AC-DC power supplies in every box and your PUE is 2.0 you don’t need the thing that allowed Google to go from 1.11 to 1.10.
dxbydt 7y ago
I did a bunch of work on managing/anticipating the power draw. ( https://news.ycombinator.com/item?id=13747481 https://news.ycombinator.com/item?id=13747481 ). Something nobody talks about is how grossly wasteful & inefficient Hadoop is. There's a bunch of cited papers (https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.580.5685&rep=rep1&type=pdf https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.58... ) that study power consumption of Hadoop nodes, but nothing on how to actually reduce the consumption. So here's something you can personally do to make a dent to this problem. Most of the Hadoop jobs that you write will involve some statistical summary over a dataset. Find the total, or the mean, median, 90th quantile, whatever. Writing a Hadoop map-reduce job is the single worst way to do this.Almost always, you can sample say a 1000 points, get a kernel density estimate via Parzen & then use a table. All quantiles, order statistics, functions of order stats...these can be hand computed for several univariate distributions & their location-scale families, and your real-life data can easily be bounded above below by these estimates. So you can get to > 95% accuracy just by hand-calculating. I can go into details if you like, but I suspect most of you already know how to do this.
- smueller1234 7y ago
  You're exactly right. Hadoop and it's ecosystem are... a very poor replica of the system whose predecessor the respective Google papers were written about. There's glaring efficiency things like not supporting complex encodings (last I used HDFS, at least, it could only do full replication) or the query engines on top not implementing sampling in their aggregates as a default/out of the box, which the Google tools do.
- lclarkmichalek 7y ago
  The good news is that barely anyone at FAANG scale is using Hadoop any more
- opportune 7y ago
  Another way to get huge efficiency gains over using Hadoop is to intelligently memoize results and compute as-you-go. But that requires making some bespoke architecture. However I contend that at any scale where it actually makes sense to use Hadoop (especially in the cloud era), it will be cheaper in the medium-term of 1 year or so to dedicate developer time to make something more efficient than a big daily pipeline of Hadoop jobs

eitally 7y ago

The models in isolation aren't useful. The knowledge to create models like them is where the value lies, but it's only possible to generate value if you can do this in a way that is broadly applicable... which is where building systems management partnerships come into play. Yes, you can argue the point you made, but it's not particularly insightful because the specific IP employed to govern power mgmt at Google DCs is only truly applicable to Google DCs.

GhettoMaestro 7y ago

Release of source-code and licensing of said code is typically a very significant consideration for a company like Google. Even a "C-level" person in the singular sense most likely does not have the authority to approve such a thing. And if you don't have approval, you probably shouldn't talk or speculate about it in a public forum. Don't hate the player. Hate the legal-liability game.

Jyaif 7y ago

You think you can drop a model into a new data center with a different architecture, different censors, different workloads, and it will magically save energy?

badrequest 7y ago
That is a fair point, though I still think if your genuine motivation was the well-being of the Earth, then you'd, I dunno, _try?_
- erikpukinskis 7y ago
  Presuming other people’s motivations from their behavior is a bit of a fool’s errand. It’s very easy to be wrong, and it’s the worst kind of wrong because you feel so justified. You have reasons.
- Jyaif 7y ago
  That I agree with: they should have shared their methodology for saving energy.
  - BeatLeJuce 7y ago
    They did, the papers behind their tech are published and available free of charge (someone else here linked to https://deepmind.com/research/publications/safe-exploration-continuous-action-spaces https://deepmind.com/research/publications/safe-exploration-... )
Barrin92 7y ago
even if they didn't, seeing the limits of those models in different environments would certainly help the community overall, and if it saves a tenth of the energy it does for Google somewhere else it would be an enormous immediate reduction in energy consumption on a global scale, so it's hard to see what the downside would be. If one was truly concerned about 'democratizing AI', as companies like Google so often claim, then sharing access to the trained models would arguably be far more effective than just sharing research papers which many companies don't even know how to implement. In fact the large majority of companies that are responsible for energy consumption don't have any data on the scale that Google has, so I would go as far as call this a bad faith deflection from the beginning.
- icebraining 7y ago
  It's exactly because the majority of companies don't know how to implement those models that it wouldn't do any good. This is the "give me teh codez" of AI. Google's models are specific to their environment, and they would be more likely to generate waste (at least in time and money required to implement them) than to reduce it.

modeless 7y ago

The trained models are likely not directly applicable to any other company's data centers. Instead, Google publishes machine learning research. Open access so that everyone can benefit. Here's a paper on Deepmind's reinforcement learning system for datacenter cooling: https://deepmind.com/research/publications/safe-exploration-continuous-action-spaces https://deepmind.com/research/publications/safe-exploration-...

mav3rick 7y ago

They could have saved Earth and you would find a fault. This is a great initiative, but you're criticizing about not "open sourcing" it. Google open sources plenty of things. The solution maybe heavily coupled to their infrastructure.

dmix 7y ago
And who knows it was probably out of the scope of the presentation and he simply didn't know if they had any plans to publish research or directly open source it. I doubt Google's being super protective of data center power optimization stuff. Demand optimization sounds like a relatively straightforward software problem for Amazon, Microsoft, etc to solve. Most of the utility would be for the smaller players.

bt848 7y ago

Google’s datacenter power management relies on software features that nobody outside Google is prepared to accept. Most operators expect to have thermal headroom for short CPU clock rate bursts and they don’t want some central daemon downclocking their machines. From what I’ve seen outside Google, most organizations would get much more benefit from focusing on utilization than trying to replicate Google fan speed control techniques. If you still have UPSs and dual AC-DC power supplies in every box and your PUE is 2.0 you don’t need the thing that allowed Google to go from 1.11 to 1.10.

dxbydt 7y ago

I did a bunch of work on managing/anticipating the power draw. ( https://news.ycombinator.com/item?id=13747481 https://news.ycombinator.com/item?id=13747481 ). Something nobody talks about is how grossly wasteful & inefficient Hadoop is. There's a bunch of cited papers (https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.580.5685&rep=rep1&type=pdf https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.58... ) that study power consumption of Hadoop nodes, but nothing on how to actually reduce the consumption. So here's something you can personally do to make a dent to this problem. Most of the Hadoop jobs that you write will involve some statistical summary over a dataset. Find the total, or the mean, median, 90th quantile, whatever. Writing a Hadoop map-reduce job is the single worst way to do this.Almost always, you can sample say a 1000 points, get a kernel density estimate via Parzen & then use a table. All quantiles, order statistics, functions of order stats...these can be hand computed for several univariate distributions & their location-scale families, and your real-life data can easily be bounded above below by these estimates. So you can get to > 95% accuracy just by hand-calculating. I can go into details if you like, but I suspect most of you already know how to do this.

smueller1234 7y ago
You're exactly right. Hadoop and it's ecosystem are... a very poor replica of the system whose predecessor the respective Google papers were written about. There's glaring efficiency things like not supporting complex encodings (last I used HDFS, at least, it could only do full replication) or the query engines on top not implementing sampling in their aggregates as a default/out of the box, which the Google tools do.
lclarkmichalek 7y ago
The good news is that barely anyone at FAANG scale is using Hadoop any more
opportune 7y ago
Another way to get huge efficiency gains over using Hadoop is to intelligently memoize results and compute as-you-go. But that requires making some bespoke architecture. However I contend that at any scale where it actually makes sense to use Hadoop (especially in the cloud era), it will be cheaper in the medium-term of 1 year or so to dedicate developer time to make something more efficient than a big daily pipeline of Hadoop jobs