18 ms·

Our data centers now work harder when the sun shines and wind blows

fhennig 6y ago
Interesting! Although there aren't any metrics on how much load is actually balanced this way. The only plot doesn't have y-axis labels.
kerberos84 6y ago
what a pity to know that they will be mining more of our personal data when we are chilling with the sunshine.
- yjftsjthsd-h 6y ago
  They're doing exactly the same amount of work, just time-shifted.
sambroner 6y ago
Seems like an intuitively good idea to me! It'd be great to see how effective this change was. Regardless of this change, I wonder if they share their forecasted non-renewable energy needs with their energy supplier so that the energy supplier can prepare for changes to the expected base load. Do any factories or other energy intensive operations do this?
- Klathmon 6y ago
  It's probably not at the same level of granularity that Google is trying to accomplish here, but I believe that power-hungry commercial systems have tried to move to when power is cheapest for many years now. Aluminum Foundries in particular are extremely power intensive and have been run during off-peak times (or are built in areas with cheap plentiful electricity like nearby hydro-electric dams). Still, i'd love to see this concept made a lot easier for the average consumer. Many people already have smart thermostats, why can't that talk to my power generation company and allow me to over heat/cool when the impact is lowest? Why can't my dish washer run automatically when it would impact the world the least? Why can't my EV automatically charge when power is most available? I know most of those things are possible, but they sure as hell aren't easy, and IMO they won't truly have an impact until they're on by default and don't require the user to do much of anything. These things seem like they are easily doable, but we just need the different industries to work together to come up with ways to have all of this stuff interoperate.
  - hnburnsy 6y ago
    Some power companies are integrating with your smart thermostat and may pre-cool in some case. Example, APS in Arizona... --- During an event, how will my thermostat be adjusted?  At the start of an event, your thermostat temperature will be automatically adjusted up a few degrees above the current temperature.  Each event will typically last an average of 2 hours, and will typically occur between 3 p.m. and 8 p.m. Events typically will not occur on holidays.  In some cases, your thermostat temperature may be adjusted down a few degrees prior to the event to pre-cool the home and ensure your comfort during an event.  Once the event is over, your thermostat will return to its normal set point and/or schedule.
    - ARandomerDude 6y ago
      My electricity provider does the same. It's opt-in and you get a $200 credit at the end of the summer if you stay in the program. Pretty good deal, I think.
    - ocdtrekkie 6y ago
      ComEd in Chicago can do this with Nest thermostats as well. At the time I had a Nest thermostat, they actually paid people to enable this. And I basically made back most of the cost of my Nest that way. (Before selling it off for a less cloud-connected thermostat.)
  - plorg 6y ago
    I believe some aluminum smelters can even bid into electrical markets, either directly as dispatchable load(/negative generation) or as ancillary services for grid stabilization.
    - beckingz 6y ago
      That's awesome. Do you have a source for that?
      - mschuster91 6y ago
- stcredzero 6y ago
  Seems like an intuitively good idea to me! Seems like getting rid of the middleman at a basic, physical level. Power is available for very low cost at certain times. So let's time-shift computation that doesn't have to be done at a specific time! Really, it's just the same trick as time-shifting your EV charging and other power draws. It costs money to run battery banks and inverters. Let's just take them out of the process, where we can. "The best part is no part." -- Elon Musk
- cdmp 6y ago
  It tends to work the other way around - at least in the markets I'm familiar with. If you're a large consumer of energy and can turn that consumption on or off at short notice (on the order of seconds) then the grid operator will pay you to allow them to scale your consumption up or down. The classic example of this is cold storage. If you have a warehouse full of freezers which need to be kept within a certain temperature threshold then it doesn't really matter when you run the freezers and you could switch off at several points during the day.
- user5994461 6y ago
  Having worked in a factory before, I can tell you that the factory was calling the electricity company before shutting down or turning on the factory. It is consuming a noticeable chunks out of what the whole surrounding city is consuming. Given how much electricity a datacenter consumes. Google surely must have a direct support contact within the electricity provider and they better start working both ways, if they don't already. Quick math. A datacenter is 60 000 servers, so 6 MW consumption at 100 W per server (moderate load). That is 1% of the peak output of a nuclear reactor. You bet the electric company wants to know when they need to adjust their reactors.
  - saagarjha 6y ago
    And reactors aren’t generally easy or desirable to adjust.
martincollignon 6y ago
COO from Tomorrow here (who provides the CO2 forecast data to Google). Happy to answer any questions!
- ZeroCool2u 6y ago
  Hey Martin, this is super cool! Congrats on the collab! Would you mind commenting on what your tech stack is like? Looking at your github repo it seems like you're combining a lot of data sources. Can you comment on your approach? Also, considering this collaboration, are you running on GCP?
  - corradio 6y ago
    Hello, Olivier here (CEO Tomorrow) Indeed we're running on GCP, using a mixture of Python and Node. In terms of the approach, I suggest checking out our blog (https://www.tmrow.com/blog https://www.tmrow.com/blog) and this talk (https://www.youtube.com/watch?v=PAelZb2ZYwI https://www.youtube.com/watch?v=PAelZb2ZYwI) which might provide a bit of clarity. Also, feel free to join our Slack at https://slack.tmrow.com https://slack.tmrow.com to ask your questions!
    - 205guy 6y ago
      That's a very interesting presentation (the youtube video), I'm glad someone is working in this space. Somewhat related, the presentation has a lot of math, which illustrates some of the points made in the recent HN discussion of the impact and effects of math instruction in Russia (https://news.ycombinator.com/item?id=22941144 https://news.ycombinator.com/item?id=22941144). Some comments pointed out how French schools rely on math for entrance exams, and this video a clear demonstration of a software startup having a math background. In fact, the startup probably wouldn't exist or even be conceived without the math and modeling of the data they provide.
- ar0 6y ago
  That looks like a good opportunity for an (at least tangentinally related) big shout-out to your great electricityMap (https://www.electricitymap.org/map https://www.electricitymap.org/map) website! I wish there would be more countries covered (in particular Switzerland), but I guess you depend on the live data being provided in these countries.
boris 6y ago
FWIW, for the build2 project we host our own CI servers on premesis and the power supply is supplemented by a solar array. We have configured our daily package rebuild window to coincide with the maximum solar output so that on a sunny day it is all done using renewable energy.
thebigshane 6y ago
Good idea, but I can't help but see this as marketing spin over the alternative title of the same story "Google: Data centers now perform LESS when the sun is not shining or the wind is not blowing"
- jorams 6y ago
  I don't see your point. That alternative title is exactly as accurate, so it seems like a good idea to pick the version that sounds a bit better and is easier to understand.
  - thebigshane 6y ago
    Of course its a better idea to pick the version that sounds better. That's what I said; I know both are accurate. My point is that by tying performance to environmental factors, you get a boost when things are great but then can have troubles when things are not great. Anyone familiar with solar panels already knows this, but if the correlation is obscured, it could be surprising. The article didn't mention a specific performance gain, but if we say you get an X% performance gain when the sun is out, it also means you get a similar X% performance loss when the sun is not out. Users of the system will get used to the improvement, which becomes the new standard, and then a particularly dreary season comes in with weeks of cloud cover, and suddenly there is concern about the degradation of service. (Like I said, it's still a good idea, it's efficient use of resources, but the PR is funny, that's all.)
    - beckingz 6y ago
      Encouraging and incentivizing compute/electricity demand to be time flexible provides the opportunity for cost savings and emissions reductions. The greater the flexibility, the greater the savings when demand can be smoothed out to better allocate resources and allow easier forecasting. If compute jobs that are run on demand can be deferred a few hours and run during a time period, that allows resources to better utilized. Like charging an EV overnight, but better.
bizzleDawg 6y ago
It seems that it must be a really difficult problem to work out the optimal solution for having spare capacity to allow time/location shifting of workloads to minimize carbon per unit of compute. This Dell paper[0] suggests that 16% of the carbon over a typical server lifecycle is from the manufacture, so you probably don't want a server sitting there unused for 23 hours per day, since the overall carbon/compute ratio would be worse overall. The post doesn't mention this metric, but it would be really nice to see something more detailed in time - especially with this overall efficiency of the server/datacentre lifecycle in mind, rather than just energy consumed from use. [0]: https://i.dell.com/sites/csdocuments/CorpComm_Docs/en/carbon-footprint-poweredge-r640.pdf https://i.dell.com/sites/csdocuments/CorpComm_Docs/en/carbon...
- webdva 6y ago
  It is ultimately, in a certain sense, a mathematical optimization problem to determine the optimal configuration of the entire infrastucture of additional power sources. Perhaps like finding the optimal location position set of cell phone towers, perhaps using k-means clustering. Furthermore, additional issues must be resolved like legal regulations compliance—the decision maker or engineering agent has preferences or desires to satisfy.
- shadowgovt 6y ago
  Carbon consumed in building a server is sunk cost and would be paid independent of whether the server does any kind of carbon-footprint-aware load shifting. Assuming the server is "sitting unused for 23 hours a day" is the wrong model for what this work changed. You're assuming the server could be running at 50% duty cycle vs. 100% duty cyle. It isn't; since we're talking the batch load, there's a roughly fixed amount of low-priority work to be done and doubling the amount of CPU active-duty time alotted to doing the work doesn't get the work done faster (the details on that are complicated, but that's the right model for what Google's describing here). One should model the duty cycle as fixed relative to the processor (i.e. "This global datacenter architecture, over the course of its life, will do a fixed N units of computronium work on these batch tasks") and then ask whether that work should be done using coal to power the electrons or wind.
meling 6y ago
Not sure I’m very impressed by the plot they show here. The results during the day looks ok, but then they only translate two nightly peaks (low carbon) into one slightly larger... couldn’t even more of the work be done at night... also it is strange that there is a dip in both ends of the plot (maybe they just plot one 24h period, ignoring the previous day’s load and the next day’s load... I think it would be more appropriate to consider previous/day as well, as a 24h snapshot over a multiday view) A more interesting measure would be the actual reduction in CO2 emissions.
ben509 6y ago
Optimizing for this is a perfect task to throw at a simple market. Especially because actually reworking the software to take advantage of resources at different times is often going to require a decent amount of work by engineers. One way to do it would be assign various jobs a value, (which could be dynamic e.g. it might get more important as information becomes stale) and have them bid on compute power. You could make the value virtual. Or you could use real money. This is the premise behind EC2's spot instances. So when power is abundant, your prices drop and the relevant jobs kick off. Using real market prices makes sense especially if you're renting out computing power, most customers will be happy to adjust workloads to save money. Even if it's entirely internal, it's good to have a facility to "optimize for cost" and then report the savings. That's helpful to get the engineering resources devoted towards it, because "I saved $X" is a great bullet point to put in anyone's promotion packet or to base a bonus on.
- bo1024 6y ago
  Yes! Especially when you have multiple data centers participating in different locations. It might be cloudy in one place but not another, so jobs get re-routed accordingly.
- Seirdy 6y ago
  On a smaller scale, you could do what Low Tech Magazine [0] does and actually have downtime when sunlight is low. Since this doesn't happen too often and users can just save articles (with RSS, email newsletters, etc.), websites like this can just be powered by a single computer, solar cell, and small battery in the owner's Barcelona apartment. Thanks to small static pages and tiny dithered images, the site is almost always up. The future doesn't always need to be as "webscale" as Google; sometimes, scaling down is the smart thing to do. The minimal approach of LTM is the technology equivalent of riding a bicycle (or electric velomobile [1]) to work instead of driving. [0]: https://solar.lowtechmagazine.com https://solar.lowtechmagazine.com [1]: https://solar.lowtechmagazine.com/2012/10/electric-velomobiles.html https://solar.lowtechmagazine.com/2012/10/electric-velomobil...
  - heavenlyblue 6y ago
shadowgovt 6y ago
While the green angle on this story is definitely good to highlight, I wonder if they're seeing any cost savings too? Solar and wind on the grid increase supply, which should drive down price per KwH (of course, the equation isn't quite that simple, since demand in most of the world near human population centers is also highest during the day).
- chickenpotpie 6y ago
  The pessimist in me says that they're only doing this in case a carbon tax passes, they'll be able to keep their pricing competitive.
  - barney54 6y ago
    I think this has very little to do with a carbon tax and everything about external and internal marketing. They are committed to using renewable power.
    - imtringued 6y ago
      They bought enough carbon credits to offset their emissions but it still left a bitter taste because data centers run 24 hours a day.
- 205guy 6y ago
  I don't think the electric grid follows a traditional supply-and-demand curve. Every kilowatt used must be generated at the same time. Generation comes in tiers: cheap baseload (coal or nuclear, each with their issues), expensive peaker plants (nat. gas), and cheap but unpredictably intermittent renewables. Prices are highly regulated and agreed to in long-term contracts that take into account peak usage and minimal generation capacity. If demand increases, it is really expensive to supply until the level is enough to build a new baseload plant, but even those are expensive now. Big users such as Google with its datacenters will of course negociate their own electricity contracts. I think renewables are the cheapest to buy right now, so by moving load around to be able to maximize use of cheap renewable electricity, they will definitely save money.
rcMgD2BwE72F 6y ago
I hope this doesn't add to much cost to their cloud customers' bills. They've worked so hard to sell their AI solutions to the fossil fuel industry, lately, so they can help them extract and burn more oil and gas[0]. [0] https://www.vox.com/recode/2020/1/3/21030688/google-amazon-ai-oil-gas https://www.vox.com/recode/2020/1/3/21030688/google-amazon-a...
- mav3rick 6y ago
  Is all your consumption from renewable sources ?
- jdm2212 6y ago
  Why would AI solutions help the fossil fuel industry burn more oil and gas? The fossil fuel industry sells oil and gas to others to burn, it doesn't light the stuff up itself for fun.
  - rcMgD2BwE72F 6y ago
    Almost all the oil we extract must be burned. There's far too little need for oil as a non-energy source, and there's a very good reason why the price of WTI is so low these days. The industry simply cannot find enough customers to burn it. How can Google, while helping others extract as much oil as possible, can still pretend it may not end up being burned? It's like a a drug dealer saying "I'm definitely not helping anyone taking drugs. I'm only selling it, they stay them somewhere if they please. Don't blame me".
    - jdm2212 6y ago
      You can make this argument more general (and IMO, even more ridiculous): Fossil fuels are global commodities. Anyone who reduces their oil consumption is reducing demand for those global commodities, which makes them cheaper. Making them cheaper makes renewable energy less competitive, which discourages investment, which means as a society we keep using fossil fuels for longer, which means in the end we produce more carbon. Ergo, anyone who tries to reduce their fossil fuel consumption by not flying, not driving, or turning off the A/C, is actually more responsible for destroying the planet.
elwell 6y ago
I like the animated illustration; very pleasant.
supernova87a 6y ago
While the story is very positive and encouraging -- Unfortunately an unintended side consequence of these kinds of efforts (unless you're very conscientious about maintaining the correct incentives, generally through pricing) is sometimes that the gains in energy efficiency and savings are clawed back by an increase in overall energy consumption because it's gotten effectively cheaper to operate for the same number of compute cycles. Just like with energy efficient LED light bulbs, although the overall energy use goes down, often it doesn't go down as much as it could have ideally, because people start lighting places that didn't have light before, because it's gotten so much more affordable to do so! Or like when you add highway lane capacity, traffic gets worse... Or in this case, the Google video engineers come up with new useless filters and resolutions to occupy the newly freed-up compute capacity. Just something to be aware of. The people who do this have to monitor and put in place controls so that the outcome is what they intended. Otherwise people are more clever than you think.
- Nasrudith 6y ago
  If it is still an improvement in both end usage and utility isn't that letting the perfect be the enemy of the good? LEDS have to be one of the worst possible example for claims of induced demand as a bad thing given that the efficency gains outstripped proliferation of additional always on devices and a cellphone per person. While Induced Demand may exist it too has its saturating limits of diminished returns.
mabbo 6y ago
(Off-topic, but regarding this site) Am I crazy or is this website capturing down-button clicks and ignoring them? I typically use down and up to slowly scroll as I read an article. This page is driving me nuts.
- dkarp 6y ago
  bizarrely it's only the down button (not the up button)
  - icelancer 6y ago
    Yeah, was able to replicate this on Chrome 81.0.4044.113 on Windows. Weird.
    - xingyzt 6y ago
      PageDown still works.
  - snazz 6y ago
    It appears to have something to do with this code (paste the URL into DevTools when you're on the site). webpack:///./frontend/js/trackers/keyboard-tracker.js There's one other keyboard event listener, but it doesn't appear to do that.
- vlasev 6y ago
  Funny enough, on Mac, fn + down works, although the jumps are larger.
  - samatman 6y ago
    This makes sense, fn + down sends Page Down, which is a different keystroke.
- crazygringo 6y ago
  Yes, down-arrow key is disabled. I'd thought it was just me and some badly behaving extension! Surprised to see that get through QA. (Up arrow works just fine.)
205guy 6y ago
This is the future. Using algorithms to optimize usage of renewable energy. Not only will it be lower carbon, it will be cheaper. What's interesting is they describe it working on forecasts (for wind and sun) instead of instantaneous renewable production. I wonder what the rationale for that was? Basing the algorithm on instantaneous information should be more accurate and thus give better savings, but maybe it varied too much to reliably run the loads they want. Imagine when your fridge can do this: freeze extra cold when the sun is shining (or wind is blowing), don't run the compressor when it's not, only run the blower after you open the door to move that extra cold from the freezer, allow a slightly larger temperature range, and of course run as necessary to avoid spoilage. It's not a simple algorithm, it has to handle various timeframes, such as solar being a daily cycle except there's less in winter and can go for a week or more with very little (storm/overcast). Maybe it could also use a bit of "learning" like the Nest thermostats to also optimize predicted usage. I know of one commercial product that sort of does this: the Zappi electric car charger. If you have grid-tied solar, it measures the current being fed back to the grid and adjusts the charging current to match. So if a cloud goes over your house, or you turn on a big appliance, the charger reduces the power to the car by the same amount. This maximizes the use of your own solar energy and minimizes the use of grid energy. https://myenergi.com/product/zappi/ https://myenergi.com/product/zappi/
- WalterBright 6y ago
  > Imagine when your fridge can do this I've been posting for years that an effective grid "battery" is internet connected refrigerators, water heaters, A/C, car chargers, etc., that only run when power is cheap, i.e. when solar/wind is providing excess power. A great deal of our demand for electricity is elastic and shiftable, which will eliminate a huge chunk of the need for grid batteries. Glad to see this finally gaining some traction!
  - maelito 6y ago
    > A great deal How much in % ?
  - konschubert 6y ago
    This will only work with fine-grained energy pricing (on the scale of minutes) and smart meters. Does this exist anywhere in the world?
btbuildem 6y ago
So Skynet will now prefer very sunny weather with strong consistent winds?
erentz 6y ago
This is a really good argument for carbon taxation to appropriately increase the cost of dirty energy. Send the correct price signal everywhere rather than making your own software do the equivalent of looking out the window at the weather and trying to decide if it’s a sunny or rainy or windy or clam day, and thus if solar or wind generation is making the grid cleaner. Or if instead those are likely offline and the grid is dirtier today. Best thing. Then you incentivize a cleaner grid overall and you don’t even have to worry eventually about this kind of thing.
- deleted 6y ago
  [deleted]
- ragebol 6y ago
  Exactly. It then also starts to make local storage (like powerwall) more interesting. When your PV generates a lot, prices will go down so better store the excess for times with higher prices. And when you don't generate PV power but can still store, you buy cheap also to use when prices go up. If there is also a dynamic price for using the grid, that usage will also spread.
- Barrin92 6y ago
  In particular a revenue neutral carbon tax with dividend should be politically as uncontroversial as it gets because it is also economically equitable. It's totally perplexing to me why these relatively low-hanging fruit solutions are not being pursued.
mempko 6y ago
Everyone wanting to really understand what is going on with the new green economy and these platitudes should watch Michael Moore's nee documentary he just released free on YouTube called Planet of the Humans. https://m.youtube.com/watch?v=Zk11vI-7czE https://m.youtube.com/watch?v=Zk11vI-7czE
- alcover 6y ago
  Thank you. This is beyond depressing. All cope and hope peddling.
- floatrock 6y ago
  This documentary is without nuance and without pragmatism. And it criticizes without proposing a path forward -- it's a bunch of cheap pot-shots, and it demands perfection instead of proposing progress. Yes, there are valid criticisms. Wind turbines are made of unrecyclable fiberglass. It takes energy to build them (truck rolls to the site, concrete for the foundations), and it's important to make sure the energy return on energy invested is net positive. We use fossil fuels to produce these renewables technologies. That's all true, but not insurmountable. They say battery storage makes up only a tiny percent of the needed capacity to overcome renewable intermittency. Sure, but it also omits how solar has dropped two orders of magnitude in price over the last few decades as we've built more of it and gotten better at making them (the "learning curve"). It follows a group of Vermont hikers hiking to a wind turbine site and then being NIMBY about it, but none of them talk about where their energy SHOULD come from. Look, it raises a lot of critical questions. But it also seems to expect a single magic pill that just doesn't exist. 2/3's of the way through they talk about the misrepresentations in biomass and point out how many organizations seem to be both for it and against it. "Which side are they really on?" says the classic accusatory documentary voiceover with scary music. Well, it's complicated! Clearly you don't want to burn all the forests all at once. And yeah, if you burn pressure-treated wood, those chemicals go into the local community. At the same time, wood does grow back. The nuance that's missing in this documentary is questions like "how many acres of rotationally-harvested woodlands are required to power a 1MW biomass plant sustainably in perpetuity? And can such projects exist in practice?" Biomass isn't a panacea solution, and the HN startup mindset of "can I scale up a technology to dominate everything" doesn't apply because biomass has limits to it's scalability. It's just one of many tools, and the problem about this documentary is it can't envision a future where many tools are used together. When a Sierra Club exec is questioned about biomass, they kept the part where she says their "position is nuanced", but then they cut to something else without explaining that nuance. That's lazy documentary filming. The complicated thing about energy is there is no silver bullet. This documentary finds the bad in each technology without considering how all the pieces could fit together. It presents the bad sides of each technology as if that should disqualify the tech instead of asking how can we improve each over time. There aren't easy answers to these questions, but this documentary just wallows in how bad everything is without asking the hard questions about how things can be made to work or what the alternative of doing nothing is.
cryptonector 6y ago
Our data centers now run slower when it's cloudy and there's no wind.
seanwilson 6y ago
Does anyone know of a good estimate for how much energy and emissions a typical Google search requires?
eitland 6y ago
I frequently criticize Google harshly for everything from search becoming more and more useless to pushing Chrome way to hard. Seems some people at Google still hasn't got the memo yet that the "not evil" days are now a thing of the past. This looks amazing and more like something I would expect from old Google.
hajderr 6y ago
Hope they crash some day too so you can stop tracking people :). How long does it take until this get -votes
PopeDotNinja 6y ago
Imagine climate modeling super computers that are carbon neutral.
duncan_bayne 6y ago
That headline reads more like a bug report than a press release. "Here's a barge full of coal. Maybe you can fix it with that."
globular-toast 6y ago
I've had an idea for the longest time that we should get rid of all the processes we've put in place to deliver everything "on-demand" and instead work with nature to get what we can. What I mean by this is that instead of deciding "I want to drive 200 miles to the beach" and buying a tank of petrol, you would instead wait for favourable wind/solar conditions in order to "save up" the energy you need such that you can afford to drive to the beach. If you are unfortunate one year you might only end up with half of what you need, but you'll still be able to do something. This goes for things like food too. Stop demanding the same food year round. Instead work with the seasons and eat what is available locally at that time of year. This would be such a huge boost to happiness. You can't see light if it's light all the time. We just don't know how great our lives are because we simply expect it to all be available at all the time. Expectations are simply assimilated and become invisible very quickly. Not only that but it turns out that meeting these expectations comes at a huge price. Let's instead take what nature gives us, but no more.
qu-everything 6y ago
Just a marketing device. Carbon neutral since 2007? Let me laugh in CO2, "green energies" are nowhere near carbon neutral. See planet of the humans by Moore.
driver8_ 6y ago
This is beautiful.

Interesting! Although there aren't any metrics on how much load is actually balanced this way. The only plot doesn't have y-axis labels.

kerberos84 6y ago

what a pity to know that they will be mining more of our personal data when we are chilling with the sunshine.

yjftsjthsd-h 6y ago
They're doing exactly the same amount of work, just time-shifted.

sambroner 6y ago

Seems like an intuitively good idea to me! It'd be great to see how effective this change was. Regardless of this change, I wonder if they share their forecasted non-renewable energy needs with their energy supplier so that the energy supplier can prepare for changes to the expected base load. Do any factories or other energy intensive operations do this?

Klathmon 6y ago
It's probably not at the same level of granularity that Google is trying to accomplish here, but I believe that power-hungry commercial systems have tried to move to when power is cheapest for many years now. Aluminum Foundries in particular are extremely power intensive and have been run during off-peak times (or are built in areas with cheap plentiful electricity like nearby hydro-electric dams). Still, i'd love to see this concept made a lot easier for the average consumer. Many people already have smart thermostats, why can't that talk to my power generation company and allow me to over heat/cool when the impact is lowest? Why can't my dish washer run automatically when it would impact the world the least? Why can't my EV automatically charge when power is most available? I know most of those things are possible, but they sure as hell aren't easy, and IMO they won't truly have an impact until they're on by default and don't require the user to do much of anything. These things seem like they are easily doable, but we just need the different industries to work together to come up with ways to have all of this stuff interoperate.
- hnburnsy 6y ago
  Some power companies are integrating with your smart thermostat and may pre-cool in some case. Example, APS in Arizona... --- During an event, how will my thermostat be adjusted?  At the start of an event, your thermostat temperature will be automatically adjusted up a few degrees above the current temperature.  Each event will typically last an average of 2 hours, and will typically occur between 3 p.m. and 8 p.m. Events typically will not occur on holidays.  In some cases, your thermostat temperature may be adjusted down a few degrees prior to the event to pre-cool the home and ensure your comfort during an event.  Once the event is over, your thermostat will return to its normal set point and/or schedule.
  - ARandomerDude 6y ago
    My electricity provider does the same. It's opt-in and you get a $200 credit at the end of the summer if you stay in the program. Pretty good deal, I think.
  - ocdtrekkie 6y ago
    ComEd in Chicago can do this with Nest thermostats as well. At the time I had a Nest thermostat, they actually paid people to enable this. And I basically made back most of the cost of my Nest that way. (Before selling it off for a less cloud-connected thermostat.)
- plorg 6y ago
  I believe some aluminum smelters can even bid into electrical markets, either directly as dispatchable load(/negative generation) or as ancillary services for grid stabilization.
  - beckingz 6y ago
    That's awesome. Do you have a source for that?
    - mschuster91 6y ago
stcredzero 6y ago
Seems like an intuitively good idea to me! Seems like getting rid of the middleman at a basic, physical level. Power is available for very low cost at certain times. So let's time-shift computation that doesn't have to be done at a specific time! Really, it's just the same trick as time-shifting your EV charging and other power draws. It costs money to run battery banks and inverters. Let's just take them out of the process, where we can. "The best part is no part." -- Elon Musk
cdmp 6y ago
It tends to work the other way around - at least in the markets I'm familiar with. If you're a large consumer of energy and can turn that consumption on or off at short notice (on the order of seconds) then the grid operator will pay you to allow them to scale your consumption up or down. The classic example of this is cold storage. If you have a warehouse full of freezers which need to be kept within a certain temperature threshold then it doesn't really matter when you run the freezers and you could switch off at several points during the day.
user5994461 6y ago
Having worked in a factory before, I can tell you that the factory was calling the electricity company before shutting down or turning on the factory. It is consuming a noticeable chunks out of what the whole surrounding city is consuming. Given how much electricity a datacenter consumes. Google surely must have a direct support contact within the electricity provider and they better start working both ways, if they don't already. Quick math. A datacenter is 60 000 servers, so 6 MW consumption at 100 W per server (moderate load). That is 1% of the peak output of a nuclear reactor. You bet the electric company wants to know when they need to adjust their reactors.
- saagarjha 6y ago
  And reactors aren’t generally easy or desirable to adjust.

martincollignon 6y ago

COO from Tomorrow here (who provides the CO2 forecast data to Google). Happy to answer any questions!

ZeroCool2u 6y ago
Hey Martin, this is super cool! Congrats on the collab! Would you mind commenting on what your tech stack is like? Looking at your github repo it seems like you're combining a lot of data sources. Can you comment on your approach? Also, considering this collaboration, are you running on GCP?
- corradio 6y ago
  Hello, Olivier here (CEO Tomorrow) Indeed we're running on GCP, using a mixture of Python and Node. In terms of the approach, I suggest checking out our blog (https://www.tmrow.com/blog https://www.tmrow.com/blog) and this talk (https://www.youtube.com/watch?v=PAelZb2ZYwI https://www.youtube.com/watch?v=PAelZb2ZYwI) which might provide a bit of clarity. Also, feel free to join our Slack at https://slack.tmrow.com https://slack.tmrow.com to ask your questions!
  - 205guy 6y ago
    That's a very interesting presentation (the youtube video), I'm glad someone is working in this space. Somewhat related, the presentation has a lot of math, which illustrates some of the points made in the recent HN discussion of the impact and effects of math instruction in Russia (https://news.ycombinator.com/item?id=22941144 https://news.ycombinator.com/item?id=22941144). Some comments pointed out how French schools rely on math for entrance exams, and this video a clear demonstration of a software startup having a math background. In fact, the startup probably wouldn't exist or even be conceived without the math and modeling of the data they provide.
ar0 6y ago
That looks like a good opportunity for an (at least tangentinally related) big shout-out to your great electricityMap (https://www.electricitymap.org/map https://www.electricitymap.org/map) website! I wish there would be more countries covered (in particular Switzerland), but I guess you depend on the live data being provided in these countries.

boris 6y ago

FWIW, for the build2 project we host our own CI servers on premesis and the power supply is supplemented by a solar array. We have configured our daily package rebuild window to coincide with the maximum solar output so that on a sunny day it is all done using renewable energy.

thebigshane 6y ago

Good idea, but I can't help but see this as marketing spin over the alternative title of the same story "Google: Data centers now perform LESS when the sun is not shining or the wind is not blowing"

jorams 6y ago
I don't see your point. That alternative title is exactly as accurate, so it seems like a good idea to pick the version that sounds a bit better and is easier to understand.
- thebigshane 6y ago
  Of course its a better idea to pick the version that sounds better. That's what I said; I know both are accurate. My point is that by tying performance to environmental factors, you get a boost when things are great but then can have troubles when things are not great. Anyone familiar with solar panels already knows this, but if the correlation is obscured, it could be surprising. The article didn't mention a specific performance gain, but if we say you get an X% performance gain when the sun is out, it also means you get a similar X% performance loss when the sun is not out. Users of the system will get used to the improvement, which becomes the new standard, and then a particularly dreary season comes in with weeks of cloud cover, and suddenly there is concern about the degradation of service. (Like I said, it's still a good idea, it's efficient use of resources, but the PR is funny, that's all.)
  - beckingz 6y ago
    Encouraging and incentivizing compute/electricity demand to be time flexible provides the opportunity for cost savings and emissions reductions. The greater the flexibility, the greater the savings when demand can be smoothed out to better allocate resources and allow easier forecasting. If compute jobs that are run on demand can be deferred a few hours and run during a time period, that allows resources to better utilized. Like charging an EV overnight, but better.

bizzleDawg 6y ago

It seems that it must be a really difficult problem to work out the optimal solution for having spare capacity to allow time/location shifting of workloads to minimize carbon per unit of compute. This Dell paper[0] suggests that 16% of the carbon over a typical server lifecycle is from the manufacture, so you probably don't want a server sitting there unused for 23 hours per day, since the overall carbon/compute ratio would be worse overall. The post doesn't mention this metric, but it would be really nice to see something more detailed in time - especially with this overall efficiency of the server/datacentre lifecycle in mind, rather than just energy consumed from use. [0]: https://i.dell.com/sites/csdocuments/CorpComm_Docs/en/carbon-footprint-poweredge-r640.pdf https://i.dell.com/sites/csdocuments/CorpComm_Docs/en/carbon...

webdva 6y ago
It is ultimately, in a certain sense, a mathematical optimization problem to determine the optimal configuration of the entire infrastucture of additional power sources. Perhaps like finding the optimal location position set of cell phone towers, perhaps using k-means clustering. Furthermore, additional issues must be resolved like legal regulations compliance—the decision maker or engineering agent has preferences or desires to satisfy.
shadowgovt 6y ago
Carbon consumed in building a server is sunk cost and would be paid independent of whether the server does any kind of carbon-footprint-aware load shifting. Assuming the server is "sitting unused for 23 hours a day" is the wrong model for what this work changed. You're assuming the server could be running at 50% duty cycle vs. 100% duty cyle. It isn't; since we're talking the batch load, there's a roughly fixed amount of low-priority work to be done and doubling the amount of CPU active-duty time alotted to doing the work doesn't get the work done faster (the details on that are complicated, but that's the right model for what Google's describing here). One should model the duty cycle as fixed relative to the processor (i.e. "This global datacenter architecture, over the course of its life, will do a fixed N units of computronium work on these batch tasks") and then ask whether that work should be done using coal to power the electrons or wind.

meling 6y ago

Not sure I’m very impressed by the plot they show here. The results during the day looks ok, but then they only translate two nightly peaks (low carbon) into one slightly larger... couldn’t even more of the work be done at night... also it is strange that there is a dip in both ends of the plot (maybe they just plot one 24h period, ignoring the previous day’s load and the next day’s load... I think it would be more appropriate to consider previous/day as well, as a 24h snapshot over a multiday view) A more interesting measure would be the actual reduction in CO2 emissions.

ben509 6y ago

Optimizing for this is a perfect task to throw at a simple market. Especially because actually reworking the software to take advantage of resources at different times is often going to require a decent amount of work by engineers. One way to do it would be assign various jobs a value, (which could be dynamic e.g. it might get more important as information becomes stale) and have them bid on compute power. You could make the value virtual. Or you could use real money. This is the premise behind EC2's spot instances. So when power is abundant, your prices drop and the relevant jobs kick off. Using real market prices makes sense especially if you're renting out computing power, most customers will be happy to adjust workloads to save money. Even if it's entirely internal, it's good to have a facility to "optimize for cost" and then report the savings. That's helpful to get the engineering resources devoted towards it, because "I saved $X" is a great bullet point to put in anyone's promotion packet or to base a bonus on.

bo1024 6y ago
Yes! Especially when you have multiple data centers participating in different locations. It might be cloudy in one place but not another, so jobs get re-routed accordingly.
Seirdy 6y ago
On a smaller scale, you could do what Low Tech Magazine [0] does and actually have downtime when sunlight is low. Since this doesn't happen too often and users can just save articles (with RSS, email newsletters, etc.), websites like this can just be powered by a single computer, solar cell, and small battery in the owner's Barcelona apartment. Thanks to small static pages and tiny dithered images, the site is almost always up. The future doesn't always need to be as "webscale" as Google; sometimes, scaling down is the smart thing to do. The minimal approach of LTM is the technology equivalent of riding a bicycle (or electric velomobile [1]) to work instead of driving. [0]: https://solar.lowtechmagazine.com https://solar.lowtechmagazine.com [1]: https://solar.lowtechmagazine.com/2012/10/electric-velomobiles.html https://solar.lowtechmagazine.com/2012/10/electric-velomobil...
- heavenlyblue 6y ago

shadowgovt 6y ago

While the green angle on this story is definitely good to highlight, I wonder if they're seeing any cost savings too? Solar and wind on the grid increase supply, which should drive down price per KwH (of course, the equation isn't quite that simple, since demand in most of the world near human population centers is also highest during the day).

chickenpotpie 6y ago
The pessimist in me says that they're only doing this in case a carbon tax passes, they'll be able to keep their pricing competitive.
- barney54 6y ago
  I think this has very little to do with a carbon tax and everything about external and internal marketing. They are committed to using renewable power.
  - imtringued 6y ago
    They bought enough carbon credits to offset their emissions but it still left a bitter taste because data centers run 24 hours a day.
205guy 6y ago
I don't think the electric grid follows a traditional supply-and-demand curve. Every kilowatt used must be generated at the same time. Generation comes in tiers: cheap baseload (coal or nuclear, each with their issues), expensive peaker plants (nat. gas), and cheap but unpredictably intermittent renewables. Prices are highly regulated and agreed to in long-term contracts that take into account peak usage and minimal generation capacity. If demand increases, it is really expensive to supply until the level is enough to build a new baseload plant, but even those are expensive now. Big users such as Google with its datacenters will of course negociate their own electricity contracts. I think renewables are the cheapest to buy right now, so by moving load around to be able to maximize use of cheap renewable electricity, they will definitely save money.

rcMgD2BwE72F 6y ago

I hope this doesn't add to much cost to their cloud customers' bills. They've worked so hard to sell their AI solutions to the fossil fuel industry, lately, so they can help them extract and burn more oil and gas[0]. [0] https://www.vox.com/recode/2020/1/3/21030688/google-amazon-ai-oil-gas https://www.vox.com/recode/2020/1/3/21030688/google-amazon-a...

mav3rick 6y ago
Is all your consumption from renewable sources ?
jdm2212 6y ago
Why would AI solutions help the fossil fuel industry burn more oil and gas? The fossil fuel industry sells oil and gas to others to burn, it doesn't light the stuff up itself for fun.
- rcMgD2BwE72F 6y ago
  Almost all the oil we extract must be burned. There's far too little need for oil as a non-energy source, and there's a very good reason why the price of WTI is so low these days. The industry simply cannot find enough customers to burn it. How can Google, while helping others extract as much oil as possible, can still pretend it may not end up being burned? It's like a a drug dealer saying "I'm definitely not helping anyone taking drugs. I'm only selling it, they stay them somewhere if they please. Don't blame me".
  - jdm2212 6y ago
    You can make this argument more general (and IMO, even more ridiculous): Fossil fuels are global commodities. Anyone who reduces their oil consumption is reducing demand for those global commodities, which makes them cheaper. Making them cheaper makes renewable energy less competitive, which discourages investment, which means as a society we keep using fossil fuels for longer, which means in the end we produce more carbon. Ergo, anyone who tries to reduce their fossil fuel consumption by not flying, not driving, or turning off the A/C, is actually more responsible for destroying the planet.

elwell 6y ago

I like the animated illustration; very pleasant.

supernova87a 6y ago

While the story is very positive and encouraging -- Unfortunately an unintended side consequence of these kinds of efforts (unless you're very conscientious about maintaining the correct incentives, generally through pricing) is sometimes that the gains in energy efficiency and savings are clawed back by an increase in overall energy consumption because it's gotten effectively cheaper to operate for the same number of compute cycles. Just like with energy efficient LED light bulbs, although the overall energy use goes down, often it doesn't go down as much as it could have ideally, because people start lighting places that didn't have light before, because it's gotten so much more affordable to do so! Or like when you add highway lane capacity, traffic gets worse... Or in this case, the Google video engineers come up with new useless filters and resolutions to occupy the newly freed-up compute capacity. Just something to be aware of. The people who do this have to monitor and put in place controls so that the outcome is what they intended. Otherwise people are more clever than you think.

Nasrudith 6y ago
If it is still an improvement in both end usage and utility isn't that letting the perfect be the enemy of the good? LEDS have to be one of the worst possible example for claims of induced demand as a bad thing given that the efficency gains outstripped proliferation of additional always on devices and a cellphone per person. While Induced Demand may exist it too has its saturating limits of diminished returns.

mabbo 6y ago

(Off-topic, but regarding this site) Am I crazy or is this website capturing down-button clicks and ignoring them? I typically use down and up to slowly scroll as I read an article. This page is driving me nuts.

dkarp 6y ago
bizarrely it's only the down button (not the up button)
- icelancer 6y ago
  Yeah, was able to replicate this on Chrome 81.0.4044.113 on Windows. Weird.
  - xingyzt 6y ago
    PageDown still works.
- snazz 6y ago
  It appears to have something to do with this code (paste the URL into DevTools when you're on the site). webpack:///./frontend/js/trackers/keyboard-tracker.js There's one other keyboard event listener, but it doesn't appear to do that.
vlasev 6y ago
Funny enough, on Mac, fn + down works, although the jumps are larger.
- samatman 6y ago
  This makes sense, fn + down sends Page Down, which is a different keystroke.
crazygringo 6y ago
Yes, down-arrow key is disabled. I'd thought it was just me and some badly behaving extension! Surprised to see that get through QA. (Up arrow works just fine.)

205guy 6y ago

This is the future. Using algorithms to optimize usage of renewable energy. Not only will it be lower carbon, it will be cheaper. What's interesting is they describe it working on forecasts (for wind and sun) instead of instantaneous renewable production. I wonder what the rationale for that was? Basing the algorithm on instantaneous information should be more accurate and thus give better savings, but maybe it varied too much to reliably run the loads they want. Imagine when your fridge can do this: freeze extra cold when the sun is shining (or wind is blowing), don't run the compressor when it's not, only run the blower after you open the door to move that extra cold from the freezer, allow a slightly larger temperature range, and of course run as necessary to avoid spoilage. It's not a simple algorithm, it has to handle various timeframes, such as solar being a daily cycle except there's less in winter and can go for a week or more with very little (storm/overcast). Maybe it could also use a bit of "learning" like the Nest thermostats to also optimize predicted usage. I know of one commercial product that sort of does this: the Zappi electric car charger. If you have grid-tied solar, it measures the current being fed back to the grid and adjusts the charging current to match. So if a cloud goes over your house, or you turn on a big appliance, the charger reduces the power to the car by the same amount. This maximizes the use of your own solar energy and minimizes the use of grid energy. https://myenergi.com/product/zappi/ https://myenergi.com/product/zappi/

WalterBright 6y ago
> Imagine when your fridge can do this I've been posting for years that an effective grid "battery" is internet connected refrigerators, water heaters, A/C, car chargers, etc., that only run when power is cheap, i.e. when solar/wind is providing excess power. A great deal of our demand for electricity is elastic and shiftable, which will eliminate a huge chunk of the need for grid batteries. Glad to see this finally gaining some traction!
- maelito 6y ago
  > A great deal How much in % ?
- konschubert 6y ago
  This will only work with fine-grained energy pricing (on the scale of minutes) and smart meters. Does this exist anywhere in the world?

btbuildem 6y ago

So Skynet will now prefer very sunny weather with strong consistent winds?

erentz 6y ago

This is a really good argument for carbon taxation to appropriately increase the cost of dirty energy. Send the correct price signal everywhere rather than making your own software do the equivalent of looking out the window at the weather and trying to decide if it’s a sunny or rainy or windy or clam day, and thus if solar or wind generation is making the grid cleaner. Or if instead those are likely offline and the grid is dirtier today. Best thing. Then you incentivize a cleaner grid overall and you don’t even have to worry eventually about this kind of thing.

deleted 6y ago
[deleted]
ragebol 6y ago
Exactly. It then also starts to make local storage (like powerwall) more interesting. When your PV generates a lot, prices will go down so better store the excess for times with higher prices. And when you don't generate PV power but can still store, you buy cheap also to use when prices go up. If there is also a dynamic price for using the grid, that usage will also spread.
Barrin92 6y ago
In particular a revenue neutral carbon tax with dividend should be politically as uncontroversial as it gets because it is also economically equitable. It's totally perplexing to me why these relatively low-hanging fruit solutions are not being pursued.

mempko 6y ago

Everyone wanting to really understand what is going on with the new green economy and these platitudes should watch Michael Moore's nee documentary he just released free on YouTube called Planet of the Humans. https://m.youtube.com/watch?v=Zk11vI-7czE https://m.youtube.com/watch?v=Zk11vI-7czE

alcover 6y ago
Thank you. This is beyond depressing. All cope and hope peddling.
floatrock 6y ago
This documentary is without nuance and without pragmatism. And it criticizes without proposing a path forward -- it's a bunch of cheap pot-shots, and it demands perfection instead of proposing progress. Yes, there are valid criticisms. Wind turbines are made of unrecyclable fiberglass. It takes energy to build them (truck rolls to the site, concrete for the foundations), and it's important to make sure the energy return on energy invested is net positive. We use fossil fuels to produce these renewables technologies. That's all true, but not insurmountable. They say battery storage makes up only a tiny percent of the needed capacity to overcome renewable intermittency. Sure, but it also omits how solar has dropped two orders of magnitude in price over the last few decades as we've built more of it and gotten better at making them (the "learning curve"). It follows a group of Vermont hikers hiking to a wind turbine site and then being NIMBY about it, but none of them talk about where their energy SHOULD come from. Look, it raises a lot of critical questions. But it also seems to expect a single magic pill that just doesn't exist. 2/3's of the way through they talk about the misrepresentations in biomass and point out how many organizations seem to be both for it and against it. "Which side are they really on?" says the classic accusatory documentary voiceover with scary music. Well, it's complicated! Clearly you don't want to burn all the forests all at once. And yeah, if you burn pressure-treated wood, those chemicals go into the local community. At the same time, wood does grow back. The nuance that's missing in this documentary is questions like "how many acres of rotationally-harvested woodlands are required to power a 1MW biomass plant sustainably in perpetuity? And can such projects exist in practice?" Biomass isn't a panacea solution, and the HN startup mindset of "can I scale up a technology to dominate everything" doesn't apply because biomass has limits to it's scalability. It's just one of many tools, and the problem about this documentary is it can't envision a future where many tools are used together. When a Sierra Club exec is questioned about biomass, they kept the part where she says their "position is nuanced", but then they cut to something else without explaining that nuance. That's lazy documentary filming. The complicated thing about energy is there is no silver bullet. This documentary finds the bad in each technology without considering how all the pieces could fit together. It presents the bad sides of each technology as if that should disqualify the tech instead of asking how can we improve each over time. There aren't easy answers to these questions, but this documentary just wallows in how bad everything is without asking the hard questions about how things can be made to work or what the alternative of doing nothing is.

cryptonector 6y ago

Our data centers now run slower when it's cloudy and there's no wind.

seanwilson 6y ago

Does anyone know of a good estimate for how much energy and emissions a typical Google search requires?

eitland 6y ago

I frequently criticize Google harshly for everything from search becoming more and more useless to pushing Chrome way to hard. Seems some people at Google still hasn't got the memo yet that the "not evil" days are now a thing of the past. This looks amazing and more like something I would expect from old Google.

hajderr 6y ago

Hope they crash some day too so you can stop tracking people :). How long does it take until this get -votes

PopeDotNinja 6y ago

Imagine climate modeling super computers that are carbon neutral.

duncan_bayne 6y ago

That headline reads more like a bug report than a press release. "Here's a barge full of coal. Maybe you can fix it with that."

globular-toast 6y ago

I've had an idea for the longest time that we should get rid of all the processes we've put in place to deliver everything "on-demand" and instead work with nature to get what we can. What I mean by this is that instead of deciding "I want to drive 200 miles to the beach" and buying a tank of petrol, you would instead wait for favourable wind/solar conditions in order to "save up" the energy you need such that you can afford to drive to the beach. If you are unfortunate one year you might only end up with half of what you need, but you'll still be able to do something. This goes for things like food too. Stop demanding the same food year round. Instead work with the seasons and eat what is available locally at that time of year. This would be such a huge boost to happiness. You can't see light if it's light all the time. We just don't know how great our lives are because we simply expect it to all be available at all the time. Expectations are simply assimilated and become invisible very quickly. Not only that but it turns out that meeting these expectations comes at a huge price. Let's instead take what nature gives us, but no more.

qu-everything 6y ago

Just a marketing device. Carbon neutral since 2007? Let me laugh in CO2, "green energies" are nowhere near carbon neutral. See planet of the humans by Moore.

driver8_ 6y ago

This is beautiful.