4 ms·

It seems that it must be a really difficult problem to work out the optimal solution for having spare capacity to allow time/location shifting of workloads to m

It seems that it must be a really difficult problem to work out the optimal solution for having spare capacity to allow time/location shifting of workloads to minimize carbon per unit of compute. This Dell paper[0] suggests that 16% of the carbon over a typical server lifecycle is from the manufacture, so you probably don't want a server sitting there unused for 23 hours per day, since the overall carbon/compute ratio would be worse overall. The post doesn't mention this metric, but it would be really nice to see something more detailed in time - especially with this overall efficiency of the server/datacentre lifecycle in mind, rather than just energy consumed from use. [0]: https://i.dell.com/sites/csdocuments/CorpComm_Docs/en/carbon-footprint-poweredge-r640.pdf https://i.dell.com/sites/csdocuments/CorpComm_Docs/en/carbon...

webdva 6y ago
It is ultimately, in a certain sense, a mathematical optimization problem to determine the optimal configuration of the entire infrastucture of additional power sources. Perhaps like finding the optimal location position set of cell phone towers, perhaps using k-means clustering. Furthermore, additional issues must be resolved like legal regulations compliance—the decision maker or engineering agent has preferences or desires to satisfy.
shadowgovt 6y ago
Carbon consumed in building a server is sunk cost and would be paid independent of whether the server does any kind of carbon-footprint-aware load shifting. Assuming the server is "sitting unused for 23 hours a day" is the wrong model for what this work changed. You're assuming the server could be running at 50% duty cycle vs. 100% duty cyle. It isn't; since we're talking the batch load, there's a roughly fixed amount of low-priority work to be done and doubling the amount of CPU active-duty time alotted to doing the work doesn't get the work done faster (the details on that are complicated, but that's the right model for what Google's describing here). One should model the duty cycle as fixed relative to the processor (i.e. "This global datacenter architecture, over the course of its life, will do a fixed N units of computronium work on these batch tasks") and then ask whether that work should be done using coal to power the electrons or wind.
- heavenlyblue 6y ago
  Doesn’t not buying servers create pressure to not build them in the first place?
  - shadowgovt 6y ago
    I shouldn't have used "sunk cost;" I should have said "fixed cost." Google builds datacenters to N-year-long plans that are expensive to modify. Whether they're running their batch jobs on solar energy or coal energy, the carbon footprint of the datacenter build plan is not going to change. They want those datacenters anyway (mostly for the non-batch work and whatever the next big thing is that hasn't been invented yet but can only be done by putting a beach's worth of thinking sand on the problem).
- gbear605 6y ago
  Suppose I'm building a new datacenter that I want to do some constant amount of work each day. It doesn't matter the time of day. I can either power it with solar power, in which case it will run for 1/Y of the day, or with coal power, in which case it will run 100% of the day. If it only runs for 1/Y of the day, then I will need to buy Y times as many computers in the solar scenario than in the coal scenario. If Y = 2 and only 16% of the carbon in a typical coal-powered computer's lifetime is from the manufacture, then solar makes sense - solar is 2*16% = 32% of the carbon of coal. But if Y = 10 - so it's running 10% of the time, meaning there need to be 10x as many computers built - and 16% of the carbon is from the manufacture, then solar power is actually worse for the environment than coal power: solar takes 60% more carbon than coal power. Of course, this is a vastly simplified situation, but it points to the idea that we need to at least consider the carbon cost of manufacturing.
  - shadowgovt 6y ago
    But again, that's the thing. There is only 1/Y work to do in the day; it's batch work. The work in this case is constrained on the input side, not the CPU resources side; building 1 or 2 or 1,000 nodes to do the work won't decrease how expensive it is to do the work (in fact, building more computers than you need will make it cost more!). ... so why does Google build more computers than they need? Keep in mind that at Google scale, they're always and forever building "As many computers as we can possibly afford to" under the assumption that there will always be work for those machines to do. You and I may need to consider cost of manufacturing; Google doesn't. They always have the "Build datacenter infrastructure" cranked to an 11 (more accurately, they are following an N-year plan of construction that is extremely expensive to modify). That's the breakdown between Google's way of thinking and the way of thinking that you've presented: Google's cost of manufacture is fixed. Those computers will be built, whether or not they're going to also then run green-streamlined batch jobs. The limiting factor on the batch work is only so much work is generated in a day, and the rate the work is completed is already good enough that completing it in half the time yields no marginal value. So may as well complete it using sun instead of coal.
ip26 6y ago
Two possibly examples- - Spawn nightly regressions when wind power starts to pick up, instead of at some arbitrary wall clock time - Dispatch compute-heavy jobs during low energy cost times; dispatch IO-heavy or memory-limited jobs during high cost times.
adrianmonk 6y ago
> for having spare capacity to allow time/location shifting Part of that calculation should be the amount of compute capacity headroom you'd choose to have anyway even if you didn't care about carbon. Compute demands can vary from one day to the next. Maybe tomorrow people uploaded 3 times as many YouTube videos as they did today. Maybe load varies based on day of the week or day of the month. To some extent, you can smooth that out by delaying jobs, but there are practical limits. You also want some spare capacity just for safety. Efficient utilization is important, but things like performance regressions or spikes in demand can happen.

webdva 6y ago

It is ultimately, in a certain sense, a mathematical optimization problem to determine the optimal configuration of the entire infrastucture of additional power sources. Perhaps like finding the optimal location position set of cell phone towers, perhaps using k-means clustering. Furthermore, additional issues must be resolved like legal regulations compliance—the decision maker or engineering agent has preferences or desires to satisfy.

shadowgovt 6y ago

Carbon consumed in building a server is sunk cost and would be paid independent of whether the server does any kind of carbon-footprint-aware load shifting. Assuming the server is "sitting unused for 23 hours a day" is the wrong model for what this work changed. You're assuming the server could be running at 50% duty cycle vs. 100% duty cyle. It isn't; since we're talking the batch load, there's a roughly fixed amount of low-priority work to be done and doubling the amount of CPU active-duty time alotted to doing the work doesn't get the work done faster (the details on that are complicated, but that's the right model for what Google's describing here). One should model the duty cycle as fixed relative to the processor (i.e. "This global datacenter architecture, over the course of its life, will do a fixed N units of computronium work on these batch tasks") and then ask whether that work should be done using coal to power the electrons or wind.

heavenlyblue 6y ago
Doesn’t not buying servers create pressure to not build them in the first place?
- shadowgovt 6y ago
  I shouldn't have used "sunk cost;" I should have said "fixed cost." Google builds datacenters to N-year-long plans that are expensive to modify. Whether they're running their batch jobs on solar energy or coal energy, the carbon footprint of the datacenter build plan is not going to change. They want those datacenters anyway (mostly for the non-batch work and whatever the next big thing is that hasn't been invented yet but can only be done by putting a beach's worth of thinking sand on the problem).
gbear605 6y ago
Suppose I'm building a new datacenter that I want to do some constant amount of work each day. It doesn't matter the time of day. I can either power it with solar power, in which case it will run for 1/Y of the day, or with coal power, in which case it will run 100% of the day. If it only runs for 1/Y of the day, then I will need to buy Y times as many computers in the solar scenario than in the coal scenario. If Y = 2 and only 16% of the carbon in a typical coal-powered computer's lifetime is from the manufacture, then solar makes sense - solar is 2*16% = 32% of the carbon of coal. But if Y = 10 - so it's running 10% of the time, meaning there need to be 10x as many computers built - and 16% of the carbon is from the manufacture, then solar power is actually worse for the environment than coal power: solar takes 60% more carbon than coal power. Of course, this is a vastly simplified situation, but it points to the idea that we need to at least consider the carbon cost of manufacturing.
- shadowgovt 6y ago
  But again, that's the thing. There is only 1/Y work to do in the day; it's batch work. The work in this case is constrained on the input side, not the CPU resources side; building 1 or 2 or 1,000 nodes to do the work won't decrease how expensive it is to do the work (in fact, building more computers than you need will make it cost more!). ... so why does Google build more computers than they need? Keep in mind that at Google scale, they're always and forever building "As many computers as we can possibly afford to" under the assumption that there will always be work for those machines to do. You and I may need to consider cost of manufacturing; Google doesn't. They always have the "Build datacenter infrastructure" cranked to an 11 (more accurately, they are following an N-year plan of construction that is extremely expensive to modify). That's the breakdown between Google's way of thinking and the way of thinking that you've presented: Google's cost of manufacture is fixed. Those computers will be built, whether or not they're going to also then run green-streamlined batch jobs. The limiting factor on the batch work is only so much work is generated in a day, and the rate the work is completed is already good enough that completing it in half the time yields no marginal value. So may as well complete it using sun instead of coal.

ip26 6y ago

Two possibly examples- - Spawn nightly regressions when wind power starts to pick up, instead of at some arbitrary wall clock time - Dispatch compute-heavy jobs during low energy cost times; dispatch IO-heavy or memory-limited jobs during high cost times.

adrianmonk 6y ago

> for having spare capacity to allow time/location shifting Part of that calculation should be the amount of compute capacity headroom you'd choose to have anyway even if you didn't care about carbon. Compute demands can vary from one day to the next. Maybe tomorrow people uploaded 3 times as many YouTube videos as they did today. Maybe load varies based on day of the week or day of the month. To some extent, you can smooth that out by delaying jobs, but there are practical limits. You also want some spare capacity just for safety. Efficient utilization is important, but things like performance regressions or spikes in demand can happen.