7 ms·

Optimizing for this is a perfect task to throw at a simple market. Especially because actually reworking the software to take advantage of resources at differen

by ben509 7y ago

Optimizing for this is a perfect task to throw at a simple market. Especially because actually reworking the software to take advantage of resources at different times is often going to require a decent amount of work by engineers. One way to do it would be assign various jobs a value, (which could be dynamic e.g. it might get more important as information becomes stale) and have them bid on compute power. You could make the value virtual. Or you could use real money. This is the premise behind EC2's spot instances. So when power is abundant, your prices drop and the relevant jobs kick off. Using real market prices makes sense especially if you're renting out computing power, most customers will be happy to adjust workloads to save money. Even if it's entirely internal, it's good to have a facility to "optimize for cost" and then report the savings. That's helpful to get the engineering resources devoted towards it, because "I saved $X" is a great bullet point to put in anyone's promotion packet or to base a bonus on.

bo1024 7y ago
Yes! Especially when you have multiple data centers participating in different locations. It might be cloudy in one place but not another, so jobs get re-routed accordingly.
Seirdy 7y ago
On a smaller scale, you could do what Low Tech Magazine [0] does and actually have downtime when sunlight is low. Since this doesn't happen too often and users can just save articles (with RSS, email newsletters, etc.), websites like this can just be powered by a single computer, solar cell, and small battery in the owner's Barcelona apartment. Thanks to small static pages and tiny dithered images, the site is almost always up. The future doesn't always need to be as "webscale" as Google; sometimes, scaling down is the smart thing to do. The minimal approach of LTM is the technology equivalent of riding a bicycle (or electric velomobile [1]) to work instead of driving. [0]: https://solar.lowtechmagazine.com https://solar.lowtechmagazine.com [1]: https://solar.lowtechmagazine.com/2012/10/electric-velomobiles.html https://solar.lowtechmagazine.com/2012/10/electric-velomobil...
- heavenlyblue 7y ago
  This is actually great: imagine a CDN in various geographical locations all of which work off sunlight in their own time zone (and turn off with no sunlight). This way you can have 24/7 fully green content delivery to consumers. Although that being said we could just try to cover the earth with as many generators everywhere and then fully connect the grids.
  - toomuchtodo 7y ago
    > Although that being said we could just try to cover the earth with as many generators everywhere and then fully connect the grids. This is the easier route: you do this by having as many businesses as possible purchase renewables PPAs, where they're specifically contracting for renewable energy. https://www.utilitydive.com/news/as-corporate-renewable-buying-surges-new-deals-pressure-utilities-to-impro/547485/ https://www.utilitydive.com/news/as-corporate-renewable-buyi...
  - samatman 7y ago
    I respect lowtechmagazine's experiment: it's designed to make you think, and it succeeds. But CPUs have an incredibly high embodied energy, we should aim for full utilization of servers, regardless of the source of that energy. If it's three CPUs with carbon-neutral electricity, or one CPU with electricity from coal, the latter is the responsible choice.
    - fuzzybear3965 7y ago
      Did you mean the reverse? I.e. "If it's one CPU with carbon-neutral electricity, or three CPUs with electricity from coal, the latter is the responsible choice." If no, then why. I can explain my confusion if needed.
      - samatman 7y ago
        I don't mean the reverse, I mean that running one CPU 24 hours a day instead of 3 CPUs 8 hours a day is better, even if the 3 CPUs have carbon-neutral energy and the 1 CPU does not. To be fair, these are artificial choices, and that's not even what lowtech was doing (they have a battery). I was responding to the post about the CDNs that stop working at night.
skybrian 7y ago
It's good to an extent, but if optimization for cost gets too intense then it will seek out the flaws in market rules. This will be true whether machines or humans are doing the optimizing. I guess it's okay as long as the people making the rules have good monitoring and are watching out for weird exploits and fixing them. The flexibility to change the rules tends to be more common internally than externally where customers want more guarantees. As we've seen, there also needs to be a balance between cost-optimization and preparedness. If the wind patterns don't match the prediction then you need to be ready for that. Also, as we've seen with cryptocurrency, real money attracts theft. A human-adjusted credit system is better. In the real world, this looks like support having the discretion to forgive big bills. But to do that they need to know their customers. It's hard to automate.
- uoaei 7y ago
  Tangent: this discussion is an interesting microcosm of the liberatarian/social-democratic dichotomy of economic theory. GP says a market will take care of itself, parent says not without significant regulation or else the perverse incentives will eventually be exposed and exploited.
  - skybrian 7y ago
    I think it's more a continuum than a dichotomy, since there can be more or less regulation. A regulated market is usually considered to be capitalism. For example, the US stock market is regulated by the SEC. But it does get nebulous as you increase the scope and goals of regulation.
pbhjpbhj 7y ago
In the UK households used to be able to get economy 7 electricity, which meant cheap electricity at night (I'd guess for 7 hours?). I've wanted to have realtime pricing like that for a while, it seems to be becoming available again. I honestly thought that was what the advanced electricity meter roll-out was going to do; but it seems not. More direct energy cost to service price charged seems like a good thing in general.
- blibble 7y ago
  https://octopus.energy/agile/ https://octopus.energy/agile/
- mikepurvis 7y ago
  TOU pricing is pretty widespread at this point; I have it in a medium-sized city in Canada, and intentionally run laundry (electric dryer) in the evening when power is half the cost of daytime use. With EVs coming online and being set up to charge at night, it would definitely be nice to have a minute by minute spot pricing scheme, though, then you'd basically have a mechanism for using the chargers and other intelligent devices on the consumption side as an intelligent buffer. It's goofy, but another one is situations where you have a lot of stored heat energy, thinking like pools, hot tubs, hot water heaters, etc— all those things could be activated in response to spot pricing with pretty simple policies (I want a shower of at least X degrees at 7am, I want the hot tub at at least Y degrees by 9pm, etc).
  - walshemj 7y ago
    I think when widespread overnight charging of EV's is a thing cheap evening electricity will be phased out
    - mikepurvis 7y ago
      Even in that world, you'd still want a way to manage the load through the night so that you don't have all the chargers clicking on at 11pm and then when the cars are full have a few hours later, demand craters until morning. With renewables the incentives for load management become even higher (per this article). The real next step after second-by-second billing would be setting up chargers with backfeeding capabilities/policies, so that you have an arrangement with your employer to charge your car at work on cheap daytime solar power, sell it back into the grid during the evening rush, then charge up again on overnight base load. Most EV batteries are way overspecced for what people need in daily use, so as long as you have a special "charge me to full and stay there" mode you can switch into, there'd be no reason (other than a bit of wear and tear) not to cycle your battery like this.
smadge 7y ago
Borg has concepts of quota and priority which function as the internal market you are talking about: Verma, et al. "Large-scale cluster management at Google with Borg".
lukev 7y ago
I really hoped this is what EC2 spot instances would be, but it doesn't seem to work that way. My spot instances usually get terminated due to "no available capacity" without any major price movement. It would also be pretty neat to integrate processing power markets with the wholesale energy markets. Energy prices are quite volatile and making load responsive to that would actually be quite helpful to stabilize them.
vladd 7y ago
> One way to do it would be assign various jobs a value. Or you could use real money. It's not the value of the outcome for that job that you're interested, but rather its sensitivity to a delayed latency in executing it. For example, preemptively converting Youtube videos to a lower resolution with optimum compression to avoid having to do it in real-time (when video is played) at a crappy compression (to be fast), is valuable for sure. It's just that it can be postponed for 24 hours without real impact. Executing a search for a single user is less valuable in terms of overall impact but much more latency-sensitive. (you can think of value and latency-sensitive in terms of two dimensions that are independent between them.) This idea helps save the planet for sure, but it requires cloud-providers to build APIs that enable devs to switch from the "here's the SSH to the server, do what you want with it" to a model where it's the devs that say instead "here's a lambda function and its desired latency execution, please schedule to run it for me and let me know when the result is ready" ( https://en.wikipedia.org/wiki/Inversion_of_control https://en.wikipedia.org/wiki/Inversion_of_control ) Google was able to do that because it owns a large part of the jobs executed in their datacenters. Hence they could build this adaptive scheduling for their own jobs quickly without necessarily passing through a cloud-based API that inverts the control of job scheduling.
- mtrovo 7y ago
  I think this is much more useful to a cloud provider than to a customer. As a customer I think you could configure somethings like you said using spot instances on AWS but that’s it, you’re going to save some small amount of dollars in a year but if you account for the engineering hours needed to set this up maybe is not really worth it. As a cloud provider you could juggle your client between datacenters depending on the load and price of energy over there. A flat rate for a cloud region means that there’s an opportunity for arbitration between datacenters that could be an increase of thousands in profits on their side.
  - vladd 7y ago
    If spot instance price is the only communication channel between you and the cloud provider to achieve this, it's hard to do a good job at it. For example, if the spot price was $0.60 one hour ago, and $0.55 right now, and you still have 13 hours of latency left for your job's execution, should you start triggering it or not? (how do you figure out your bid level?) You could have statistical data to have an intuition what's the lowest price that's been hit historically on similar days of the week in the past etc, but it's inexact and overly complicated. If the cloud provider becomes aware of the remaining latency you have at your disposal for the job's execution, they can do a much better job. They would be able to look across the entire job execution queue in that datacenter, each job having a specific remaining duration for its execution, they would know the predicted pattern of carbon/solar/wind split for the next hours, and the implementation of the system would sit just on the cloud provider side (making the life of the customers easier and simple). Of course, in the end, the benefit is towards our planet, but this is much more likely to succeed if the proper cloud API exists and the implementation initiatives are properly aligned to avoid redundant implementations on the customer side.
johnb 7y ago
Tangentially related but in Australia we run a household utility company that operates of that same assumption: https://www.amberelectric.com.au/ https://www.amberelectric.com.au/ Our hypothesis is that market signals combined with the right tools (friendly app and home automation) can help households shift demand into less carbon intensive periods. So far it's working pretty well.

by ben509 7y ago

bo1024 7y ago

Yes! Especially when you have multiple data centers participating in different locations. It might be cloudy in one place but not another, so jobs get re-routed accordingly.

Seirdy 7y ago

On a smaller scale, you could do what Low Tech Magazine [0] does and actually have downtime when sunlight is low. Since this doesn't happen too often and users can just save articles (with RSS, email newsletters, etc.), websites like this can just be powered by a single computer, solar cell, and small battery in the owner's Barcelona apartment. Thanks to small static pages and tiny dithered images, the site is almost always up. The future doesn't always need to be as "webscale" as Google; sometimes, scaling down is the smart thing to do. The minimal approach of LTM is the technology equivalent of riding a bicycle (or electric velomobile [1]) to work instead of driving. [0]: https://solar.lowtechmagazine.com https://solar.lowtechmagazine.com [1]: https://solar.lowtechmagazine.com/2012/10/electric-velomobiles.html https://solar.lowtechmagazine.com/2012/10/electric-velomobil...

heavenlyblue 7y ago
This is actually great: imagine a CDN in various geographical locations all of which work off sunlight in their own time zone (and turn off with no sunlight). This way you can have 24/7 fully green content delivery to consumers. Although that being said we could just try to cover the earth with as many generators everywhere and then fully connect the grids.
- toomuchtodo 7y ago
  > Although that being said we could just try to cover the earth with as many generators everywhere and then fully connect the grids. This is the easier route: you do this by having as many businesses as possible purchase renewables PPAs, where they're specifically contracting for renewable energy. https://www.utilitydive.com/news/as-corporate-renewable-buying-surges-new-deals-pressure-utilities-to-impro/547485/ https://www.utilitydive.com/news/as-corporate-renewable-buyi...
- samatman 7y ago
  I respect lowtechmagazine's experiment: it's designed to make you think, and it succeeds. But CPUs have an incredibly high embodied energy, we should aim for full utilization of servers, regardless of the source of that energy. If it's three CPUs with carbon-neutral electricity, or one CPU with electricity from coal, the latter is the responsible choice.
  - fuzzybear3965 7y ago
    Did you mean the reverse? I.e. "If it's one CPU with carbon-neutral electricity, or three CPUs with electricity from coal, the latter is the responsible choice." If no, then why. I can explain my confusion if needed.
    - samatman 7y ago
      I don't mean the reverse, I mean that running one CPU 24 hours a day instead of 3 CPUs 8 hours a day is better, even if the 3 CPUs have carbon-neutral energy and the 1 CPU does not. To be fair, these are artificial choices, and that's not even what lowtech was doing (they have a battery). I was responding to the post about the CDNs that stop working at night.

skybrian 7y ago

It's good to an extent, but if optimization for cost gets too intense then it will seek out the flaws in market rules. This will be true whether machines or humans are doing the optimizing. I guess it's okay as long as the people making the rules have good monitoring and are watching out for weird exploits and fixing them. The flexibility to change the rules tends to be more common internally than externally where customers want more guarantees. As we've seen, there also needs to be a balance between cost-optimization and preparedness. If the wind patterns don't match the prediction then you need to be ready for that. Also, as we've seen with cryptocurrency, real money attracts theft. A human-adjusted credit system is better. In the real world, this looks like support having the discretion to forgive big bills. But to do that they need to know their customers. It's hard to automate.

uoaei 7y ago
Tangent: this discussion is an interesting microcosm of the liberatarian/social-democratic dichotomy of economic theory. GP says a market will take care of itself, parent says not without significant regulation or else the perverse incentives will eventually be exposed and exploited.
- skybrian 7y ago
  I think it's more a continuum than a dichotomy, since there can be more or less regulation. A regulated market is usually considered to be capitalism. For example, the US stock market is regulated by the SEC. But it does get nebulous as you increase the scope and goals of regulation.

pbhjpbhj 7y ago

In the UK households used to be able to get economy 7 electricity, which meant cheap electricity at night (I'd guess for 7 hours?). I've wanted to have realtime pricing like that for a while, it seems to be becoming available again. I honestly thought that was what the advanced electricity meter roll-out was going to do; but it seems not. More direct energy cost to service price charged seems like a good thing in general.

blibble 7y ago
https://octopus.energy/agile/ https://octopus.energy/agile/
mikepurvis 7y ago
TOU pricing is pretty widespread at this point; I have it in a medium-sized city in Canada, and intentionally run laundry (electric dryer) in the evening when power is half the cost of daytime use. With EVs coming online and being set up to charge at night, it would definitely be nice to have a minute by minute spot pricing scheme, though, then you'd basically have a mechanism for using the chargers and other intelligent devices on the consumption side as an intelligent buffer. It's goofy, but another one is situations where you have a lot of stored heat energy, thinking like pools, hot tubs, hot water heaters, etc— all those things could be activated in response to spot pricing with pretty simple policies (I want a shower of at least X degrees at 7am, I want the hot tub at at least Y degrees by 9pm, etc).
- walshemj 7y ago
  I think when widespread overnight charging of EV's is a thing cheap evening electricity will be phased out
  - mikepurvis 7y ago
    Even in that world, you'd still want a way to manage the load through the night so that you don't have all the chargers clicking on at 11pm and then when the cars are full have a few hours later, demand craters until morning. With renewables the incentives for load management become even higher (per this article). The real next step after second-by-second billing would be setting up chargers with backfeeding capabilities/policies, so that you have an arrangement with your employer to charge your car at work on cheap daytime solar power, sell it back into the grid during the evening rush, then charge up again on overnight base load. Most EV batteries are way overspecced for what people need in daily use, so as long as you have a special "charge me to full and stay there" mode you can switch into, there'd be no reason (other than a bit of wear and tear) not to cycle your battery like this.

smadge 7y ago

Borg has concepts of quota and priority which function as the internal market you are talking about: Verma, et al. "Large-scale cluster management at Google with Borg".

lukev 7y ago

I really hoped this is what EC2 spot instances would be, but it doesn't seem to work that way. My spot instances usually get terminated due to "no available capacity" without any major price movement. It would also be pretty neat to integrate processing power markets with the wholesale energy markets. Energy prices are quite volatile and making load responsive to that would actually be quite helpful to stabilize them.

vladd 7y ago

> One way to do it would be assign various jobs a value. Or you could use real money. It's not the value of the outcome for that job that you're interested, but rather its sensitivity to a delayed latency in executing it. For example, preemptively converting Youtube videos to a lower resolution with optimum compression to avoid having to do it in real-time (when video is played) at a crappy compression (to be fast), is valuable for sure. It's just that it can be postponed for 24 hours without real impact. Executing a search for a single user is less valuable in terms of overall impact but much more latency-sensitive. (you can think of value and latency-sensitive in terms of two dimensions that are independent between them.) This idea helps save the planet for sure, but it requires cloud-providers to build APIs that enable devs to switch from the "here's the SSH to the server, do what you want with it" to a model where it's the devs that say instead "here's a lambda function and its desired latency execution, please schedule to run it for me and let me know when the result is ready" ( https://en.wikipedia.org/wiki/Inversion_of_control https://en.wikipedia.org/wiki/Inversion_of_control ) Google was able to do that because it owns a large part of the jobs executed in their datacenters. Hence they could build this adaptive scheduling for their own jobs quickly without necessarily passing through a cloud-based API that inverts the control of job scheduling.

mtrovo 7y ago
I think this is much more useful to a cloud provider than to a customer. As a customer I think you could configure somethings like you said using spot instances on AWS but that’s it, you’re going to save some small amount of dollars in a year but if you account for the engineering hours needed to set this up maybe is not really worth it. As a cloud provider you could juggle your client between datacenters depending on the load and price of energy over there. A flat rate for a cloud region means that there’s an opportunity for arbitration between datacenters that could be an increase of thousands in profits on their side.
- vladd 7y ago
  If spot instance price is the only communication channel between you and the cloud provider to achieve this, it's hard to do a good job at it. For example, if the spot price was $0.60 one hour ago, and $0.55 right now, and you still have 13 hours of latency left for your job's execution, should you start triggering it or not? (how do you figure out your bid level?) You could have statistical data to have an intuition what's the lowest price that's been hit historically on similar days of the week in the past etc, but it's inexact and overly complicated. If the cloud provider becomes aware of the remaining latency you have at your disposal for the job's execution, they can do a much better job. They would be able to look across the entire job execution queue in that datacenter, each job having a specific remaining duration for its execution, they would know the predicted pattern of carbon/solar/wind split for the next hours, and the implementation of the system would sit just on the cloud provider side (making the life of the customers easier and simple). Of course, in the end, the benefit is towards our planet, but this is much more likely to succeed if the proper cloud API exists and the implementation initiatives are properly aligned to avoid redundant implementations on the customer side.

johnb 7y ago

Tangentially related but in Australia we run a household utility company that operates of that same assumption: https://www.amberelectric.com.au/ https://www.amberelectric.com.au/ Our hypothesis is that market signals combined with the right tools (friendly app and home automation) can help households shift demand into less carbon intensive periods. So far it's working pretty well.