5 ms·

I loved this part at the end: "By contrast, allowing a temperature increase for chips in large data centers could have surprising environmental benefits. To thi

I loved this part at the end: "By contrast, allowing a temperature increase for chips in large data centers could have surprising environmental benefits. To this point, Keysight’s White recalled that a company once requested JEDEC increase the spec for an operating temperature by five degrees. The estimate of the potential savings was stunning. Based on how much energy they consumed annually for cooling, they calculated a five degree change could translate to shutting down three coal power plants per year. JEDEC ultimately compromised on the suggestion."

gjsman-1000 4y ago
Only three? That's not an immediate win. What if the temperature increase causes just so slightly more failures, causing so slightly more replacements, and each replacement requires energy to make, ship, install, replace, recycle, the effects of increased demand... What if it lasts slightly less long, causing more early failures and eWaste? After all that potential risk, is it a benefit still, and if so, how much? We don't know and it is hard to know - but I don't blame JEDEC and would not call it a "compromise" on their part like it was a superior option.
- picture 4y ago
  Cmon.. logically it has to be more than just three right? "Cooling efficiency" sometimes come in unit of W/°C difference, so I'd imagine that a few more degrees would be a huge deal.
  - gjsman-1000 4y ago
    I don't know how much power efficiency would be saved - my concern is more that it is completely logical that running any part at higher temperatures causes increased risk of failure, whether it be a computer part or a mechanical part. How much? I don't know - I just don't blame JEDEC for recognizing this is not a clear and obvious win. Imagine if the failure rate was raised by as little as 1%. RAM Failure is not uncommon compared to other parts - I've had it happen before and render a system unable to boot, that's why we have Memtest86 and not CPUtest86 or SSDtest86. A 1% increase in failure over 5 years would have effects just as unbelievable as the power saving that increasing the temperature would be. How many smartphones would be junked? How many PCs would be thrown out for not working by people who are average Joes who can't diagnose them, and the extra waste that generates from both disposing the old PC and purchasing a new one? Perhaps the new PC is more efficient, but which is better, greater emissions or more eWaste in the ground due to the new PC being likely more efficient than the old one? The point is that it is not a clear win. With further research it might be, and I might be all for it. I'm only nitpicking the description of it as being a "compromise" as though it were obvious. [@picture: I'm at my posting limit for the day because HN is, well... I'll leave their censorship policies for another day. I would agree with you if the RAM with the 90C limit were strictly ECC RAM because that is most often used in data centers and not consumer parts. Maybe we have non-ECC/85 RAM and ECC/90 RAM options...]
    - picture 4y ago
      That's for sure. I 100% agree that increased temperature will statistically increase failure rate. I'm just thinking that, the most common mechanisms of thermal failure in electronics are caused by repeated thermal cycling which cause fatigue and stress failures at interconnects (solder bumps, silicon bonding, etc). Data centers are designed to be operated in a relatively very constant temperature environment, so I would suspect that the failure rate may not be raised significantly.
    - dcow 4y ago
      Well now you're just being hyperbolic. As you say, this is an engineering problem so solutions are far from their ideals states in either direction. However, 1% increase in ram failure rates ruining the world? That doesn't sound right. Errors are encountered in RAM all the time and guess what, they're corrected often by the hardware before even bothering the system. I'm sure we could deal with a 1% increase...
    - Spooky23 4y ago
      Most datacenter hardware is fine at 95 degrees F (inlet temp). Approved configurations are usually available to 105 degrees F or slightly higher. Some devices can run as high as 130F. In the operating range, you’re not going to have any measurable change in operations or failure rate - if you do the parts are defective. All of the stories you hear about this and that are conjecture.
      - smolder 4y ago
        Interestingly, computer chips can often be run at lower voltage and wattage for a given frequency if they are kept at a colder temperature. As a home user I can significantly reduce power draw for a CPU/GPU by improving the cooling solution and lowering voltages. The reasons this doesn't work for datacenters is two-fold, I think: First, they won't see efficiency improvements just by keeping their CPUs and GPUs (or RAM) cooler because the power levels/tables for the chips are baked-in, and operators aren't going to the trouble of tweaking voltages themselves. Second, even if they did tweak voltages, the cost of sustaining lower temperatures with better cooling likely won't outweigh the savings resulting from lower power draw for the chips. Still, this raises the question of whether designing hardware for higher operating temperatures is always the right move. At some point there's going to be a cost in performance and/or efficiency that outweighs the savings from allowing higher temperatures. Ideally these tradeoffs should be balanced as a whole.
  - marcosdumay 4y ago
    You usually have to budget a few °C of difference just for pushing enough energy through heat exchangers. So the rate of chip temperature / external temperature is lower than the rate that effectively determines the cooling efficiency.
- uoaei 4y ago
  One single company changing one single design parameter and enabling savings on the scale of multiple power plants? That is as immediate as wins get.
- Spooky23 4y ago
  For one company? That’s pretty impressive. When I was on an architecture team that consolidated ~80 datacenter to 3 circa 2010, this was a key dollar driver. We raised the temperature ~6 degrees from the average temp, which meant kicking out a few vendors initially. The cost savings for doing that was essentially the total operational costs of 5 datacenters. The annual failure rates for the hardware did not change at all by any metric. Number of service impacting hardware failures went to zero due to the consolidation. In general, if you operate within the operating ranges of your hardware, you won’t have failure. You will have complaints from employees, because computers will operate at temperatures not comfortable for humans.
  - sllabres 4y ago
    We did the same several years ago. At the time I found http://www.cs.toronto.edu/~bianca/papers/temperature_cam.pdf http://www.cs.toronto.edu/~bianca/papers/temperature_cam.pdf quite interesting. I didn't find many other precise papers about issues when running at a higher temperatures, but many "should" and "can". Based on ASHARE there are two guidelines from HPE and IBM: https://www.chiltrix.com/documents/HP-ASHRAE.pdf https://www.chiltrix.com/documents/HP-ASHRAE.pdf https://www.ibm.com/downloads/cas/1Q94RPGE https://www.ibm.com/downloads/cas/1Q94RPGE We found (by measurement) that some places in the datacenter with suboptimal airflow are well over the average or simulated temperature so one can leave the safe temperature envelope if one isn't careful.
    - Spooky23 4y ago
      The beauty of a big project like is you get the engineering resources to lake sure the datacenter is working right. The hyper scale people take this to the next degree.
  - benlivengood 4y ago
    It's almost certainly Google, since they've historically ran their data centers hotter than most [0]. Cooling efficiency increases with a higher delta-T to the working fluid, and Google uses a gigawatt or two continuously [1]. From PUE numbers that's hundreds of MW spent on cooling, so making it more efficient is quite worth it. [0] https://www.google.com/about/datacenters/efficiency/ https://www.google.com/about/datacenters/efficiency/ [1] https://www.cnbc.com/2022/04/13/google-data-center-goal-100percent-green-energy-by-2030.html https://www.cnbc.com/2022/04/13/google-data-center-goal-100p...
JJMcJ 4y ago
I've heard of some large companies that run their data centers hot. Cheaper to have a slightly higher failure rate, or have the computers throttle their clock speed, than to pay for extra air conditioning.
- klysm 4y ago
  Hard to do the math here though because you don't know the failure statistics in advance. Kind of a multi-armed bandit problem of sorts.
  - magicalhippo 4y ago
    Wouldn't the Arrhenius equation[1] be a good approximation? It's used in the industry[2] from what I know. Of course you'll need some data for calibrating the model, but if you got that? [1]: https://en.wikipedia.org/wiki/Arrhenius_equation https://en.wikipedia.org/wiki/Arrhenius_equation [2]: https://www.ti.com/lit/an/snva509a/snva509a.pdf https://www.ti.com/lit/an/snva509a/snva509a.pdf
    - buescher 4y ago
      Well, sometimes, and more frequently now at really small process nodes, but mostly no. Most electronic failures are basically mechanical and thermal cycling will cause fatigue failures more than elevated temperatures will accelerate things like electromigration. Lots of people still use 1960s style handbook methods anyway because there's no plug-and-chug replacement. The groundbreaking work here was by Michael Pecht back in the early nineties: https://apps.dtic.mil/sti/pdfs/ADA275029.pdf https://apps.dtic.mil/sti/pdfs/ADA275029.pdf
- woleium 4y ago
  Could also be more likely that it's cheaper to extend the life of an older DC by accepting higher temperatures and failure rates than to upgrade the hvac to accommodate newer higher density designs
  - 4y ago

gjsman-1000 4y ago

Only three? That's not an immediate win. What if the temperature increase causes just so slightly more failures, causing so slightly more replacements, and each replacement requires energy to make, ship, install, replace, recycle, the effects of increased demand... What if it lasts slightly less long, causing more early failures and eWaste? After all that potential risk, is it a benefit still, and if so, how much? We don't know and it is hard to know - but I don't blame JEDEC and would not call it a "compromise" on their part like it was a superior option.

picture 4y ago
Cmon.. logically it has to be more than just three right? "Cooling efficiency" sometimes come in unit of W/°C difference, so I'd imagine that a few more degrees would be a huge deal.
- gjsman-1000 4y ago
  I don't know how much power efficiency would be saved - my concern is more that it is completely logical that running any part at higher temperatures causes increased risk of failure, whether it be a computer part or a mechanical part. How much? I don't know - I just don't blame JEDEC for recognizing this is not a clear and obvious win. Imagine if the failure rate was raised by as little as 1%. RAM Failure is not uncommon compared to other parts - I've had it happen before and render a system unable to boot, that's why we have Memtest86 and not CPUtest86 or SSDtest86. A 1% increase in failure over 5 years would have effects just as unbelievable as the power saving that increasing the temperature would be. How many smartphones would be junked? How many PCs would be thrown out for not working by people who are average Joes who can't diagnose them, and the extra waste that generates from both disposing the old PC and purchasing a new one? Perhaps the new PC is more efficient, but which is better, greater emissions or more eWaste in the ground due to the new PC being likely more efficient than the old one? The point is that it is not a clear win. With further research it might be, and I might be all for it. I'm only nitpicking the description of it as being a "compromise" as though it were obvious. [@picture: I'm at my posting limit for the day because HN is, well... I'll leave their censorship policies for another day. I would agree with you if the RAM with the 90C limit were strictly ECC RAM because that is most often used in data centers and not consumer parts. Maybe we have non-ECC/85 RAM and ECC/90 RAM options...]
  - picture 4y ago
    That's for sure. I 100% agree that increased temperature will statistically increase failure rate. I'm just thinking that, the most common mechanisms of thermal failure in electronics are caused by repeated thermal cycling which cause fatigue and stress failures at interconnects (solder bumps, silicon bonding, etc). Data centers are designed to be operated in a relatively very constant temperature environment, so I would suspect that the failure rate may not be raised significantly.
  - dcow 4y ago
    Well now you're just being hyperbolic. As you say, this is an engineering problem so solutions are far from their ideals states in either direction. However, 1% increase in ram failure rates ruining the world? That doesn't sound right. Errors are encountered in RAM all the time and guess what, they're corrected often by the hardware before even bothering the system. I'm sure we could deal with a 1% increase...
  - Spooky23 4y ago
    Most datacenter hardware is fine at 95 degrees F (inlet temp). Approved configurations are usually available to 105 degrees F or slightly higher. Some devices can run as high as 130F. In the operating range, you’re not going to have any measurable change in operations or failure rate - if you do the parts are defective. All of the stories you hear about this and that are conjecture.
    - smolder 4y ago
      Interestingly, computer chips can often be run at lower voltage and wattage for a given frequency if they are kept at a colder temperature. As a home user I can significantly reduce power draw for a CPU/GPU by improving the cooling solution and lowering voltages. The reasons this doesn't work for datacenters is two-fold, I think: First, they won't see efficiency improvements just by keeping their CPUs and GPUs (or RAM) cooler because the power levels/tables for the chips are baked-in, and operators aren't going to the trouble of tweaking voltages themselves. Second, even if they did tweak voltages, the cost of sustaining lower temperatures with better cooling likely won't outweigh the savings resulting from lower power draw for the chips. Still, this raises the question of whether designing hardware for higher operating temperatures is always the right move. At some point there's going to be a cost in performance and/or efficiency that outweighs the savings from allowing higher temperatures. Ideally these tradeoffs should be balanced as a whole.
- marcosdumay 4y ago
  You usually have to budget a few °C of difference just for pushing enough energy through heat exchangers. So the rate of chip temperature / external temperature is lower than the rate that effectively determines the cooling efficiency.
uoaei 4y ago
One single company changing one single design parameter and enabling savings on the scale of multiple power plants? That is as immediate as wins get.
Spooky23 4y ago
For one company? That’s pretty impressive. When I was on an architecture team that consolidated ~80 datacenter to 3 circa 2010, this was a key dollar driver. We raised the temperature ~6 degrees from the average temp, which meant kicking out a few vendors initially. The cost savings for doing that was essentially the total operational costs of 5 datacenters. The annual failure rates for the hardware did not change at all by any metric. Number of service impacting hardware failures went to zero due to the consolidation. In general, if you operate within the operating ranges of your hardware, you won’t have failure. You will have complaints from employees, because computers will operate at temperatures not comfortable for humans.
- sllabres 4y ago
  We did the same several years ago. At the time I found http://www.cs.toronto.edu/~bianca/papers/temperature_cam.pdf http://www.cs.toronto.edu/~bianca/papers/temperature_cam.pdf quite interesting. I didn't find many other precise papers about issues when running at a higher temperatures, but many "should" and "can". Based on ASHARE there are two guidelines from HPE and IBM: https://www.chiltrix.com/documents/HP-ASHRAE.pdf https://www.chiltrix.com/documents/HP-ASHRAE.pdf https://www.ibm.com/downloads/cas/1Q94RPGE https://www.ibm.com/downloads/cas/1Q94RPGE We found (by measurement) that some places in the datacenter with suboptimal airflow are well over the average or simulated temperature so one can leave the safe temperature envelope if one isn't careful.
  - Spooky23 4y ago
    The beauty of a big project like is you get the engineering resources to lake sure the datacenter is working right. The hyper scale people take this to the next degree.
- benlivengood 4y ago
  It's almost certainly Google, since they've historically ran their data centers hotter than most [0]. Cooling efficiency increases with a higher delta-T to the working fluid, and Google uses a gigawatt or two continuously [1]. From PUE numbers that's hundreds of MW spent on cooling, so making it more efficient is quite worth it. [0] https://www.google.com/about/datacenters/efficiency/ https://www.google.com/about/datacenters/efficiency/ [1] https://www.cnbc.com/2022/04/13/google-data-center-goal-100percent-green-energy-by-2030.html https://www.cnbc.com/2022/04/13/google-data-center-goal-100p...

JJMcJ 4y ago

I've heard of some large companies that run their data centers hot. Cheaper to have a slightly higher failure rate, or have the computers throttle their clock speed, than to pay for extra air conditioning.

klysm 4y ago
Hard to do the math here though because you don't know the failure statistics in advance. Kind of a multi-armed bandit problem of sorts.
- magicalhippo 4y ago
  Wouldn't the Arrhenius equation[1] be a good approximation? It's used in the industry[2] from what I know. Of course you'll need some data for calibrating the model, but if you got that? [1]: https://en.wikipedia.org/wiki/Arrhenius_equation https://en.wikipedia.org/wiki/Arrhenius_equation [2]: https://www.ti.com/lit/an/snva509a/snva509a.pdf https://www.ti.com/lit/an/snva509a/snva509a.pdf
  - buescher 4y ago
    Well, sometimes, and more frequently now at really small process nodes, but mostly no. Most electronic failures are basically mechanical and thermal cycling will cause fatigue failures more than elevated temperatures will accelerate things like electromigration. Lots of people still use 1960s style handbook methods anyway because there's no plug-and-chug replacement. The groundbreaking work here was by Michael Pecht back in the early nineties: https://apps.dtic.mil/sti/pdfs/ADA275029.pdf https://apps.dtic.mil/sti/pdfs/ADA275029.pdf
woleium 4y ago
Could also be more likely that it's cheaper to extend the life of an older DC by accepting higher temperatures and failure rates than to upgrade the hvac to accommodate newer higher density designs
- 4y ago