5 ms·

For most purposes, electricity is almost a rounding error in datacenter asset management concerns. Server compute is expensive. It is all about performance. ARM

For most purposes, electricity is almost a rounding error in datacenter asset management concerns. Server compute is expensive. It is all about performance. ARM is just starting to get competitive in that ballpark. So we will see. All of these ARM server chips are going to be very low on the pecking order for TSMC fab time, so they won't have the edge that Apple had by buying up the first slot on the latest manufacturing node. Even being a process node ahead, M1 only just squeaks by with comparable single threaded performance to Zen 3 cores, so ARM still has some catching up to do in the performance realm.

hencoappel 5y ago
As far as I'm aware, efficiency is key for datacenters. It can lower direct electricity costs, but also the cost of cooling. I've always been under the impression that the cost of running is much more than the cost of the hardware which is why they are willing to pay so much for the best/most efficient hardware?
tdrdt 5y ago
"For most purposes, electricity is almost a rounding error in datacenter asset management concerns." I don't think this is true. It is not only power consumption but also power backup. If you can use smaller diesel engines and smaller battery packs this will lower the cost. And why do you think datacenters have been raising temperatures? It saves a ton of money: https://www.datacenterknowledge.com/archives/2008/10/14/google-raise-your-data-center-temperature https://www.datacenterknowledge.com/archives/2008/10/14/goog... HP estimated they saved 8 million. That is not a rounding error. A server that uses less power will also generate less heat. I have to admit it is years ago I worked for a company owning datacenters but at that time the highest costs were always: power and connectivity.
- easygenes 5y ago
  That article cites a 100,000 sq. ft. data-center as the source for that $8 million savings estimate, along with roughly a 30% power savings figure. No mention on the period, so presumably it would be a TCO for the servers going into the initial buildout. 100,000 sq. ft. is roughly enough space for about 250,000 1U servers. Say conservatively the servers are in the ballpark of $2,000 each. That's $500 million just in server hardware costs. If the total electric cost is around 3*$8 = ~$24 million over their lifetime, then we're talking about <5% of just the server expenses. Never mind the facility and staff costs (and, as you've said, connectivity). So maybe not a rounding error, but way down the priorities list.
  - deleted 5y ago
    [deleted]
  - klelatti 5y ago
    These are AC costs not total costs and the $8m is probably an annual figure - although it's not clear.
    - easygenes 5y ago
      Yeah, unfortunately it's a bit vague on the period. It is not unreasonable to assume an annual figure, in which case you're looking at something closer to 20% of the server costs for A/C power (assuming an average life of about 4 years). Problem is, that sounds high... the coefficient of performance for chillers is around 4 to 7 [1]. That puts ~15-30% of the total energy demand from chillers, though often datacenters do budget a factor up to about 60% of equipment power demand for cooling power demand. Sum energy related expenses for datacenters tends to be around 10-15% of the total costs [2]. So it would be odd to have such high cooling costs. A TCO over 4 years seems more in line with typical figures. 1: https://www.energy.gov.au/sites/default/files/hvac-factsheet-chiller-efficiency.pdf 2: https://www.missioncriticalmagazine.com/ext/resources/MC/Home/Files/PDFs/(TUI3011B)SimpleModelDetermingTrueTCO.pdf
oefrha 5y ago
Are you speaking from experience doing data center asset management or are you just speculating? I don’t know about data centers, but for colocation, electricity cost is far from a rounding error. Also, cooling is a major factor that is directly related to power consumption, as pointed out by siblings.
- easygenes 5y ago
  See [1], figure 1 there is a fairly typical data center costs breakdown at the highest level for TCO. Total energy is by far the lowest, at around 10-15%. Site infrastructure, IT infrastructure, and staff are the real cost priorities. There's a caveat to that in the planning phase of a data center: A lot of the site infrastructure costs are a function of the total power requirement. So if - when building out a data center - you can get more power efficiency, then that does translate to a significant cost savings. Cooling tends to be somewhere between a 20 and 60% add on to the direct power consumption of a server. 1: https://www.missioncriticalmagazine.com/ext/resources/MC/Home/Files/PDFs/(TUI3011B)SimpleModelDetermingTrueTCO.pdf
iorrus 5y ago
In Ireland the national grid is under strain due to electricity usage from data centres, so much so that there is discussion to deny permission for new ones. I very much doubt that the electricity costs are negligible, it’s also one of the reason data centres are in Ireland as due to the relatively mild climate it reduces hearing/cooling costs. https://www.irishtimes.com/news/politics/data-centres-could-use-70-of-ireland-s-electricity-by-2030-committee-to-hear-1.4685589 https://www.irishtimes.com/news/politics/data-centres-could-...
- Symbiote 5y ago
  Isn't more of the reason political (taxes etc)? The same climate benefits apply to northern England and Scotland, but Great Britain's grid's capacity has around 10 times the capacity. (Ireland's grid's peak demand was 6.8GW, Great Britain's 63GW.) Although the same argument then applies for siting the datacentres in continental Europe, where the grid is around 667GW. Perhaps this is part of Facebook's reasoning for a second datacentre in Denmark. (Most of Denmark, including Facebook's existing datacentre, is connected to the continental European grid. The eastern islands are part of the Scandinavian grid.) [1] https://www.thelocal.dk/20211013/facebook-eyes-second-danish-data-centre-after-buying-large-plot/ https://www.thelocal.dk/20211013/facebook-eyes-second-danish...
- easygenes 5y ago
  Will be interesting to see where datacenters wind up in Europe this next decade. My money is on Norway. Norway seems poised to build out significantly. They're actively inviting datacenters [1]. They also seem to have lower electric costs for large businesses [2] vs Ireland [3]. FAANG are already populating the Nordics too [4]. 1: https://www.datacenterdynamics.com/en/news/norway-wants-data-centers-to-locate-there-and-be-more-sustainable-too/ 2: https://www.statista.com/statistics/595859/electricity-industry-price-norway/ 3: https://www.statista.com/statistics/595806/electricity-industry-price-ireland/ 4: https://www.zdnet.com/pictures/the-nordic-datacenter-boom/
Symbiote 5y ago
The average consumer cost for 1kWh of electricity in the EU is 20¢. A well-used server might consume 500W on average. units '500W * 1 year * 0.20 €/kWh' '€' € 876 I'm sure businesses get a discount, but we haven't paid for cooling yet. This is hardly a rounding error.
- ksec 5y ago
  >I'm sure businesses get a discount Not as large as some may expect once you factor in power delivery guarantee and redundancy. But basically yes. Every single HyperScaler has been pointing out electricity as one of their largest item on their TCO, 2nd only to hardware cost.
- anthk 5y ago
  Which units is that? Mine does return stuff in Euros but with a fraction.
  - Symbiote 5y ago
    GNU Units version 2.19 with readline, with utf8, locale en_IE and no customizations. units '500W * 1 year * 0.20/kWh' '1' would also work.
    - anthk 5y ago
      Do you have a custom ~/.units file? user@xirl>units '500W * 1 year * 0.20/kWh' '1' * 876.58128 / 0.0011407955

hencoappel 5y ago

As far as I'm aware, efficiency is key for datacenters. It can lower direct electricity costs, but also the cost of cooling. I've always been under the impression that the cost of running is much more than the cost of the hardware which is why they are willing to pay so much for the best/most efficient hardware?

tdrdt 5y ago

"For most purposes, electricity is almost a rounding error in datacenter asset management concerns." I don't think this is true. It is not only power consumption but also power backup. If you can use smaller diesel engines and smaller battery packs this will lower the cost. And why do you think datacenters have been raising temperatures? It saves a ton of money: https://www.datacenterknowledge.com/archives/2008/10/14/google-raise-your-data-center-temperature https://www.datacenterknowledge.com/archives/2008/10/14/goog... HP estimated they saved 8 million. That is not a rounding error. A server that uses less power will also generate less heat. I have to admit it is years ago I worked for a company owning datacenters but at that time the highest costs were always: power and connectivity.

easygenes 5y ago
That article cites a 100,000 sq. ft. data-center as the source for that $8 million savings estimate, along with roughly a 30% power savings figure. No mention on the period, so presumably it would be a TCO for the servers going into the initial buildout. 100,000 sq. ft. is roughly enough space for about 250,000 1U servers. Say conservatively the servers are in the ballpark of $2,000 each. That's $500 million just in server hardware costs. If the total electric cost is around 3*$8 = ~$24 million over their lifetime, then we're talking about <5% of just the server expenses. Never mind the facility and staff costs (and, as you've said, connectivity). So maybe not a rounding error, but way down the priorities list.
- deleted 5y ago
  [deleted]
- klelatti 5y ago
  These are AC costs not total costs and the $8m is probably an annual figure - although it's not clear.
  - easygenes 5y ago
    Yeah, unfortunately it's a bit vague on the period. It is not unreasonable to assume an annual figure, in which case you're looking at something closer to 20% of the server costs for A/C power (assuming an average life of about 4 years). Problem is, that sounds high... the coefficient of performance for chillers is around 4 to 7 [1]. That puts ~15-30% of the total energy demand from chillers, though often datacenters do budget a factor up to about 60% of equipment power demand for cooling power demand. Sum energy related expenses for datacenters tends to be around 10-15% of the total costs [2]. So it would be odd to have such high cooling costs. A TCO over 4 years seems more in line with typical figures. 1: https://www.energy.gov.au/sites/default/files/hvac-factsheet-chiller-efficiency.pdf 2: https://www.missioncriticalmagazine.com/ext/resources/MC/Home/Files/PDFs/(TUI3011B)SimpleModelDetermingTrueTCO.pdf

oefrha 5y ago

Are you speaking from experience doing data center asset management or are you just speculating? I don’t know about data centers, but for colocation, electricity cost is far from a rounding error. Also, cooling is a major factor that is directly related to power consumption, as pointed out by siblings.

easygenes 5y ago
See [1], figure 1 there is a fairly typical data center costs breakdown at the highest level for TCO. Total energy is by far the lowest, at around 10-15%. Site infrastructure, IT infrastructure, and staff are the real cost priorities. There's a caveat to that in the planning phase of a data center: A lot of the site infrastructure costs are a function of the total power requirement. So if - when building out a data center - you can get more power efficiency, then that does translate to a significant cost savings. Cooling tends to be somewhere between a 20 and 60% add on to the direct power consumption of a server. 1: https://www.missioncriticalmagazine.com/ext/resources/MC/Home/Files/PDFs/(TUI3011B)SimpleModelDetermingTrueTCO.pdf

iorrus 5y ago

In Ireland the national grid is under strain due to electricity usage from data centres, so much so that there is discussion to deny permission for new ones. I very much doubt that the electricity costs are negligible, it’s also one of the reason data centres are in Ireland as due to the relatively mild climate it reduces hearing/cooling costs. https://www.irishtimes.com/news/politics/data-centres-could-use-70-of-ireland-s-electricity-by-2030-committee-to-hear-1.4685589 https://www.irishtimes.com/news/politics/data-centres-could-...

Symbiote 5y ago
Isn't more of the reason political (taxes etc)? The same climate benefits apply to northern England and Scotland, but Great Britain's grid's capacity has around 10 times the capacity. (Ireland's grid's peak demand was 6.8GW, Great Britain's 63GW.) Although the same argument then applies for siting the datacentres in continental Europe, where the grid is around 667GW. Perhaps this is part of Facebook's reasoning for a second datacentre in Denmark. (Most of Denmark, including Facebook's existing datacentre, is connected to the continental European grid. The eastern islands are part of the Scandinavian grid.) [1] https://www.thelocal.dk/20211013/facebook-eyes-second-danish-data-centre-after-buying-large-plot/ https://www.thelocal.dk/20211013/facebook-eyes-second-danish...
easygenes 5y ago
Will be interesting to see where datacenters wind up in Europe this next decade. My money is on Norway. Norway seems poised to build out significantly. They're actively inviting datacenters [1]. They also seem to have lower electric costs for large businesses [2] vs Ireland [3]. FAANG are already populating the Nordics too [4]. 1: https://www.datacenterdynamics.com/en/news/norway-wants-data-centers-to-locate-there-and-be-more-sustainable-too/ 2: https://www.statista.com/statistics/595859/electricity-industry-price-norway/ 3: https://www.statista.com/statistics/595806/electricity-industry-price-ireland/ 4: https://www.zdnet.com/pictures/the-nordic-datacenter-boom/

Symbiote 5y ago

The average consumer cost for 1kWh of electricity in the EU is 20¢. A well-used server might consume 500W on average. units '500W * 1 year * 0.20 €/kWh' '€' € 876 I'm sure businesses get a discount, but we haven't paid for cooling yet. This is hardly a rounding error.

ksec 5y ago
>I'm sure businesses get a discount Not as large as some may expect once you factor in power delivery guarantee and redundancy. But basically yes. Every single HyperScaler has been pointing out electricity as one of their largest item on their TCO, 2nd only to hardware cost.
anthk 5y ago
Which units is that? Mine does return stuff in Euros but with a fraction.
- Symbiote 5y ago
  GNU Units version 2.19 with readline, with utf8, locale en_IE and no customizations. units '500W * 1 year * 0.20/kWh' '1' would also work.
  - anthk 5y ago
    Do you have a custom ~/.units file? user@xirl>units '500W * 1 year * 0.20/kWh' '1' * 876.58128 / 0.0011407955