3 ms·

It's very interesting to see this. I wonder what Amazon, Facebook, Google, and Microsoft are going to do in the next few generations. Will they be moving to 2U1

by aftbit 3y ago

It's very interesting to see this. I wonder what Amazon, Facebook, Google, and Microsoft are going to do in the next few generations. Will they be moving to 2U1N as well? Or do they have liquid cooling in mind? Or maybe they'll be keeping lower power budgets? Or building data centers that run cooler and support higher power density (probably not great for the environment). I knew fan power usage could be a substantial amount of a server's power draw, but I didn't realize they would need 320W to move enough air to cool 500W of dissapation.

nagisa 3y ago
The major cloud vendors have their own data centers & do not need to deal with space or power limits at a rack unit. They have the entire data center worth of space and power supply to work with.
- jonatron 3y ago
  If the data centre is owned by digital reality, equinix, telehouse, etc, is it really theirs? https://aws.amazon.com/directconnect/locations/ https://aws.amazon.com/directconnect/locations/
  - arrakeenrevived 3y ago
    Direct Connect locations are not the locations where the main region data centers are. AWS has a network link from their DC to Equinix, etc to make it easier for customers who are in Equinix's colo to make a direct connect to the AWS network.
- toast0 3y ago
  All data centers have space and power/cooling limits. If you're in someone else's DC, they are usually going to quote you as per-rack power, but if you really want you can probably convince them to let you use 2 racks worth of power in one rack and hold a second rack empty. In owned and operated DCs, it's kind of similar, if the DC was built a while ago, as server power density has increased, a significant portion of the floor space may be unusable for racks. 10-20 years ago, 1U and blades made a lot of sense because space was a major constraint; now 2U or larger can make sense because 1U has a lot of tradeoffs for size, and power is the bigger constraint. If space density is still needed, 2 nodes in 2u or similar narrow but tall configurations provide a more square profile that allows generating airflow with larger fans which is more efficient than the 40mm fans needed in 1U systems.
folmar 3y ago
OVH is doing liquid cooling to some extent since 2003. https://blog.ovhcloud.com/water-cooling-from-innovation-to-disruption-part-i/ https://blog.ovhcloud.com/water-cooling-from-innovation-to-d...
infogulch 3y ago
Yes it's insane to waste 70% of the CPU power again on cooling the CPU (read: cutting into power density). Oxide's rack uses 80mm fans that run at less than half the stock minimum speed to reduce this waste and run super quiet: https://oxide.computer/blog/the-cloud-computer https://oxide.computer/blog/the-cloud-computer
mcronce 3y ago
It's nuts how much energy server fans can use. I recently picked up an R730xd. By default, a non-Dell-branded PCI-E device will force the fans to ~16000RPM, but there's an IPMI command to disable that and go back to regular, temperature-driven, fan control. Total server power came down by 70W after issuing that command.
zozbot234 3y ago
The latest Zen4c-based EPYC chips are reaching very high power densities. They're reportedly targeted towards hyperscalers that are badly cramped for space in their datacenters but can be more flexible wrt. power budgets. In a way, these concerns had to be there already with HPC hardware, so there's probably well-established approaches for dealing with them too.
donavanm 3y ago
This is very uninteresting and way behind the big cloud vendors high density installs. Theyre just saying “when power/cooling limited use a form factor that optimizes your remaining constraints, like RU.” So yes, 2p & 3p edge sites are commonly limited to 5-10kva, and you end up with low density racks and “extra” physical space you can use. In the modern dense 1p deployments youll see 20kva, or more, per position. The physical form factor always follows those constraints based on cooling/power/connectivity/weight/etc.
- bradknowles 3y ago
  I thought current rack standard installs were like 40kW per rack, with high end all-AI type racks being like 100kW each? Where the heck are you seeing 5-10kva?
- wmf 3y ago
  Even if you had unlimited power per rack you wouldn't want to waste it on fans.

by aftbit 3y ago

nagisa 3y ago

The major cloud vendors have their own data centers & do not need to deal with space or power limits at a rack unit. They have the entire data center worth of space and power supply to work with.

jonatron 3y ago
If the data centre is owned by digital reality, equinix, telehouse, etc, is it really theirs? https://aws.amazon.com/directconnect/locations/ https://aws.amazon.com/directconnect/locations/
- arrakeenrevived 3y ago
  Direct Connect locations are not the locations where the main region data centers are. AWS has a network link from their DC to Equinix, etc to make it easier for customers who are in Equinix's colo to make a direct connect to the AWS network.
toast0 3y ago
All data centers have space and power/cooling limits. If you're in someone else's DC, they are usually going to quote you as per-rack power, but if you really want you can probably convince them to let you use 2 racks worth of power in one rack and hold a second rack empty. In owned and operated DCs, it's kind of similar, if the DC was built a while ago, as server power density has increased, a significant portion of the floor space may be unusable for racks. 10-20 years ago, 1U and blades made a lot of sense because space was a major constraint; now 2U or larger can make sense because 1U has a lot of tradeoffs for size, and power is the bigger constraint. If space density is still needed, 2 nodes in 2u or similar narrow but tall configurations provide a more square profile that allows generating airflow with larger fans which is more efficient than the 40mm fans needed in 1U systems.

folmar 3y ago

OVH is doing liquid cooling to some extent since 2003. https://blog.ovhcloud.com/water-cooling-from-innovation-to-disruption-part-i/ https://blog.ovhcloud.com/water-cooling-from-innovation-to-d...

infogulch 3y ago

Yes it's insane to waste 70% of the CPU power again on cooling the CPU (read: cutting into power density). Oxide's rack uses 80mm fans that run at less than half the stock minimum speed to reduce this waste and run super quiet: https://oxide.computer/blog/the-cloud-computer https://oxide.computer/blog/the-cloud-computer

mcronce 3y ago

It's nuts how much energy server fans can use. I recently picked up an R730xd. By default, a non-Dell-branded PCI-E device will force the fans to ~16000RPM, but there's an IPMI command to disable that and go back to regular, temperature-driven, fan control. Total server power came down by 70W after issuing that command.

zozbot234 3y ago

The latest Zen4c-based EPYC chips are reaching very high power densities. They're reportedly targeted towards hyperscalers that are badly cramped for space in their datacenters but can be more flexible wrt. power budgets. In a way, these concerns had to be there already with HPC hardware, so there's probably well-established approaches for dealing with them too.

donavanm 3y ago

This is very uninteresting and way behind the big cloud vendors high density installs. Theyre just saying “when power/cooling limited use a form factor that optimizes your remaining constraints, like RU.” So yes, 2p & 3p edge sites are commonly limited to 5-10kva, and you end up with low density racks and “extra” physical space you can use. In the modern dense 1p deployments youll see 20kva, or more, per position. The physical form factor always follows those constraints based on cooling/power/connectivity/weight/etc.

bradknowles 3y ago
I thought current rack standard installs were like 40kW per rack, with high end all-AI type racks being like 100kW each? Where the heck are you seeing 5-10kva?
wmf 3y ago
Even if you had unlimited power per rack you wouldn't want to waste it on fans.