4 ms·

So every time they plug in a server they also plug in water lines?

by ambicapter 1y ago

jedberg 1y ago
Looks like it. New server means power, internet, and water.
- fudgy73 1y ago
  just like humans.
liquidgecka 1y ago
[I am not a current Google Employee so my understanding of this is based on externally written articles and “leap of faith” guestimation] Yes. A supply and return line along with power. Though if I had to guess how its setup this would be done with some super slick “it just works” kind of mount that lets them just slide the case in and lock it in place. When I was there almost all hardware replacement was made downright trivial so it could just be more or less slide in place and walk away.
- nielsbot 1y ago
  Maybe similar to a gasoline hose breakaway https://www.opwglobal.com/products/us/retail-fueling-products/standard-dispensing-equipment/swivels-and-breakaways/reconnectable-breakaways https://www.opwglobal.com/products/us/retail-fueling-product...
- michaelt 1y ago
  Interestingly, entire supercomputers have been decommissioned [1] due to faulty quick disconnects causing water spray. So you can get a single, blind-mating connector combining power, data and water - but you might not want to :) [1] https://gsaauctions.gov/auctions/preview/282996 https://gsaauctions.gov/auctions/preview/282996
- semi-extrinsic 1y ago
  https://amphenol-industrial.com/products/liquid-cooling-systems/ https://amphenol-industrial.com/products/liquid-cooling-syst...
  - tesseract 1y ago
    CPC is in this market too https://www.cpcworldwide.com/Liquid-Cooling/Products/Blind-Mate https://www.cpcworldwide.com/Liquid-Cooling/Products/Blind-M... Non-spill fluid quick disconnects are established tech in industries like medical, chemical processing, beverage dispensing, and hydraulic power, so there are plenty of design concepts to draw on.
- scrlk 1y ago
  You can see the male quick disconnect fittings for the liquid cooling at each corner of the server in this photo: https://substackcdn.com/image/fetch/$s_!8aMm!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbe62fbd-2950-4584-b597-ccfbabd48bf6_2000x1331.jpeg https://substackcdn.com/image/fetch/$s_!8aMm!,f_auto,q_auto:... Looks like the power connector is in the centre. I'm not sure if backplane connectors are covered up by orange plugs?
  - Romario77 1y ago
    it's not the center one, it's the side ones. Center seems to be a power supply.
ajb 1y ago
I remember reading somewhere that they don't operate at the level of servers; if one dies they leave it in place until they're ready to replace the whole rack. Don't know if that's true now, though. It does sound like connections do involve water lines though. As they are isolating different water circuits, in theory they could have a dry connection between heat exchanger plates, or one made through thermal paste. It doesn't sound like they're doing that though.
- cavisne 1y ago
  Definitely not true for these workloads. TPUs are interconnected, one dying makes the whole cluster significantly less useful.
- liquidgecka 1y ago
  It has not been true for a LONG time. That was part of Google early “compute unit” strategy that involved things like sealed containers and such. Turns out that’s not super efficient or useful because you leave large swaths of hardware idle. In my day we had software that would “drain” a machine and release it to hardware ops to swap the hardware on. This could be a drive, memory, CPU or a motherboard. If it was even slightly complicated they would ship it to Mountain View for diagnostic and repair. But every machine was expected to be cycled to get it working as fast as possible. We did a disk upgrade on a whole datacenter that involved switching from 1TB to 2TB disks or something like that (I am dating myself) and total downtime was so important they hired temporary workers to work nights to get the swap done as quickly as possible. If I remember correctly that was part of the “holy cow gmail is out of space!” chaos though, so added urgency.
  - throwaway2037 1y ago
    > part of the “holy cow gmail is out of space!” chaos This sounds like an interesting story. Can you share more details.
  - Cthulhu_ 1y ago
    I'd love to work in a datacenter at that scale sometime, it sounds like it's like working in a warehouse where you get a list of orders, servers to remove and pick up, but at the scales of the Googles et al, that's hundreds of server replacements a day, and production lines of new servers being built and existing ones being repaired or decommissioned. It's a fascinating industry, but only in my head as the only info you get about it is carefully polished articles and the occasional anecdote on HN, which is also carefully polished due to NDAs.
jayd16 1y ago
Maybe we can declutter things if they get PWoE(power and water over ethernet) or just a USB-W standard.
- Nition 1y ago
  It worked for MONIAC.
- taneq 1y ago
  Great, now I’ll have to figure out if my USB Hi-Flow cable is slowing down my USB Full-Flow drink bottle refilling.
  - toast0 1y ago
    It's not. Full flow is less than hi-flow... Your bottle might slow down other fills on the bus though :p
- Cthulhu_ 1y ago
  Related, I vaguely recall a concept from a Tesla charger for semis that proposed a charging cable with active coolant flowing through it as well to keep the wire cooled. edit: https://www.teslarati.com/tesla-liquid-cooled-supercharger-cables/ https://www.teslarati.com/tesla-liquid-cooled-supercharger-c...
  - jayd16 1y ago
    This is why I risk jokes on HN. Normally they're frowned upon but it's worth it to find interesting tech that meets the eccentric idea.
Hilift 1y ago
And a 12V battery.

So every time they plug in a server they also plug in water lines?

jedberg 1y ago

Looks like it. New server means power, internet, and water.

fudgy73 1y ago
just like humans.

liquidgecka 1y ago

[I am not a current Google Employee so my understanding of this is based on externally written articles and “leap of faith” guestimation] Yes. A supply and return line along with power. Though if I had to guess how its setup this would be done with some super slick “it just works” kind of mount that lets them just slide the case in and lock it in place. When I was there almost all hardware replacement was made downright trivial so it could just be more or less slide in place and walk away.

nielsbot 1y ago
Maybe similar to a gasoline hose breakaway https://www.opwglobal.com/products/us/retail-fueling-products/standard-dispensing-equipment/swivels-and-breakaways/reconnectable-breakaways https://www.opwglobal.com/products/us/retail-fueling-product...
michaelt 1y ago
Interestingly, entire supercomputers have been decommissioned [1] due to faulty quick disconnects causing water spray. So you can get a single, blind-mating connector combining power, data and water - but you might not want to :) [1] https://gsaauctions.gov/auctions/preview/282996 https://gsaauctions.gov/auctions/preview/282996
semi-extrinsic 1y ago
https://amphenol-industrial.com/products/liquid-cooling-systems/ https://amphenol-industrial.com/products/liquid-cooling-syst...
- tesseract 1y ago
  CPC is in this market too https://www.cpcworldwide.com/Liquid-Cooling/Products/Blind-Mate https://www.cpcworldwide.com/Liquid-Cooling/Products/Blind-M... Non-spill fluid quick disconnects are established tech in industries like medical, chemical processing, beverage dispensing, and hydraulic power, so there are plenty of design concepts to draw on.
scrlk 1y ago
You can see the male quick disconnect fittings for the liquid cooling at each corner of the server in this photo: https://substackcdn.com/image/fetch/$s_!8aMm!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbe62fbd-2950-4584-b597-ccfbabd48bf6_2000x1331.jpeg https://substackcdn.com/image/fetch/$s_!8aMm!,f_auto,q_auto:... Looks like the power connector is in the centre. I'm not sure if backplane connectors are covered up by orange plugs?
- Romario77 1y ago
  it's not the center one, it's the side ones. Center seems to be a power supply.

ajb 1y ago

I remember reading somewhere that they don't operate at the level of servers; if one dies they leave it in place until they're ready to replace the whole rack. Don't know if that's true now, though. It does sound like connections do involve water lines though. As they are isolating different water circuits, in theory they could have a dry connection between heat exchanger plates, or one made through thermal paste. It doesn't sound like they're doing that though.

cavisne 1y ago
Definitely not true for these workloads. TPUs are interconnected, one dying makes the whole cluster significantly less useful.
liquidgecka 1y ago
It has not been true for a LONG time. That was part of Google early “compute unit” strategy that involved things like sealed containers and such. Turns out that’s not super efficient or useful because you leave large swaths of hardware idle. In my day we had software that would “drain” a machine and release it to hardware ops to swap the hardware on. This could be a drive, memory, CPU or a motherboard. If it was even slightly complicated they would ship it to Mountain View for diagnostic and repair. But every machine was expected to be cycled to get it working as fast as possible. We did a disk upgrade on a whole datacenter that involved switching from 1TB to 2TB disks or something like that (I am dating myself) and total downtime was so important they hired temporary workers to work nights to get the swap done as quickly as possible. If I remember correctly that was part of the “holy cow gmail is out of space!” chaos though, so added urgency.
- throwaway2037 1y ago
  > part of the “holy cow gmail is out of space!” chaos This sounds like an interesting story. Can you share more details.
- Cthulhu_ 1y ago
  I'd love to work in a datacenter at that scale sometime, it sounds like it's like working in a warehouse where you get a list of orders, servers to remove and pick up, but at the scales of the Googles et al, that's hundreds of server replacements a day, and production lines of new servers being built and existing ones being repaired or decommissioned. It's a fascinating industry, but only in my head as the only info you get about it is carefully polished articles and the occasional anecdote on HN, which is also carefully polished due to NDAs.

jayd16 1y ago

Maybe we can declutter things if they get PWoE(power and water over ethernet) or just a USB-W standard.

Nition 1y ago
It worked for MONIAC.
taneq 1y ago
Great, now I’ll have to figure out if my USB Hi-Flow cable is slowing down my USB Full-Flow drink bottle refilling.
- toast0 1y ago
  It's not. Full flow is less than hi-flow... Your bottle might slow down other fills on the bus though :p
Cthulhu_ 1y ago
Related, I vaguely recall a concept from a Tesla charger for semis that proposed a charging cable with active coolant flowing through it as well to keep the wire cooled. edit: https://www.teslarati.com/tesla-liquid-cooled-supercharger-cables/ https://www.teslarati.com/tesla-liquid-cooled-supercharger-c...
- jayd16 1y ago
  This is why I risk jokes on HN. Normally they're frowned upon but it's worth it to find interesting tech that meets the eccentric idea.

Hilift 1y ago

And a 12V battery.