5 ms·

I'd disagree, the fairly newish DC we're in is designed for supplying and cooling 20kW/rack. That's enough for 40 or so 1u servers burning ~450 watts.

sulam 10y ago
Hmm, good to know. And to update another assumption-- my guess is that this would be a pretty expensive configuration, and you may be better off picking a lower TDP server that can still get the job done, yes? Also, do you actually have room for 40 nodes in a rack? Betwee TOR switches, storage, and miscellaneous other, it seems unlikely.
- shiftpgdn 10y ago
  I would think 40 nodes in a rack is realistic if your battery is located elsewhere and you use side mount PDUs. Keep in mind in many places (NYC is a great example here) that real estate space for racks is the killer and not cooling or power.
dfox 10y ago
One thing that burned us in the past was DC that provided 20kW power per rack in power, but required you to only dissipate 5kW in heat. Needless to say we changed the DC provider in like a month after they made this clear (and they combined that with offer of some kind of "cloud provider" package which meant 15kW of both and triple the price). After few another such DC-related incidents (and mostly unbelievable: circuit breakers catching fire, 600V peak on 230V AC line, double flooring collapsing from overload and such) I'm quite relieved that I don't deal with DC procurement anymore.
- mikeash 10y ago
  What the...? Were they expecting you to radiate the remaining 15kW as microwaves or something?
  - dfox 10y ago
    That's exactly the question I asked them, with "or you want us to radiate the remaining the remaining 15kW into this single mode fiber going to your switch? we can get laser capable of that"
    - pierrebai 10y ago
      Does the wattage coming in entirely converts to heat? I always assumed it didn't, but have no idea in what proportion.
      - mikeash 10y ago
        Energy in has to equal energy out over the long term. For electronics equipment that isn't doing mechanical work then there aren't a lot of options for how that energy can come out: it'll either be electromagnetic radiation of some kind (i.e. light or radio or similar) or heat. Most computer equipment emits relatively little electromagnetic radiation, so it's essentially all heat.
        0xffff2 10y ago
        Isn't heat just lower frequency EM radiation?
        mikeash 10y ago
        Heat can be transmitted through EM radiation (of all frequencies, but IR is predominant for temperatures we're accustomed to) but also by direct contact between materials. For typical temperatures, radiation doesn't transmit much heat. Vacuum makes for a pretty good insulator. If you want to remove 20kW of heat from a small space, you'll need to do most of that removal by transferring the heat to a fluid of some kind, e.g. by putting it into the air. To see the difference in practical terms, pick an item that's noticeably warm, but not hot enough to burn you. Hold your finger very close to it. The heat you feel there is radiated. Then touch it, and feel the heat that's conducted through touch. You'll feel much more heat with the latter.
deegles 10y ago
250 KW per rack is possible using immersion cooling: http://investors.3m.com/news/press-release-details/2015/3M-Novec-7100-Fluid-Used-in-Worlds-Largest-Two-Phase-Immersion-Cooling-Project/default.aspx http://investors.3m.com/news/press-release-details/2015/3M-N...
- wmf 10y ago
  Just not in a real datacenter.

sulam 10y ago

Hmm, good to know. And to update another assumption-- my guess is that this would be a pretty expensive configuration, and you may be better off picking a lower TDP server that can still get the job done, yes? Also, do you actually have room for 40 nodes in a rack? Betwee TOR switches, storage, and miscellaneous other, it seems unlikely.

shiftpgdn 10y ago
I would think 40 nodes in a rack is realistic if your battery is located elsewhere and you use side mount PDUs. Keep in mind in many places (NYC is a great example here) that real estate space for racks is the killer and not cooling or power.

dfox 10y ago

One thing that burned us in the past was DC that provided 20kW power per rack in power, but required you to only dissipate 5kW in heat. Needless to say we changed the DC provider in like a month after they made this clear (and they combined that with offer of some kind of "cloud provider" package which meant 15kW of both and triple the price). After few another such DC-related incidents (and mostly unbelievable: circuit breakers catching fire, 600V peak on 230V AC line, double flooring collapsing from overload and such) I'm quite relieved that I don't deal with DC procurement anymore.

mikeash 10y ago
What the...? Were they expecting you to radiate the remaining 15kW as microwaves or something?
- dfox 10y ago
  That's exactly the question I asked them, with "or you want us to radiate the remaining the remaining 15kW into this single mode fiber going to your switch? we can get laser capable of that"
  - pierrebai 10y ago
    Does the wattage coming in entirely converts to heat? I always assumed it didn't, but have no idea in what proportion.
    - mikeash 10y ago
      Energy in has to equal energy out over the long term. For electronics equipment that isn't doing mechanical work then there aren't a lot of options for how that energy can come out: it'll either be electromagnetic radiation of some kind (i.e. light or radio or similar) or heat. Most computer equipment emits relatively little electromagnetic radiation, so it's essentially all heat.
      - 0xffff2 10y ago
        Isn't heat just lower frequency EM radiation?
        mikeash 10y ago
        Heat can be transmitted through EM radiation (of all frequencies, but IR is predominant for temperatures we're accustomed to) but also by direct contact between materials. For typical temperatures, radiation doesn't transmit much heat. Vacuum makes for a pretty good insulator. If you want to remove 20kW of heat from a small space, you'll need to do most of that removal by transferring the heat to a fluid of some kind, e.g. by putting it into the air. To see the difference in practical terms, pick an item that's noticeably warm, but not hot enough to burn you. Hold your finger very close to it. The heat you feel there is radiated. Then touch it, and feel the heat that's conducted through touch. You'll feel much more heat with the latter.

deegles 10y ago

250 KW per rack is possible using immersion cooling: http://investors.3m.com/news/press-release-details/2015/3M-Novec-7100-Fluid-Used-in-Worlds-Largest-Two-Phase-Immersion-Cooling-Project/default.aspx http://investors.3m.com/news/press-release-details/2015/3M-N...

wmf 10y ago
Just not in a real datacenter.