5 ms·

Right? What temperatures would servers continue to operate at, but would be unsafe for humans?

uniformlyrandom 5y ago
The hardware is heat-resistant, so probably around ~75C (167F).
offmycloud 5y ago
If they did have a fire, they're probably waiting for all the smoke and FM200 (or other extinguishing agent) to clear out before they send anyone in.
JacobDotVI 5y ago
it might not be the fire itself, but the fire suppression system might uses gases fatal to humans and so they must wait for those gases to vent / dissipate.
lmilcin 5y ago
I was in a server room that overheated and servers started to shut down. AC failed and there was no environmental monitoring. The air was too hot to breathe. I had to take and hold breath to jump in to shut down most important assets. There was redundant air conditioning except for exactly one element: air inlet on the roof. A thin plastic bag blocked it.
- geoduck14 5y ago
  BRB. Going to go update my FMEA
- ta988 5y ago
  Was there some work after to add other inlets?
- jacquesm 5y ago
  Classic. $0.05 bit of junk causing a very large multiple of damage. Nice case to prove that plastic waste is hazardous though!
  - lmilcin 5y ago
    Yes, that plastic bag cost couple million. But the offset was definitely increased understanding of the concept of single point of failure.
    - rdines 5y ago
      oh my. that reminds me of this story that I hadn't thought of in a long time, LOL: https://www.datacenterknowledge.com/archives/2012/07/09/outages-surviving-electric-squirrels-ups-failures https://www.datacenterknowledge.com/archives/2012/07/09/outa...
      - jacquesm 5y ago
        That article references the explosion at 'The Planet' in 2008, I got caught up in that, that wasn't fun at all. Fortunately we had good off-site backups. Never knew about the squirrel angle, thank you!
        rdines 5y ago
        I forgot about that! Glad you came out ok in The Planet explosion, I remember that, it was insane. Fun fact: IBM now owns The Planet’s assets via Softlayer.
        jacquesm 5y ago
        By the way, we also were hit in 2003 when a transformer exploded. This may be a hint that you don't want to colocate where I stuff my machines ;) EV1 was pretty reliable right up to the point that it wasn't. Taught me some good lessons about multi-DC availability. Some of which were recently re-learned by the folks at OVH.
- serpix 5y ago
  so, well over 140 Celsius then as there are saunas that are that hot and breathing in them is fine.
  - thebeardisred 5y ago
    You're also not fully clothed and running around in a stress induced panic while in the sauna. Actually, I apologize that was a huge presumption. When _I_ am in a sauna I'm normally not fully clothed and running around in a stress induced panic.
  - hnlmorg 5y ago
    Presumably you mean Fahrenheit? 100 degrees C is the boiling point of water at sea level. I wouldn't want to sit in a room hotter than boiling water. Where as 140 degrees F is 60 degrees C and a little under twice your body's natural temperature, or a bit above a really hot Mediterranean summers day. Which seems a much more realistic temperature.
    - celticninja 5y ago
      60 degrees C is a lot above a really hot Mediterranean summer day. Average temp is 33 degrees in the hottest summer months, and in extremes it may be 40 degrees Celsius. 60 degrees would therefore be almost twice as hot as average or 50% hotter than even the hottest days. Also 60 degrees Celsius would mean it made an appearance here: https://en.m.wikipedia.org/wiki/Highest_temperature_recorded_on_Earth#:~:text=The%20current%20official%20highest%20registered,Valley%20in%20the%20United%20States https://en.m.wikipedia.org/wiki/Highest_temperature_recorded.... Given that the highest recorded air temperature on earth is 56.7 degrees, you may wish to revise your last sentence.
      - hnlmorg 5y ago
        > 60 degrees C is a lot above a really hot Mediterranean summer day. Average temp is 33 degrees in the hottest summer months, and in extremes it may be 40 degrees Celsius. I was thinking 40 degrees. Given "average" isn't implied by "really hot", I'd say your 40 degree figure falls in line with what I had in mind. But you're right that it's still a big jump to 60 degrees. Maybe I shouldn't have put vague comparisons in my post because it's rather distracting from the core point that the GP got their Celsius and Fahrenheit mixed up.
tyingq 5y ago
Pulling a spec sheet for a typical Dell 1U server, it has an operating range of 5°C to 40°C, and a non-operating (in storage) range of -40°C to 65°C. I imagine it would survive much higher temps, but it doesn't seem recommended. I would be wary of how reliable anything in that room might be later. Perhaps not the PCB itself, but there's typically lots of thin wire around with thin insulation.
- ta988 5y ago
  Insulation usually resist at must higher temperatures.
  - tyingq 5y ago
    It depends. Some insulation melts at 60-90C.
deleted 5y ago
[deleted]
redis_mlc 5y ago
If you can breathe, it should be ok. A lot of DCs after 2000 went from cold to warm based on the failure rates not changing. If you get a chance to tour HE Fremont (the old Apple factory), it's not cold. Factoid: most startups in SV started with a rack in a closet, that one day lost air circulation or cooling and went to the boiling point. I recommend people monitor the temp and cut 2 inches off the bottom and top of their closet door to allow passive cooling while you drive back to the office.
ta988 5y ago
I've had servers running in a 35°C room for almost a day. Curiously 3 years later only one hard drive died. And they are >6yo machines. Recent machines are supposed to handle 35°C air intake according to Anandtech and special servers can take more https://www.anandtech.com/show/7723/free-cooling-the-server-side-of-the-story/3 https://www.anandtech.com/show/7723/free-cooling-the-server-....

The hardware is heat-resistant, so probably around ~75C (167F).

If they did have a fire, they're probably waiting for all the smoke and FM200 (or other extinguishing agent) to clear out before they send anyone in.

JacobDotVI 5y ago

it might not be the fire itself, but the fire suppression system might uses gases fatal to humans and so they must wait for those gases to vent / dissipate.

lmilcin 5y ago

I was in a server room that overheated and servers started to shut down. AC failed and there was no environmental monitoring. The air was too hot to breathe. I had to take and hold breath to jump in to shut down most important assets. There was redundant air conditioning except for exactly one element: air inlet on the roof. A thin plastic bag blocked it.

geoduck14 5y ago
BRB. Going to go update my FMEA
ta988 5y ago
Was there some work after to add other inlets?
jacquesm 5y ago
Classic. $0.05 bit of junk causing a very large multiple of damage. Nice case to prove that plastic waste is hazardous though!
- lmilcin 5y ago
  Yes, that plastic bag cost couple million. But the offset was definitely increased understanding of the concept of single point of failure.
  - rdines 5y ago
    oh my. that reminds me of this story that I hadn't thought of in a long time, LOL: https://www.datacenterknowledge.com/archives/2012/07/09/outages-surviving-electric-squirrels-ups-failures https://www.datacenterknowledge.com/archives/2012/07/09/outa...
    - jacquesm 5y ago
      That article references the explosion at 'The Planet' in 2008, I got caught up in that, that wasn't fun at all. Fortunately we had good off-site backups. Never knew about the squirrel angle, thank you!
      - rdines 5y ago
        I forgot about that! Glad you came out ok in The Planet explosion, I remember that, it was insane. Fun fact: IBM now owns The Planet’s assets via Softlayer.
        jacquesm 5y ago
        By the way, we also were hit in 2003 when a transformer exploded. This may be a hint that you don't want to colocate where I stuff my machines ;) EV1 was pretty reliable right up to the point that it wasn't. Taught me some good lessons about multi-DC availability. Some of which were recently re-learned by the folks at OVH.
serpix 5y ago
so, well over 140 Celsius then as there are saunas that are that hot and breathing in them is fine.
- thebeardisred 5y ago
  You're also not fully clothed and running around in a stress induced panic while in the sauna. Actually, I apologize that was a huge presumption. When _I_ am in a sauna I'm normally not fully clothed and running around in a stress induced panic.
- hnlmorg 5y ago
  Presumably you mean Fahrenheit? 100 degrees C is the boiling point of water at sea level. I wouldn't want to sit in a room hotter than boiling water. Where as 140 degrees F is 60 degrees C and a little under twice your body's natural temperature, or a bit above a really hot Mediterranean summers day. Which seems a much more realistic temperature.
  - celticninja 5y ago
    60 degrees C is a lot above a really hot Mediterranean summer day. Average temp is 33 degrees in the hottest summer months, and in extremes it may be 40 degrees Celsius. 60 degrees would therefore be almost twice as hot as average or 50% hotter than even the hottest days. Also 60 degrees Celsius would mean it made an appearance here: https://en.m.wikipedia.org/wiki/Highest_temperature_recorded_on_Earth#:~:text=The%20current%20official%20highest%20registered,Valley%20in%20the%20United%20States https://en.m.wikipedia.org/wiki/Highest_temperature_recorded.... Given that the highest recorded air temperature on earth is 56.7 degrees, you may wish to revise your last sentence.
    - hnlmorg 5y ago
      > 60 degrees C is a lot above a really hot Mediterranean summer day. Average temp is 33 degrees in the hottest summer months, and in extremes it may be 40 degrees Celsius. I was thinking 40 degrees. Given "average" isn't implied by "really hot", I'd say your 40 degree figure falls in line with what I had in mind. But you're right that it's still a big jump to 60 degrees. Maybe I shouldn't have put vague comparisons in my post because it's rather distracting from the core point that the GP got their Celsius and Fahrenheit mixed up.

tyingq 5y ago

Pulling a spec sheet for a typical Dell 1U server, it has an operating range of 5°C to 40°C, and a non-operating (in storage) range of -40°C to 65°C. I imagine it would survive much higher temps, but it doesn't seem recommended. I would be wary of how reliable anything in that room might be later. Perhaps not the PCB itself, but there's typically lots of thin wire around with thin insulation.

ta988 5y ago
Insulation usually resist at must higher temperatures.
- tyingq 5y ago
  It depends. Some insulation melts at 60-90C.

deleted 5y ago

[deleted]

redis_mlc 5y ago

If you can breathe, it should be ok. A lot of DCs after 2000 went from cold to warm based on the failure rates not changing. If you get a chance to tour HE Fremont (the old Apple factory), it's not cold. Factoid: most startups in SV started with a rack in a closet, that one day lost air circulation or cooling and went to the boiling point. I recommend people monitor the temp and cut 2 inches off the bottom and top of their closet door to allow passive cooling while you drive back to the office.

ta988 5y ago

I've had servers running in a 35°C room for almost a day. Curiously 3 years later only one hard drive died. And they are >6yo machines. Recent machines are supposed to handle 35°C air intake according to Anandtech and special servers can take more https://www.anandtech.com/show/7723/free-cooling-the-server-side-of-the-story/3 https://www.anandtech.com/show/7723/free-cooling-the-server-....