I was winter. The #@! coldest winter I saw in a long time and I was on call. Sometime in the middle of the night, I received an SMS: "Datacenter temperature is

I was winter. The #@! coldest winter I saw in a long time and I was on call. Sometime in the middle of the night, I received an SMS: "Datacenter temperature is high: 24C" (yes, centigrade about 74F). I saw the graphs and I could see there was a temperature spike. In the last hour the datacenter went from 16C to 24C and it was going up. Three of the four industrial HVAC machines failed that night. Nobody knew why - I called the HVAC technician (he would come around 9am) and meanwhile I opened all windows and doors in the datacenter floor to stabilize temperature. 26C inside the DC, -6C outside. Turns out HVAC systems used water... and someone broke the pipe isolation and the water froze inside. It was fixed in the morning aplying a blowtorch to the 3 frozen pipes, then when we had water in the circuit instead of ice, HVAC machines made the DC cold again. Temperature peaked at 27C in the morning (-1C ouside) - at 28C (84F) the VP of Operations would call everybody for a controlled datacenter shutdown that - fortunately - didn't happened. 6 months later I had a similar problem with HVAC. This time water evaporated inside the pipes and we had a "normal" panic because both the inside and the ouside were hot. That was my last on-call duty.