One of my early jobs, I worked for a group of remote sensing scientists that analyzed weather satellite data and the equipment I was responsible for was in an i

One of my early jobs, I worked for a group of remote sensing scientists that analyzed weather satellite data and the equipment I was responsible for was in an improvised raised floor room with probably the smallest unit Liebert produced. I found out one weekend why there were huge fans scattered about the office - the Liebert sometimes failed for various reasons and times and the 5x 52U racks of Dell PE and Sun E3800 servers and storage got super hot super quick. I had to rush in the office with my boss, a brilliant remote-sensing PhD who flew into hurricanes with P-3 Orions to make sure the data they were gathering was accurate with what they were getting from the microwave scatterometer on the satellite, helping him setup fans and shut down equipment that wasn't absolutely necessary. After that, I resolved to be alerted proactively and scared up a setup with some Sensatronics temp sensors, cacti and nagios so I could get data over time as well as alerts when over a threshold. I also enabled the openmanage alerting and found you could get the ambient temp at the air intake of a poweredge so I got that into SNMP and setup alerts on that as well. Fun times. Side note: when I first walked in to the server room, I saw some Apple Xserve RAIDs and thought that was odd next to some Dell stuff and he said, those are the only storage units that can handle the G-forces we experience in hurricanes. "The Powervaults just fall right over" They still had Sun equipment since RHEL was still at v4 and was not fully 64-bit, while Solaris + SPARC had always been 64-bit, and the dataset sizes they were crunching required being able to access RAM-per-process of more than 4GB. MUCH more than.