6 ms·
At work, a server that I am responsible for the software on, but not the general IT-type management, was becoming increasingly flaky. It kept powering down at o
by jerf 17y ago
At work, a server that I am responsible for the software on, but not the general IT-type management, was becoming increasingly flaky. It kept powering down at odd times, once managing to corrupt a MySQL database pretty badly. Shortly after a firmware update, it ceased coming up. I honestly did assume it was something in my software update, as updating this server was often the next step after QAing the software. Once I realized that I couldn't even ping the device (which I am 2000 miles from), I had to send in the local IT personnel to log in physically.
This is when they discovered that the device would no longer physically turn on. Next step is to de-rack the device, at which point some strange discoloration was discovered in the holes in the case. Next step was to open the case, where it was discovered that the entire machine was full of mold. The roof was leaking, and this was on the top of the rack directly under the leak. In Silicon Valley, leaky roofs can apparently take a while to discover, but they will eventually make their presence known.
Now, this was in a local office, albeit a relatively well-equipped on that holds in the hundreds of rack machines, not a colo. But you know what? Shit happens. And you don't really have a guarantee that shit won't happen just because someone has magically slapped the word "data center" on a building.
- z8000 17y agoonce managing to corrupt a MySQL database pretty badly. Rare as a Himalayan Snowcock spotting!