2 ms·

I remember the first time a server caught fire in the closet we kept the rack in. Backups were kept on a server right below the one on fire. But, y'know, we sav

I remember the first time a server caught fire in the closet we kept the rack in. Backups were kept on a server right below the one on fire. But, y'know, we saved money.

eastbound 4y ago
Don’t worry, we do incremental backups during weekdays and a full backup on Sunday. We use 2 tapes only, so one is always outside of the building. But you know, we saved money.
- deleted 4y ago
  [deleted]
- deleted 4y ago
  [deleted]
- bilbo0s 4y ago
  A bit off topic, but I gotta say you guys are a riot! If there was a comedy tour for IT/Programmer types, I'd pay to see you guys in it. Best thing about your stuff is that it's literally all funny precisely because it's all true.
  - deleted 4y ago
    [deleted]
  - geoduck14 4y ago
    I don't know why you are being down voted. I'd watch a Netflix special on this stuff
- Proven 4y ago
- treeman79 4y ago
  We had a million dollars worth of hardware installed in a closet. It had a portable AC hooked up that needed it’s water bin changed every so often. Well I was in the middle of that. When the Director decided to show off the new security doors. So he closed the room up. Then found out that new security doors didn’t work. I find out as I’m coming back to turn AC back on. Room will get hot really fast. We get office Security to unlock door. He says he doesn’t have authority. His supervisor will be by later in the day. Completely deadpan, and in front of several VPs of a forth one 50. I turn to guy to my right who lived nearby. “Go home and get your chainsaw” We were quickly let in. Also got fast approval to install proper cooling.
  - deleted 4y ago
    [deleted]
pbronez 4y ago
This is my fear about my homelab lol Fire extinguisher nearby, smart temp sensors, but still...
- W-Stool 4y ago
  What are you using for a homelab priced temperature sensor?
  - xani_ 4y ago
    Homelab-priced sensor is the temp sensor in your server, it's free! Actual servers have a bunch, usually have one at intake, "random old PC" servers can use motherboard temp as rough proxy for environment temp. Hell, even in DC you can look at temperatures and see in front of which server technican was standing just by those sensors. Second cheapest would be USB-to-1wire module + some DS18B20 1-sire sensors. Easy hobby job to make. They also come with unique ID which means if you put it in TSDB by that ID it doesn't matter where you plug those sensors.
- rovr138 4y ago
  oh, nice idea with temp sensor. I have extinguishers all over the house, but hadn't considered a temperature sensor set to send alerts. Do you have any recommendations?
- xani_ 4y ago
  No more or less chance that really anything else connected to power to catch on fire. Just make sure RCD works. Hell, some weeks ago my oven decided the lower heater's line is now connected to ground and blows fuses... Our 8 racks in DC only had single event of something blowing (power supply) and aside of smell and fuse blowing nothing really happened Servers are essentially metal boxes with a bit of glass-reinforced epoxy and some plastic inside so there is limited amount of stuff that can burn. UPS is probably bigger problem For example the OVH datacenter fire was more of "stuff around servers was flammable" (they had wooden ceilings for some reason...) rather than "just" servers.
  - 4y ago
walnutclosefarm 4y ago
I never saw an actual fire, but I we did see smoke, in a closet, on a floor that was 90% patient care, but happened to have a research area as well because the research needed access to expensive radiography equipment. The close was literally stuffed with what amounted to gaming machines, purchased with grant money through an importer, directly from China. The guy who set them up was smart enough to put them all behind a little firewall, so the enterprise network couldn't see them. It was a (literally) hot mess, both infrastructure and security-wise.

I remember the first time a server caught fire in the closet we kept the rack in. Backups were kept on a server right below the one on fire. But, y'know, we saved money.

eastbound 4y ago

Don’t worry, we do incremental backups during weekdays and a full backup on Sunday. We use 2 tapes only, so one is always outside of the building. But you know, we saved money.

deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
bilbo0s 4y ago
A bit off topic, but I gotta say you guys are a riot! If there was a comedy tour for IT/Programmer types, I'd pay to see you guys in it. Best thing about your stuff is that it's literally all funny precisely because it's all true.
- deleted 4y ago
  [deleted]
- geoduck14 4y ago
  I don't know why you are being down voted. I'd watch a Netflix special on this stuff
Proven 4y ago
treeman79 4y ago
We had a million dollars worth of hardware installed in a closet. It had a portable AC hooked up that needed it’s water bin changed every so often. Well I was in the middle of that. When the Director decided to show off the new security doors. So he closed the room up. Then found out that new security doors didn’t work. I find out as I’m coming back to turn AC back on. Room will get hot really fast. We get office Security to unlock door. He says he doesn’t have authority. His supervisor will be by later in the day. Completely deadpan, and in front of several VPs of a forth one 50. I turn to guy to my right who lived nearby. “Go home and get your chainsaw” We were quickly let in. Also got fast approval to install proper cooling.
- deleted 4y ago
  [deleted]

pbronez 4y ago

This is my fear about my homelab lol Fire extinguisher nearby, smart temp sensors, but still...

W-Stool 4y ago
What are you using for a homelab priced temperature sensor?
- xani_ 4y ago
  Homelab-priced sensor is the temp sensor in your server, it's free! Actual servers have a bunch, usually have one at intake, "random old PC" servers can use motherboard temp as rough proxy for environment temp. Hell, even in DC you can look at temperatures and see in front of which server technican was standing just by those sensors. Second cheapest would be USB-to-1wire module + some DS18B20 1-sire sensors. Easy hobby job to make. They also come with unique ID which means if you put it in TSDB by that ID it doesn't matter where you plug those sensors.
rovr138 4y ago
oh, nice idea with temp sensor. I have extinguishers all over the house, but hadn't considered a temperature sensor set to send alerts. Do you have any recommendations?
xani_ 4y ago
No more or less chance that really anything else connected to power to catch on fire. Just make sure RCD works. Hell, some weeks ago my oven decided the lower heater's line is now connected to ground and blows fuses... Our 8 racks in DC only had single event of something blowing (power supply) and aside of smell and fuse blowing nothing really happened Servers are essentially metal boxes with a bit of glass-reinforced epoxy and some plastic inside so there is limited amount of stuff that can burn. UPS is probably bigger problem For example the OVH datacenter fire was more of "stuff around servers was flammable" (they had wooden ceilings for some reason...) rather than "just" servers.
- 4y ago

walnutclosefarm 4y ago

I never saw an actual fire, but I we did see smoke, in a closet, on a floor that was 90% patient care, but happened to have a research area as well because the research needed access to expensive radiography equipment. The close was literally stuffed with what amounted to gaming machines, purchased with grant money through an importer, directly from China. The guy who set them up was smart enough to put them all behind a little firewall, so the enterprise network couldn't see them. It was a (literally) hot mess, both infrastructure and security-wise.