2 ms·

Funny Story. My first service order way back when was to a engineering office who were in a panic because their primary CAD file server had failed right in the

Funny Story. My first service order way back when was to a engineering office who were in a panic because their primary CAD file server had failed right in the middle of a deadline submittal. I got there when they had just pulled out an identical file server from some other department. They had 'everything' backed up nightly on to a Zip drive. Nice, I thought. So, I plug in the Zip drive and can see all these image files created by their backup software. I ask for the backup app install files and they say its kept in a directory -- on the failed server !!! I should mention this was not in the USA and resulted in a 6+hr long international 42kbps zmodem download session from some random BBS server of the best guess software product and version. I still have one of the failed HDDs from that server as a paper weight on my desk. PS. We got it working in time and the obvious moral of the story is to always test your backup systems (and that goes for both failover and fallback)

gerdesj 5y ago
Every couple of weeks, I or a colleague wander into our bijou data centre. We turn the power off. The UPSs beep for 10 seconds and the genny fires up. A couple of SMSs land on several mobiles and a Teams channel gets notified. Icinga2 notes the power is up but provided by the genny. I particularly like to do this when showing someone around. You whitter on about power and then you swiftly turn the key on the distribution panel and it makes a satisfying clunk. The status lights switch around and the UPSs start beeping. Then they stop. There is a barely perceptible hum from the genny in the boiler room, three walls (one of them a firewall in the traditional sense) away. It is extremely satisfying to do and customers appreciate it - they have all jumped a bit when they first see it. The UPSs obviously (monitored and fortnightly tested) have enough runtime to switch back if something fails on the genny. At a second location there is just enough IT gear to run my company for a while. Replication via Veeam - 'nuff said. Every couple of weeks, Veeam fires up a small part of my company's IT from backups/replicas at the second location. It runs some scripts which tests some functionality and sends out some status reports via email. I still worry about business continuity. There is way more to it than just backups. However, get backups sorted out first. Nine months back a customer had a fire on the production floor. They are still running out of our place, 50 miles away. We set up offsite backups to our place only the week before: "There but for the Grace of God, go I" etc etc. I had them up and running inside three hours from a standing start. Rather lucky there really - they have a functioning business and I have a success story rather than a potential lawsuit from an ex-customer's liquidators trying to claw back some loot.

gerdesj 5y ago

Every couple of weeks, I or a colleague wander into our bijou data centre. We turn the power off. The UPSs beep for 10 seconds and the genny fires up. A couple of SMSs land on several mobiles and a Teams channel gets notified. Icinga2 notes the power is up but provided by the genny. I particularly like to do this when showing someone around. You whitter on about power and then you swiftly turn the key on the distribution panel and it makes a satisfying clunk. The status lights switch around and the UPSs start beeping. Then they stop. There is a barely perceptible hum from the genny in the boiler room, three walls (one of them a firewall in the traditional sense) away. It is extremely satisfying to do and customers appreciate it - they have all jumped a bit when they first see it. The UPSs obviously (monitored and fortnightly tested) have enough runtime to switch back if something fails on the genny. At a second location there is just enough IT gear to run my company for a while. Replication via Veeam - 'nuff said. Every couple of weeks, Veeam fires up a small part of my company's IT from backups/replicas at the second location. It runs some scripts which tests some functionality and sends out some status reports via email. I still worry about business continuity. There is way more to it than just backups. However, get backups sorted out first. Nine months back a customer had a fire on the production floor. They are still running out of our place, 50 miles away. We set up offsite backups to our place only the week before: "There but for the Grace of God, go I" etc etc. I had them up and running inside three hours from a standing start. Rather lucky there really - they have a functioning business and I have a success story rather than a potential lawsuit from an ex-customer's liquidators trying to claw back some loot.