4 ms·
No, pets are not fine. Just the other day I had to perform some maintenance on a long-running VM hosting some monitoring software. A backup VM is supposedly al
by oofnik 4y ago
No, pets are not fine.
Just the other day I had to perform some maintenance on a long-running VM hosting some monitoring software. A backup VM is supposedly always running and ready to handle the workload in case of downtime. The switchover seemed to go fine, at first.
Turns out, someone long ago had manually added a cron job to the primary server without adding it to the backup server, without documenting what it does, what permissions it needs, how it works, or why it's needed. This was only discovered after some manager in a different department complained that he stopped receiving the daily report to his e-mail inbox.
If whoever deployed the report generation script took an extra hour to document what the script did, or even better, added it to VCS as part of the provisioning process for the server and re-deployed the server to ensure that the process works as expected, a day's worth of headache could have been averted.