3 ms·
The "restart after upgrade" lesson is one we learned the hard way too, though typically at the OS level, not the application level. We had cases where machines
by perlgeek 9y ago
The "restart after upgrade" lesson is one we learned the hard way too, though typically at the OS level, not the application level.
We had cases where machines with 200+ days of uptime changed from one operations team to another, and the new team did a reboot -- and found that some NFS mounts or ip routes or firewall routes had been added manually at run time, without persistent configuration (like calling mount directly instead of adding stuff to /etc/fstab). Of course, those things were lost after a reboot, and had to be reconstructed somehow.
And this is in an organization with several professional sysadmins per team, not "just" a team of volunteers.
As for the choice of OS: it makes sense to use one that people are familiar with, and that fits their style of operation. I don't want to pass any value judgment on that topic.
All in all, impressive debugging work!
- jlg23 9y agoWhat worked for me with volunteers and paid sysops: Monitor changes in a cronjob, mail diffs to every sysop involved. Someone usually spots that only the live fw rules have changed but not the config and after a few (friendly) applications of the LART people tend to be more careful ;)
- linsomniac 9y agoI used to be all about how long my uptime was. Around a decade ago, I started hating big uptime. "That machine has been up 200 days? WAY past time for a reboot!" In my previous job I implemented a policy of rebooting at least every 6 months, or whenever kernel or glibc/etc updates were done. This was very effective at ensuring that changes to systems were ALWAYS reflected in boot scripts, etc... Way better than finding out that all sorts of problems existed on a hundred servers during an emergency when power went out, and nobody could remember the change that was made a year or more ago.
- grandinj 9y agoEven that is too long. Once a week at a scheduled time is best. That way if anything goes wrong, it's relatively easy to find out what changed because people's memory is still fresh.