3 ms·

At my last sysadmin job, we had a policy of bouncing a server if it's uptime exceeded one year, not to clean up any resources, but to make sure the config in th

At my last sysadmin job, we had a policy of bouncing a server if it's uptime exceeded one year, not to clean up any resources, but to make sure the config in the files matched the config that was running. Of course, not only were we running Linux (gentoo, in fact), but a lot of our stuff was DJB services, including his service manager. I'd argue that a 5 why's analysis of "server up too long", leads to "server wasn't written well." YMMV

mpk 16y ago
Now this is an excellent reason and timeframe for doing periodic reboots. It's basically a scheduled sanity check for your infrastructure. It boils down to robustness and assurance. If parts of your system won't survive a reboot this is an unknown and therefore a risk. If a system can't survive a reboot it's better to find that out when it's during a scheduled reboot and personnel is on hand to deal with it rather than relying on on-call personnel responding to alarms from monitoring at some point. Apart from the repair time being longer when relying on on-call technicians, there is also the general observation that productive systems tend to become more valuable or have more exposure over time as their usage increases. A failure at a later point therefore also becomes more costly as time goes by.
- rbanffy 16y ago
  > Now this is an excellent reason and timeframe for doing periodic reboots. Most of the time you don't need to do complete reboots. Restarting services usually takes care of the majority of the sanity checks.
  - mpk 16y ago
    > Restarting services usually takes care of the majority of the sanity checks. Sure, that usually takes care of most checks, but the point is to eliminate as much as possible unusual scenarios. A service-level restart will catch service config mismatches, but it won't catch, for example, manual interface or routing table updates. It also won't catch the 'fragile root' problem where the device that the kernel is loaded from has become corrupt. Bad things happen in the strangest places if you run enough systems.
    - rbanffy 16y ago
      True. You still would need to reboot from time to time, it's just that you can restart services much more frequently, even automatically if your humans know what is going down today and what can they expect if things go wrong. But I agree - the objective is to detect those situations when things have gone wrong a long time ago and we just don't know it yet. ;-)
jacques_chester 16y ago
Mainframe shops regularly reboot their systems on a rolling basis. A common pattern is that each logical partition gets rebooted once per month, on a rotating basis. Say you have 4 LPARs: LPAR 1 is rebooted on the first week, LPAR 2 on the second and so on. The goal is to demonstrate that in case of catastrophic failure that the system will come up. If it doesn't come back up during the scheduled reboot, you have the advantage that you are rebooting outside production hours and have some time to fix the problem minus frantic phonecalls. Meanwhile, the other LPARs will keep running as if nothing is going on.
gwern 16y ago
Richard P. Gabriel makes that point somewhere - that one of the downsides of an 'organic' system like a Lisp or Smalltalk image where you can rewrite or do anything dynamically is that you tend to do just that, and can wind up in a situation where you are no longer able to stop using the image because it has too many valuable changes which are too difficult to reverse-engineer.
- cwp 16y ago
  That's a bit of an understatement. Basically all Squeak and VisualWorks images have been running since some time in the late '70s. They've been migrated from 16- to 32- to 64-bits and across chip architectures, but it's the same image. Other dialects were bootstrapped more recently, and some of them have retained the ability to create an image from scratch. I guess bootstrapping is more common in the Lisp world.
- epo 16y ago
  I remember believing (though without any supporting evidence) that this was one reason why SmallTalk couldn't possibly catch on commercially. The system was so mutable that any kind of support was impossible, there as no way the support person could know what your system configuration was.
JoeAltmaier 16y ago
Yahoo reboots IM servers daily. Easier than bothering to find all the horrible leaks. Its a business decision. Not worth getting emotionally involved, just weight cost of fixing vs cost of rebooting.

mpk 16y ago

Now this is an excellent reason and timeframe for doing periodic reboots. It's basically a scheduled sanity check for your infrastructure. It boils down to robustness and assurance. If parts of your system won't survive a reboot this is an unknown and therefore a risk. If a system can't survive a reboot it's better to find that out when it's during a scheduled reboot and personnel is on hand to deal with it rather than relying on on-call personnel responding to alarms from monitoring at some point. Apart from the repair time being longer when relying on on-call technicians, there is also the general observation that productive systems tend to become more valuable or have more exposure over time as their usage increases. A failure at a later point therefore also becomes more costly as time goes by.

rbanffy 16y ago
> Now this is an excellent reason and timeframe for doing periodic reboots. Most of the time you don't need to do complete reboots. Restarting services usually takes care of the majority of the sanity checks.
- mpk 16y ago
  > Restarting services usually takes care of the majority of the sanity checks. Sure, that usually takes care of most checks, but the point is to eliminate as much as possible unusual scenarios. A service-level restart will catch service config mismatches, but it won't catch, for example, manual interface or routing table updates. It also won't catch the 'fragile root' problem where the device that the kernel is loaded from has become corrupt. Bad things happen in the strangest places if you run enough systems.
  - rbanffy 16y ago
    True. You still would need to reboot from time to time, it's just that you can restart services much more frequently, even automatically if your humans know what is going down today and what can they expect if things go wrong. But I agree - the objective is to detect those situations when things have gone wrong a long time ago and we just don't know it yet. ;-)

jacques_chester 16y ago

Mainframe shops regularly reboot their systems on a rolling basis. A common pattern is that each logical partition gets rebooted once per month, on a rotating basis. Say you have 4 LPARs: LPAR 1 is rebooted on the first week, LPAR 2 on the second and so on. The goal is to demonstrate that in case of catastrophic failure that the system will come up. If it doesn't come back up during the scheduled reboot, you have the advantage that you are rebooting outside production hours and have some time to fix the problem minus frantic phonecalls. Meanwhile, the other LPARs will keep running as if nothing is going on.

gwern 16y ago

Richard P. Gabriel makes that point somewhere - that one of the downsides of an 'organic' system like a Lisp or Smalltalk image where you can rewrite or do anything dynamically is that you tend to do just that, and can wind up in a situation where you are no longer able to stop using the image because it has too many valuable changes which are too difficult to reverse-engineer.

cwp 16y ago
That's a bit of an understatement. Basically all Squeak and VisualWorks images have been running since some time in the late '70s. They've been migrated from 16- to 32- to 64-bits and across chip architectures, but it's the same image. Other dialects were bootstrapped more recently, and some of them have retained the ability to create an image from scratch. I guess bootstrapping is more common in the Lisp world.
epo 16y ago
I remember believing (though without any supporting evidence) that this was one reason why SmallTalk couldn't possibly catch on commercially. The system was so mutable that any kind of support was impossible, there as no way the support person could know what your system configuration was.

JoeAltmaier 16y ago

Yahoo reboots IM servers daily. Easier than bothering to find all the horrible leaks. Its a business decision. Not worth getting emotionally involved, just weight cost of fixing vs cost of rebooting.