3 ms·

if swap is 'cargo cult administration', pretending it won't save you a few times even with all the safeguards you claim that you perfectly do is 'narcissist adm

by cat199 9y ago

if swap is 'cargo cult administration', pretending it won't save you a few times even with all the safeguards you claim that you perfectly do is 'narcissist administration'. if your systems are so advanced that you've done all of this up front profiling, you'd know why the system in the pipe decided 100ms is as good as 10 because you'd be monitoring and alerting on memory/swap and would get the message when it hits 80%-90%, and be logging in while leaky process is blowing up into your swap space. no need for a 'postmortem' if you can have a surgery..

jacquesm 9y ago
If you're architecting your systems in such a way that there is a dynamic component that can cause you to hit the swap then your could also simply buy that much more memory so it won't save you. All you need to do is to monitor your memory usage, any growth that you do not understand is a reason to stop what you are doing immediately and to figure out what is going on. A few GB extra swap space will not save your bacon, will make it much harder to get the system back under control because it is no longer responsive compared to a process killed with a supervisory process restarting that process immediately and logging a fault. And if your systems can't handle that you have bigger problems, likely you then also won't be able to deal with hardware faults, crashes, power failures and other errors. A postmortem is far preferable when it is about a process that is re-launched in a small fraction of a second after which the system is back to normal if the alternative is a system that causes a whole cascade of stuff to go out of whack down the line. Pretty much only Erlang/OTP get this sort of thing right to begin with. Errors - including out of memory errors - should be expected and should be dealt with in a deterministic manner.
- mmjaa 9y ago
  There have been many times, in 30 years, where I've been glad I've got swap enabled so I can recover a nearly-out-of-control production system in time, with the right procedures, instead of everything just being killed. Thats enough for me to turn it on. But, also, gigs of RAM is another must-do cargo-cult thing...
  - jacquesm 9y ago
    Did you do root cause analysis and did you make sure that same condition could never happen again or did you figure that since the swap file saved you that no further action was needed? The 'many times' has me worried.
    - mmjaa 9y ago
      Many times .. over 30 years. And yes, it was mostly due to bugs in the code, which I wouldn't have been able to assess if my machine had just OOM'ed everything. That's the point: swap gives you a little leeway for these analyses.

by cat199 9y ago

jacquesm 9y ago

If you're architecting your systems in such a way that there is a dynamic component that can cause you to hit the swap then your could also simply buy that much more memory so it won't save you. All you need to do is to monitor your memory usage, any growth that you do not understand is a reason to stop what you are doing immediately and to figure out what is going on. A few GB extra swap space will not save your bacon, will make it much harder to get the system back under control because it is no longer responsive compared to a process killed with a supervisory process restarting that process immediately and logging a fault. And if your systems can't handle that you have bigger problems, likely you then also won't be able to deal with hardware faults, crashes, power failures and other errors. A postmortem is far preferable when it is about a process that is re-launched in a small fraction of a second after which the system is back to normal if the alternative is a system that causes a whole cascade of stuff to go out of whack down the line. Pretty much only Erlang/OTP get this sort of thing right to begin with. Errors - including out of memory errors - should be expected and should be dealt with in a deterministic manner.

mmjaa 9y ago
There have been many times, in 30 years, where I've been glad I've got swap enabled so I can recover a nearly-out-of-control production system in time, with the right procedures, instead of everything just being killed. Thats enough for me to turn it on. But, also, gigs of RAM is another must-do cargo-cult thing...
- jacquesm 9y ago
  Did you do root cause analysis and did you make sure that same condition could never happen again or did you figure that since the swap file saved you that no further action was needed? The 'many times' has me worried.
  - mmjaa 9y ago
    Many times .. over 30 years. And yes, it was mostly due to bugs in the code, which I wouldn't have been able to assess if my machine had just OOM'ed everything. That's the point: swap gives you a little leeway for these analyses.