If you have an idea where do OOMs come from, you can (and should!) use "oom_score_adj"/"oom_adj" so that OOM kills processes are not random. There are many int

If you have an idea where do OOMs come from, you can (and should!) use "oom_score_adj"/"oom_adj" so that OOM kills processes are not random. There are many interesting strategies here: - Set your watchdog and logging to "never kill". Set watchdog to reboot on OOM, after uploading error report. This way, if the system OOMs, you can log the problems to your logging server, and then watchdog will reboot the system to recover. - If you have a "UI" process which can also eat all the memory, set it to "kill me first". Yes, the UI will crash, but the rest of system will restart it, and your updater would still work. - If you have separate resource-intensive process, you can set it to "kill me first" -- then the rest of the system can stay up and maybe even show a descriptive error message. - The latter approach works pretty nicely for general systems. Is your OOM killer activating because your build system started too many gcc instances? run it with "kill me first" setting -- it inherits to children processes, so you only build processes would be killed, the rest of the system (UI, browser, etc..) would all stay alive.