4 ms·
Another useful setting one could potentially use in conjunction with the panic timer is vm.panic_on_oom = 2 In my experience, most systems get into a prec
by LinuxBender 4y ago
Another useful setting one could potentially use in conjunction with the panic timer is
vm.panic_on_oom = 2
In my experience, most systems get into a precarious state when not engineered to only use a percentage of available memory, factoring in the apps that spike memory using when forking child processes. This is most useful on NFS diskless machines, any machine that is managed as cattle vs kittens i.e. treating them like disposable VM's, dev/qa machines and especially machines used for application load testing.
One could even use in production after a significant amount of testing but it requires a few more settings and having automation doing a little bit of math as OOM/vmscan often picks the wrong processes to terminate and leaves the machine in a broken and sticky state and increases downtime impact on SLA. The probability of an OOM panic can be reduced by using:
vm.overcommit_ratio = 0
The default of 50 or 50%, 1.5X ram+swap allows the kernel to permit an application to spawn even when there is no memory available. This is handy on development systems but can lead to problems when automation does not properly calculate how much memory is available and permits an application to request too much memory as the kernel does not protect it's own future memory requirements.
One must then use a little bit of math based on the total memory installed in the system to calculate the values of vm.admin_reserve_kbytes, vm.user_reserve_kbytes, and most importantly vm.min_free_kbytes to prevent applications from grabbing too much memory. This also requires adjusting automation to deal with grabbing enough information when an application does not spawn correctly. This can be further improved by containerizing applications or using cgroup memory constraints when feasible. A percentage of memory must remain unallocated by applications for the kernel to use.
If using idrac/ilo/ilom cards, some of them have a feature to grab screenshots and will save the screenshots that led up to the panic, though will not always get the right moments.
[Edit] I should also add that the math calculations assume all deployment teams and all of their automation tools are aware of one another and math-all-the-things. e.g. production code, monitoring tools, analytic teams all add their numbers together plus overhead. If this is a Kiosk then maybe that is just one team.