6 ms·

That looks like an interesting way to intercept vmscan's oom killer. I prefer to avoid it all together. I prefer to set vm.overcommit_ratio=0 and then increas

That looks like an interesting way to intercept vmscan's oom killer. I prefer to avoid it all together. I prefer to set vm.overcommit_ratio=0 and then increase vm.vfs_cache_pressure to somewhere between 400 and 10000 depending on the server role, then set vm.min_free_kbytes, vm.admin_reserve_kbytes and vm.user_reserve_kbytes higher based on the amount of ram in the system using a simple formula. For systems that are ephemeral, I also set vm.panic_on_oom to 2 so that they self heal. Avoid doing this on databases. Of course, THP should also be set to madvise and memory defrag disabled unless you really need it. That will also free up some leaky memory and artificial pausing. Beyond these things, I agree that CGroups is another way to set memory constraints around applications to further protect the server. There are many that won't have CGroups for a while however, as many systems are still on older non systemd distributions.

Twirrim 8y ago
> Of course, THP should also be set to madvise and memory defrag disabled unless you really need it. That will also free up some leaky memory and artificial pausing. Keep an eye out for patches making their way to distribution kernels around THP / memory. There has been some major work done upstream and THP is no longer the alloc-stall causing monster that it was before. It used to cripple some fleets around here (and so we'd have it completely disabled), but once Oracle backported the fixes in to UEK4 it has been smooth as silk with it turned back on.
- LinuxBender 8y ago
  Oh nice! Do you happen to know if those patches have been accepted into RHEL7?
  - Twirrim 8y ago
    I haven't been tracking that, can't honestly say. If you're interested in experimenting and seeing if the patches fix any THP problems you see, UEK4 is completely free, and can be installed on RHEL7/CentOS7 etc. (http://public-yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_64/index.html http://public-yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_...). Could maybe use it to provide details in a bugzilla report?
ebikelaw 8y ago
If you set THP to madvise without also having some hack that calls madvise on your program's text, you'll be throwing out significant amount of CPU rate ... 5-15% in my experience. That's something to weigh against the cost of the extra space.
- LinuxBender 8y ago
  For sure, any system level changes should be tested extensively to determine pros and cons. Performance testing should ideally be a part of the QA process. We found THP to be nothing but trouble on our systems across a myriad of applications. Most of our systems have 512GB+ ram so the problems induced by it are exacerbated quite a bit. The real issue in our case is the associated defrag. It exhibits nearly identical intermittent system lag spikes to Java FGC's.
JdeBP 8y ago
One does not need systemd in order to have control groups. Control groups are a kernel mechanism. However note that the headlined software needs version 2 of Linux control groups.
- LinuxBender 8y ago
  Agreed. I tried to use it on centos 6 but it was very incomplete. The kernel is just too old. I started using kernels from elrepo to work around this, but not many will do that in a production environment.
  - mdaniel 8y ago
    > The kernel is just too old For my curiosity, is there a technical reason for the distro to use an old kernel? I would think one would wish to have all the bug fixes available, and I don't think the kernel has a bad backward compatibility story, so from the outside it seems like a sure thing -- leading me to believe there must be more to the story.
    - LinuxBender 8y ago
      Enterprise distros such as Redhat fork the kernel, then patch for bugs and vulnerabilities, to avoid adding too many new features to a major version release of the OS. This actually makes a lot of sense from a stability and predictability perspective. It would be significantly harder to support the upstream kernel releases, as new features and behavioral changes occur rather often.
bogomipz 8y ago
>I prefer to set vm.overcommit_ratio=0 and then increase vm.vfs_cache_pressure to somewhere between 400 and 10000 depending on the server role, then set vm.min_free_kbytes, vm.admin_reserve_kbytes and vm.user_reserve_kbytes higher based on the amount of ram in the system using a simple formula." I understand vm.overcommit_ratio=0 but can you elaborate on what the net effect of setting all the rest of of these tuneables to these values is? What does this achieve and how do these work together? >"For systems that are ephemeral, I also set vm.panic_on_oom to 2 so that they self heal. Avoid doing this on databases." I'm guessing this triggers a reboot for you when the OOM evicts a process? Do you do this in conjunction with the above setting? Or is this an alternative?
- LinuxBender 8y ago
  Each of the memory settings are described in the kernel sysctl [1] documentation. The goal is to create a larger margin for user error or lack of memory budgeting. This often precludes ever needing swap and forces user space applications into the memory that is actually available. [1] - https://www.kernel.org/doc/Documentation/sysctl/vm.txt https://www.kernel.org/doc/Documentation/sysctl/vm.txt All of the settings should be calculated per server role and memory capacity after extensive testing of your application settings and performance. > I'm guessing this triggers a reboot for you when the OOM evicts a process? It triggers a panic if you hit OOM, rather than kicking in OOM killer which often leaves a system broken, sticky and confused. This should only be used for ephemeral systems or systems that do batch work and can handle reboots. This requires people to find the root cause of their over-allocation and fix them.
  - bogomipz 8y ago
    >"All of the settings should be calculated per server role and memory capacity after extensive testing of your application settings and performance." So you're basically turning every node/instance into a special snowflake of sysctl settings all in the hopes of preventing an OOM? And you do this every time dev push a new rev of code? This doesn't seem very agile or sustainable at any of kind of scale or release velocity.

Twirrim 8y ago

> Of course, THP should also be set to madvise and memory defrag disabled unless you really need it. That will also free up some leaky memory and artificial pausing. Keep an eye out for patches making their way to distribution kernels around THP / memory. There has been some major work done upstream and THP is no longer the alloc-stall causing monster that it was before. It used to cripple some fleets around here (and so we'd have it completely disabled), but once Oracle backported the fixes in to UEK4 it has been smooth as silk with it turned back on.

LinuxBender 8y ago
Oh nice! Do you happen to know if those patches have been accepted into RHEL7?
- Twirrim 8y ago
  I haven't been tracking that, can't honestly say. If you're interested in experimenting and seeing if the patches fix any THP problems you see, UEK4 is completely free, and can be installed on RHEL7/CentOS7 etc. (http://public-yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_64/index.html http://public-yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_...). Could maybe use it to provide details in a bugzilla report?

ebikelaw 8y ago

If you set THP to madvise without also having some hack that calls madvise on your program's text, you'll be throwing out significant amount of CPU rate ... 5-15% in my experience. That's something to weigh against the cost of the extra space.

LinuxBender 8y ago
For sure, any system level changes should be tested extensively to determine pros and cons. Performance testing should ideally be a part of the QA process. We found THP to be nothing but trouble on our systems across a myriad of applications. Most of our systems have 512GB+ ram so the problems induced by it are exacerbated quite a bit. The real issue in our case is the associated defrag. It exhibits nearly identical intermittent system lag spikes to Java FGC's.

JdeBP 8y ago

One does not need systemd in order to have control groups. Control groups are a kernel mechanism. However note that the headlined software needs version 2 of Linux control groups.

LinuxBender 8y ago
Agreed. I tried to use it on centos 6 but it was very incomplete. The kernel is just too old. I started using kernels from elrepo to work around this, but not many will do that in a production environment.
- mdaniel 8y ago
  > The kernel is just too old For my curiosity, is there a technical reason for the distro to use an old kernel? I would think one would wish to have all the bug fixes available, and I don't think the kernel has a bad backward compatibility story, so from the outside it seems like a sure thing -- leading me to believe there must be more to the story.
  - LinuxBender 8y ago
    Enterprise distros such as Redhat fork the kernel, then patch for bugs and vulnerabilities, to avoid adding too many new features to a major version release of the OS. This actually makes a lot of sense from a stability and predictability perspective. It would be significantly harder to support the upstream kernel releases, as new features and behavioral changes occur rather often.

bogomipz 8y ago

>I prefer to set vm.overcommit_ratio=0 and then increase vm.vfs_cache_pressure to somewhere between 400 and 10000 depending on the server role, then set vm.min_free_kbytes, vm.admin_reserve_kbytes and vm.user_reserve_kbytes higher based on the amount of ram in the system using a simple formula." I understand vm.overcommit_ratio=0 but can you elaborate on what the net effect of setting all the rest of of these tuneables to these values is? What does this achieve and how do these work together? >"For systems that are ephemeral, I also set vm.panic_on_oom to 2 so that they self heal. Avoid doing this on databases." I'm guessing this triggers a reboot for you when the OOM evicts a process? Do you do this in conjunction with the above setting? Or is this an alternative?

LinuxBender 8y ago
Each of the memory settings are described in the kernel sysctl [1] documentation. The goal is to create a larger margin for user error or lack of memory budgeting. This often precludes ever needing swap and forces user space applications into the memory that is actually available. [1] - https://www.kernel.org/doc/Documentation/sysctl/vm.txt https://www.kernel.org/doc/Documentation/sysctl/vm.txt All of the settings should be calculated per server role and memory capacity after extensive testing of your application settings and performance. > I'm guessing this triggers a reboot for you when the OOM evicts a process? It triggers a panic if you hit OOM, rather than kicking in OOM killer which often leaves a system broken, sticky and confused. This should only be used for ephemeral systems or systems that do batch work and can handle reboots. This requires people to find the root cause of their over-allocation and fix them.
- bogomipz 8y ago
  >"All of the settings should be calculated per server role and memory capacity after extensive testing of your application settings and performance." So you're basically turning every node/instance into a special snowflake of sysctl settings all in the hopes of preventing an OOM? And you do this every time dev push a new rev of code? This doesn't seem very agile or sustainable at any of kind of scale or release velocity.