5 ms·
Open sourcing oomd, a new approach to handling OOMs
- wheaties 8y agoThat's neat but I question if many of us are at the level of Facebook w.r.t. the need for this. To wit, who here runs their own data center?
- pmlnr 8y agoThere are companies out there who prefer to be in control of their own data and core business, but their numbers are shrinking.
- dagenix 8y agoWhat does running their own data center have to do with it? As I understand it, this daemon could be deployed to a VM instance.
- ebikelaw 8y agoGreat question! You only care about this if you care about cost utilization. If you don't care about cost then you will just statically pre-allocate the space on a host, waste the slack, and never overcommit. If you do care about cost then the pessimistic pre-allocation of space will look wasteful and you'll want something like this. The other aspect of your question is about how "many of us" are working at places with huge datacenters. If you count up all the folks at Amazon, Google, Microsoft, Facebook, and Apple how does that compare to the long tail of people operating smaller datacenters at smaller companies?
- VectorLock 8y agoOOM issues have arisen at every at-scale operation I've ever been involved in. OOM killer does its job, but this seems like it could make it even more manageable.
- uncleamerica 8y agoUncle America says this is communism- do not use this technology.
- LinuxBender 8y agoThat looks like an interesting way to intercept vmscan's oom killer. I prefer to avoid it all together. I prefer to set vm.overcommit_ratio=0 and then increase vm.vfs_cache_pressure to somewhere between 400 and 10000 depending on the server role, then set vm.min_free_kbytes, vm.admin_reserve_kbytes and vm.user_reserve_kbytes higher based on the amount of ram in the system using a simple formula. For systems that are ephemeral, I also set vm.panic_on_oom to 2 so that they self heal. Avoid doing this on databases. Of course, THP should also be set to madvise and memory defrag disabled unless you really need it. That will also free up some leaky memory and artificial pausing. Beyond these things, I agree that CGroups is another way to set memory constraints around applications to further protect the server. There are many that won't have CGroups for a while however, as many systems are still on older non systemd distributions.
- Twirrim 8y ago> Of course, THP should also be set to madvise and memory defrag disabled unless you really need it. That will also free up some leaky memory and artificial pausing. Keep an eye out for patches making their way to distribution kernels around THP / memory. There has been some major work done upstream and THP is no longer the alloc-stall causing monster that it was before. It used to cripple some fleets around here (and so we'd have it completely disabled), but once Oracle backported the fixes in to UEK4 it has been smooth as silk with it turned back on.
- LinuxBender 8y agoOh nice! Do you happen to know if those patches have been accepted into RHEL7?
- Twirrim 8y agoI haven't been tracking that, can't honestly say. If you're interested in experimenting and seeing if the patches fix any THP problems you see, UEK4 is completely free, and can be installed on RHEL7/CentOS7 etc. (http://public-yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_64/index.html http://public-yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_...). Could maybe use it to provide details in a bugzilla report?
- stuaxo 8y agoThere needs to be a signal apps can be sent when resources are low... there are quite a few apps that could do things like empty various caches under ram pressure for instance. Similarly for disk space being low, or under IO pressure, some apps could hold off checking certain files / throttle things for a while.
- voidlogic 8y agoThis is a good point, in Go all instances of sync.Pool (Go's generic object pool with thread local storage) are registered with the allocator/GC so that under heap pressure they can be drained, it would be great if the OS could request this draining too.
- skyde 8y agoOn android this Is done using LRU cache for background app. If your app has a cached process and it retains memory that it currently does not need, then your app—even while the user is not using it— affects the system's overall performance. As the system runs low on memory, it kills processes in the LRU cache beginning with the process least recently used. The system also accounts for processes that hold onto the most memory and can terminate them to free up RAM.
- JdeBP 8y agoThis has been invented long since. On AIX it is SIGDANGER.
- stevekemp 8y agoThis might become useful in the future: https://lwn.net/SubscriberLink/759781/1ebdd9d9518b8f14/ https://lwn.net/SubscriberLink/759781/1ebdd9d9518b8f14/
- AndyKelley 8y agoI would be interested to see this compared to what I consider to be the null hypothesis: disabling overcommit altogether and handling out of memory at the application level.
- Hello71 8y agopoor, because 1. fork exists, and 2. the application incurring OOM is statistically unlikely to be the one that should be killed on multi-application servers, and 3. 99.9% of applications do not correctly handle malloc returning NULL in 100% of cases. some make an effort, but except for stuff like SQLite, almost nobody tests it, so it's usually broken at least 1% of the time.
- toast0 8y ago> 3. 99.9% of applications do not correctly handle malloc returning NULL in 100% of cases. 0% of applications correctly handle being unable to write to a page they were given as writable. It's certainly true that few applications correctly handle a failed malloc, and in many cases there's not really a correct way to handle it, but at least it's available to handle.
- fillest 8y ago>the application incurring OOM is statistically unlikely to be the one that should be killed on multi-application servers Could you please explain this?
- voidlogic 8y agoDoes doing this in user-space have any advantage beyond they could dev a PoC faster? It seems like building a better OOMK and having it live in user-space are orthogonal.
- testvox 8y agoIt's because this supports plugins which are intended to interact with other user-space programs in order to allow for cooperative resource management. These plugins would be much harder to write if they had to be implemented in kernal-space (you couldn't use standard user space libs for making RPC calls, would be easy to introduce kernal corruption if you don't know kernal programming). So yes it's for development reasons but not to allow for a faster POC. It's to allow non kernal devs to easily write plugins.
- josefbacik 8y agoIt would be nice, but for complicated systems in the kernel you sometimes have to provide evidence that the old way sucks and the new way is better. With this work we can easily point at our production workloads and say the in kernel oom killer sucks and this new way is obviously better and hopefully work towards a better in kernel solution. That being said the user is always going to know what they care about more than the kernel, so being able to say “kill chef first, alway” can be a nice thing to have.
- scottlamb 8y agoThe "pressure stall information" looks interesting. It's exponentially weighted moving averages of the fraction of time "some" or "full" (all non-idle) tasks are stalled on each of CPU, memory, and IO. Also a counter of absolute time. [1] Printing this seems like a good addition to "Linux Performance Analysis in 60,000 Milliseconds". [2] More useful that the load average printed by uptime, which hasn't aged well. Linux mixes cpu-bound and io-bound stuff into one bucket [3] which is a terrible idea, and the number of tasks is confusing at best when you have a mix of thread-per-CPU and thread-per-connection stuff running. Looks like it's also exported on a per-cgroups basis, which is great. [1] http://git.cmpxchg.org/cgit.cgi/linux-psi.git/tree/Documentation/accounting/psi.txt http://git.cmpxchg.org/cgit.cgi/linux-psi.git/tree/Documenta... [2] https://medium.com/netflix-techblog/linux-performance-analysis-in-60-000-milliseconds-accc10403c55 https://medium.com/netflix-techblog/linux-performance-analys... [3] http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html http://www.brendangregg.com/blog/2017-08-08/linux-load-avera...
- peterwwillis 8y agoI was excited at first, and then realized this is a kernel patch and not a userland tool. Boooo. FB says "PSI tracks three major system resources — CPU, memory, and I/O — and provides a canonical view into how the usage of these resources changes over time." It should have been possible to do this in userland and make this (somewhat) kernel version-independent. I'm sure I'll use this once there's OSes running a stable kernel with these patches, but I don't run non-vendor kernels in production [unless I'm forced to], and depending on kernel features makes it hard to support legacy systems.
- Hello71 8y agohow would that work edit: you seem to be implying that FB is stupid and likes writing kernel code for no good reason.
- deleted 8y ago[deleted]
- 8y ago
- kevinoid 8y agoInteresting approach. I'm curious to try it out. After playing around with vm.overcommit_ratio, different swap sizes, earlyoom[1], and a few other variables, I still haven't found a happy medium between high memory utilization and low risk of swapping to death. vm.overcommit_ratio=0 is safe, but on systems where occasional swapping is tolerable and memory is limited (e.g. my laptop), I'd rather allow some overcommit. The risk is that if many cold or unallocated pages get touched while the system is under high memory pressure, the system can become totally unresponsive. At the moment I use "Magic SysRq"+f to manually start the oom_killer, when possible. Obviously it's not a great solution. Is there some kernel tunable to keep the system responsive that I'm unaware of? What do you guys do for desktop/laptop systems? 1. https://github.com/rfjakob/earlyoom https://github.com/rfjakob/earlyoom
- 1996 8y agoNot convinced that has any advantage compared to a naive approach of: - introspection at the application level: exit if memory usage goes way above usual (say by 2x) - management by systemd to gracefully restart the application when it exits (dependancies etc), and also monitor memory use in case the application doesn't monitor it well (say exit when 2.5x the usual) I could be made better, but at the cost of more complexity.
- ebikelaw 8y agoI guess this would be a good way to operate dunning-kruger.com, but if you want your site to stay up this seems problematic. If you introspective application forms some belief about "usual" memory usage when it's idle an unloaded, and you suddenly shift all of your traffic from another datacenter to this one, and the heap usage on that application increases by 10x, then all your replicas will suicide.
- 1996 8y agoIf the memory usage can increase 10x while this is not reflected in the assumptions, both the measurements and the assumptions are flawed To give more context: I run many copies of a daemon with known memory leaks, on machines each with a load close to 0.7 and about 200M of free ram left. I consider that an heavy load. This approach restarts one of the copies of the daemon when its own leaks become dangerous to the rest of the system. It is not perfect, but good enough when the numbers are measured at peak. The 200Mb free RAM are all the extra leeway I care to leave.
- __flo 8y agoAndroid also has its own low memory killer: https://android.googlesource.com/platform/system/core/+/master/lmkd/ https://android.googlesource.com/platform/system/core/+/mast...