21 ms·

What has to happen with Unix virtual memory when you have no swap space

214 points by stargrave 7y ago

herpderperator 7y ago
What am I missing here? Why can't, as others have mentioned in other HN comments elsewhere, the OOM killer just get invoked when there's less than X amount of RAM left, and kill the highest-offending process? In my case, I would prefer that to anything else. Why does this page or that page matter?
- the8472 7y ago
  Because you're not truly out of o memory at that point. there are still pages that can be evicted. to invoke the oomkiller sooner you have to use heuristics to determine when a request could be satisfied in theory but not practically due to page thrashing. The oom killer only kicks in sometimes, e.g. when programs make truly egregious allocation requests. The benefit of having swap is that it turns things into a soft degradation since it's much easier for the system to start with swapping out rarely used pages. The gradual loss of performance makes it easier for the human to intervene compared to the cliff you encounter when it starts dropping shared code pages.
  - wahern 7y ago
    > The oom killer only kicks in sometimes, e.g. when programs make truly egregious allocation requests. This is a myth. Allocation failures happen at least as much on small allocations as on big ones. In fact, I see OOMs every day and the vast majority of the time the trigger was a small allocation. For example, the kernel trying and failing to allocate a socket buffer. And that's really the root of the issue. You have a giant application with a 200GB committed working memory set doing important, critical work; and it gets shot down because some other process just tried to initiate an HTTP request. It's a ludicrous situation. And people defending Linux here by saying the same problem exists everywhere else are wishful apologists--the situation is absolutely not the same everywhere else. Even setting aside the issue of strict memory accounting--which, BTW, both Windows and Solaris are perfectly capable of doing, and do by default--Linux could still do dramatically better. Clearly there's some level of unreliability people are willing to put up with for the benefits of efficiency, but Linux blew past that equilibrium long ago.
    - the8472 7y ago
      E.g. == for example, other cases are permitted. What I am saying is that it only kicks in under some circumstances, not necessarily when one wants it to.
  - tsimionescu 7y ago
    > Because you're not truly out of o memory at that point. there are still pages that can be evicted. No, you are truly out of RAM at that point - the amount of RAM all processes need exceeds the amount of memory the system has, and the user has indicated that no swapping should be done. Now, if the kernel truly wanted to handle this case gracefully by swapping disk-backed files, I think it should also tell the process scheduler about this, and enter a special mode where only processes whose code is currently resident would be allowed to run, until they hit a hard wait. This might prevent the thrashing behavior in many cases (assuming processes don't interact too much with memory-mapped files). Otherwise, every context switch initiated by the scheduler is likely to cause another page fault event.
    - the8472 7y ago
      > the amount of RAM all processes need exceeds the amount of memory the system has The word need does the heavy lifting here. Strictly speaking it does not. It CPU only needs a few data and instruction pages at a time to execute code. The former may, the latter frequently will be backed by memory-mapped files. If your program contains huge unicode lookup tables, megabytes of startup code and so on then it is perfectly reasonable to discard those pages. It would be wasteful to keep those resident, especially on memory-constrained devices. Not having swap is not the same as not wanting paging to happen. What the human wants is totally different from what the kernel needs to keep chugging along (at glacial pace). Bridging the gap is what is discussed further downstream in the mailing list linked by the article, and it's only possible based on previous work (PSI) that was added fairly recently.
      - mrob 7y ago
        Why is the kernel deciding if the memory is still needed or not? The kernel can only know how it is used, not why it is used. It can guess the purpose with heuristics, but only the applications know exactly why the memory is used. The correct solution is to disable overcommit, which will force programmers to make use of that superior knowledge to free memory when they no longer need it. We're in this crazy situation because somebody decided to sacrifice reliability to increase efficiency. Efficiency is easier to measure so they got away with it. Fixing the live-lock problem makes things slightly better, but it's not solving the underlying problem. Andries Brouwer's classic comment remains as relevant as ever: https://lwn.net/Articles/104185/ https://lwn.net/Articles/104185/
- xenadu02 7y ago
  Because simplistic policies like this can never satisfy everyone. On a server the process using the most memory might be the mail server so killing it is a bad idea... unless it normally never uses more than 2GB and for some reason it’s eating 14GB at the time. You have to define quality of service, set relative priorities within those categories, be able to correlate memory usage with load to detect abnormal spikes vs normal spikes, etc. For example a daemon’s memory usage is probably correlated with its open socket count. That in turn relies on a system much smarter than init scripts that understands the system is under memory load so re-starting a daemon that was just killed might not be appropriate (which gets into policy questions too). For interactive use things are even more complicated; you need a mechanism for the active application being used by the user to donate its ultra-high priority to any daemons or child processes it is talking to. If image_convert is running because I initiated it as the user then it should get more memory (and CPU/IO). If it is running because my desktop window manager is refreshing it’s icon cache then it should get its priority smashed hard - if the system is under pressure or burning battery it should even be killed and prevented from restarting temporarily. Who is going to do all the work in the kernel, userspace libraries, then get all the apps to properly tag their threads and requests with the correct priority? tl;dr: setting policy is hard and anyway it becomes the herding cats problem.
  - pmontra 7y ago
    I think that all processes are needed on the typical server, monitoring included. If one often runs out of RAM, buy more physical or virtual RAM. Furthermore we probably have more than one server if we really care about the service they offer. One server crashes, another one spins up, the others copie with the load meanwhile. The option to kill processes is more interesting on personal machines, where sometimes adding RAM is not even an option. My anecdotal experience with a laptop without swap since 2014 (16 GB, then 32) is that I got near the limit a couple of times. I wouldn't mind if Linux killed any of the browsers, emacs (which is tiny nowadays), Thunderbird. They can recover. Maybe kill even databases. Leave me XOrg, the desktop environment and a terminal to investigate the problem.
    - AstralStorm 7y ago
- posterboy 7y ago
  Why can't the process itself decide how to handle what should be a null pointer return from malloc or mmap et al? In fact, firefox' and chromes example is the bane of my existence. The recovery feature itself takes a good amount of space. However efficient it may be, it's no use for private sessions. I'm aware of tab suspender add ons but never tried it. You can't tell me to clean up the ship and instead you burn it down and let it sink in the ocean? Is it my messy browsing (400 tabs can grow to 8gb quickly and it should always remain snappy, too, no eager unloading) or is that just symptomatic of a let it crash darwinism? It is indeed just as impossible for me to memorize all the pages, 90% wikipedia articles and related content, nothing important I'm sure. But it is utterly ridiculous that a gif which rendered would take hundreds of megabytes is a surefire way to crash the session that had been skimming the edge of 200 mb free resident memory left for weeks. I had resorted to polling 'free' frequently for display in a taskbar plugin. But sometimes over night something would hog memory and not free it, either a systems cron-job, an aberrant javascript (though no-script was rarely "temporarily" disabled) or a memory leak (same thing?). Of course a browser with leaks could not be expected to handle oom gracefully.
  - lclarkmichalek 7y ago
    > Why can't the process itself decide how to handle what should be a null pointer return from malloc or mmap et al? At the point where the kernel knows that it should be returning a null pointer, you're probably already operating in a severely degraded state. You will be scanning the page table constantly, looking for pages to reclaim, wasting tons of CPU. You will have reclaimed all of the reclaimable pages you can reclaim, so io utilisation will be through the roof - calling a random function will cause a disk read to page the executable into memory. Your system will not be functional. And that's ignoring the reality that modern MMUs don't actually let the kernel know when all the memory is in use. If you want to handle memory pressure better for a given application, shove the application in a cgroup and use a user space oom killer. It's not possible for a program to react gracefully to a system OOM.
    - tremon 7y ago
      At the point where the kernel knows that it should be returning a null pointer, you're probably already operating in a severely degraded state That's only currently the case. It doesn't need to be. As per your example, you could just as easily provide an upper bound to the page table scan, and return NULL when enough pages can't be found in 100us. It's not possible for a program to react gracefully to a system OOM. A program should never receive a system OOM, the fact that Linux' memory accounting is so bad the system itself can run OOM is the problem. It is perfectly possible for a system kernel to execute in bounded memory, and never let programs claim parts of that. Linux just isn't designed that way.
- mrob 7y ago
  There's no reason the OOM killer can't be made more aggressive, and there are user-space implementations of that behavior. I use the "Early OOM Daemon"[0], which is packaged in Debian. I had problems with my system locking up under memory pressure before, but so far earlyoom has always managed to kill processes early enough to prevent this. [0] https://github.com/rfjakob/earlyoom https://github.com/rfjakob/earlyoom
- deleted 7y ago
  [deleted]
wahern 7y ago
> If you can't on your particular Unix, I'd actually say that your Unix is probably not letting you get full use out of your RAM. What use is "full use" if your system live locks!? This is similar logic to that of overcommit--more "efficient" use of RAM in most cases, with the tiny, insignificant cost of your processes randomly[1] being killed? What happens in practice is that people resort to over provisioning RAM anyhow. But even then that's no guarantee that your processes won't be OOM-killed. We're seeing precisely that in serious production environment--OOM killer shooting down processes even though there's plenty (e.g. >10%, tens of gigabytes) of unallocated memory because it can't evict the buffer cache quickly enough--where quickly enough is defined by some magic heuristics deep in the eviction code. [1] And before you say that it's not random but can be controlled... you really have no idea of the depth of the problem. Non-strict, overcommit-dependent logic is baked deep into the Linux kernel. There are plenty of ways to wedge the kernel where it will end up shooting down the processes that are supposed to be the most protected, sometimes shooting down much of the system. In the many cases people simply reboot the server entirely rather than wait around to see what the wreckage looks like. This is 1990s Windows revisited--reboot and move along, unreliability is simply the nature of computers....
- the8472 7y ago
  You can disable the oom killer if you think its heuristics are misfiring. > OOM killer shooting down processes even though there's plenty (e.g. >10%, tens of gigabytes) of unallocated memory because it can't evict the buffer cache quickly enough Is that due to dirty pages? Have you tried tweaking the dirty ratio to get dirty files flushed sooner? I believe there were also some improvements in the default behavior towards the end of the 4.x series that are supposed to result in more steady flushing.
  - wahern 7y ago
    > You can disable the oom killer if you think its heuristics are misfiring. You can't completely disable the OOM killer. Some in-kernel allocations will trigger an OOM reap and kill regardless. Too many parts of Linux were written with the assumption of overcommit. Heck, as this issue shows even when there's technically free memory the OOM killer can still kick in. I had heard about these issues before but never saw that in practice--at least not often enough that I looked closely enough. I knew Google and Facebook and others have been relying on userspace OOM killers for years to improve reliability and consistency. But only after having recently joined a DevOps team at a large enterprise with large enterprise, big data customers did I begin to see the depth and breadth of the problem. Before getting into the hell-hole of helpless despair that is DevOps I primarily did systems programming. And, for the record, I always made sure to handle allocation failure in those languages where I could (C, Lua, etc) because as a software engineer it was always clear to me that, much as with security, an over reliance on policy-based mechanisms invariably resulted in the worst possible QoS. (Of course, in Linux even disabling overcommit is no guarantee that page fault won't trigger an OOM kill, but there are other techniques to improve reliability under load.) > Is that due to dirty pages? Yes, people have tweaked and turned those knobs and reduced incidence rate by maybe 20-30%. Yet nobody is sleeping better at night. Computing shouldn't be a black art. The irony is that overcommit and loose memory accounting more generally was originally intended to remove the necessity for professional system administrators to babysit a system to keep it humming along. But now we've come completely full circle. And it was entirely predictable. And before anyone says that cloud computing means you increase reliability by scaling horizontally, well "cloud scale" systems is exactly what are being run. But unless you massively over provision, and especially when certain tasks, even when split up across multiple nodes, can take minutes or even hours, an OOM kill incident can reverberate and cascade. Again, ironic because the whole idea of loose memory accounting is supposedly to obviate the need to over provision.
diamondo25 7y ago
Although I am not a fan of Apple, their Memory Pressure messages on iOS are really useful to prevent this. You get warned as a programmer to clean up your memory if possible, otherwise iOS shuts you down.
- macdice 7y ago
  Also AIX has SIGDANGER (best signal name ever). There is also "oomd", I think like the iOS thing, for Linux, in user space. Not sure how useful any of these things really are but wondering if FreeBSD should get one...
- lclarkmichalek 7y ago
  Which we now have on Linux :D https://facebookmicrosites.github.io/cgroup2/docs/pressure-metrics.html https://facebookmicrosites.github.io/cgroup2/docs/pressure-m...
- gridlockd 7y ago
  Yes, but as a result, memory-intensive applications (usually games) crash all the time, especially on older devices, which the App Store won't let you filter out. What are you going to anyway, just start deallocating the memory your application needs to actually work, so that you can display an "I'm sorry, we're out of memory" message? Might as well crash then. The amount of memory available also varies wildly from user to user and you can't put a memory requirement on an application. So what do people do? Disregard the issue. Even the best games on the App Store have tons of one star reviews by people with older devices. The flip side is that application that are prone to crash are pressured to optimize for relatively quick and painless restart.
popeye77 7y ago
I'd like to add that RHEL5 was more than tolerable in this aspect. Whatever kernel series it lived through, it seemed logical. We knew when RHEL5 systems started to swap that was our sign to A) look for long term usage growth that is beginning to bump some threshold somewhere B) someone turned a knob they shouldn't have C) a recent code change has caused a problem in the application stack Then RHEL6 came along, and it swapped all the time. Gone was our warning. The stats shows tens of gigabytes of cache engaged. WUT? How do we have tens of gigabytes of memory doing nothing? Before you could finish that thought, OOM killer was killing off programs due to memory pressure. WTF? The system is swallowing RAM to cache I/O, but couldn't spare a drop for programs? ...I could go on, but simply put, RHEL6 was garbage. And really I mean the RHEL6 kernels and the way that Red Hat tweaked whatever they did for it. RHEL7 was a little better, but still seeing echoes of the ugliness of RHEL6. RHEL5 was just a faded pleasant dream. The last 3 Fedoras on the other hand, the memory management seems like we're finally digging ourselves out of the nightmare. That nightmare lasted almost a full decade.... sheesh
trhway 7y ago
i routinely see these days systems without or with very low swap. It is like swap has become faux pas. That is especially strange giving the SSD drives available on the machines. Gradual degradation of service vs. the service sudden disappearance and/or stall or heavy overprovisioning and still ... Also comes to mind - while not generic swap - kind of edge case, a modern version of swap, ie. extending virtual memory space onto flash storage - Facebook replacement of some RAM with NVM https://research.fb.com/wp-content/uploads/2018/03/reducing-dram-footprint-with-nvm-in-facebook.pdf https://research.fb.com/wp-content/uploads/2018/03/reducing-...
- IshKebab 7y ago
  Yeah because swap on Linux doesn't really work. When it is needed your system grinds to a halt.
  - Dylan16807 7y ago
    I strongly disagree with that characterization. At some amount of active memory use Linux will grind to a halt. This happens with or without swap. It's a problem of how it handles low memory situations. But if you have swap, this point comes much later. Often many gigabytes of unimportant data can be moved out of memory. If you can swap out 6 gigabytes without causing problems, and you only needed an extra 4 gigabytes of ram, then swap saves your day with almost no downside.
  - trhway 7y ago
    i would venture a guess that probably you saw something like for example large JVM being partially swapped on HDD - in a situation like that it isn't swap per.se. which is at fault, it is the fact that JVM memory access pattern - randomly jumping all over - is extremely unsuitable for HDD based swap.
    - neel_k 7y ago
      Actually, there is a critically bad interaction between the way that garbage collection algorithms work and the LRU heuristic for swap. Tracing garbage collection algorithms work by doing a graph traversal that traces all the live pointers starting from the "root set" (ie, pointers in registers and on the stack) to find all the live objects. Now, note that (a) the root set are the hottest objects in memory, and (b) they are the ones that gc algorithms touch first. As a result, when you are doing a gc, the LRU (least-recently-used) heuristic for managing swap ends up putting all the hottest data in the program onto disk. This is pretty much the pessimal thing for the kernel to do, and can tank performance by triple-digit factors. However, this is entirely fixable! The gc and the kernel need to cooperate to figure out what is okay to push to disk or not. Matthew Hertz, Yi Feng and Emery Berger have a really nice PLDI 2005 paper, Garbage Collection without Paging, which shows how you can make gc performance hundreds of times better with just a little kernel assistance. However, because kernel developers do not program in garbage collected languages (they program in C, because that's what the kernel is written in), they don't understand garbage collection and are hostile to kernel changes to support it. Emery Berger told me they tried and failed to get the Linux kernel to support their changes, and people who worked on .NET told me they had similar experiences with the Windows kernel team.
lmilcin 7y ago
Well, the issue really is that applications are not being designed correctly. To create reliable applications you need to design them to work within limits of memory allocated to them. Unattended applications (like RDBMS or application container) should not allocate memory dynamically based on user input. By definition, reliable application will not fail because of external input. Allocating memory from OS cannot be done reliably unless that memory was set aside somehow. If the memory was not set aside then we say we are in overprovisioning situation and this means we accept that it might happen that a process wanting to allocate memory will fail to get it. So the solution (the simplest but not the only one) is to allocate the memory ahead of time for the maximum load the app can experience and to make a limit on the load (number of concurrent connections, etc.) so that this limit is never exceeded. Some systems do it halfway decently. For example Oracle will explicitly allocate all its memory spaces and then work within those. The OS really is a big heap of compromises. It does not know anything about the processes it runs but it is somehow expected to do the right thing. We see it is not behaving gracefully when memory runs out but the truth is, OS is built for the situation where it is being used mostly correctly. If memory is running out it means the user already made a mistake in desiging their use of resources (or didn't think about it at all) and there is not really much the OS can do to help it.
- lclarkmichalek 7y ago
  > Unattended applications (like RDBMS or application container) should not allocate memory dynamically based on user input. This is incredibly easy to say, and incredibly hard to do for any application of even moderate complexity. Getting people to accurately tune thread counts or concurrency limits is painful when working with diverse workloads, and getting a guaranteed upper bound on memory usage would be a lot harder. It's much more common to apply higher level techniques such as concurrency limits and load shedding to avoid OOMs and other resource starvation issues. The other thing we can do is reduce the impact of OOMs, and accept that they exist as a tail event. There is software running on your system that doesn't have hard memory limits, and you will always have to consider OOMs and other similar failure cases when ensuring your system is resilient. As long as we can prevent those tail events becoming a correlated failure across our fleet, we can get a pretty reliable system.
anticristi 7y ago
I feel that the problem is ill-posed: "At what amount of free RAM should we start freeing RAM by killing processes?" Maybe one should solve the problem: "At what eviction-induced latency should the OOM killer be invoked." Thanks to the infrastructure around latencytop, the latency might be available already. Of course, the never-ending dilemma of what process to kill is still there.
- AstralStorm 7y ago
  Preferably none. Saner APIs like Android will call low memory signals and start swapping out applications both ahead of time and as needed. It also has a relatively smart memory manager to detect unexpected vs expected spikes as well as slow leaks. POSIX has no such API. It was designed in a simple time.
  - anticristi 7y ago
    Can you point to documentation on how Android deals with low-memory situations? My understanding is that: (A) They moved the problem to user-space. (B) Apps are supposed to constantly save the state they care about and expect to be killed whenever Android sees fit (or even at every app switch for developers).
known 7y ago
Unlike traditional swap, Android's Memory Manager kills inactive processes to free up memory
- AstralStorm 7y ago
  That's because the whole Android API and ecosystem was designed to make it possible. You cannot get away with that and POSIX applications. They have zero or bad session support. There's no global always available database to help you (Android has 4), nor event scheduling bus. (Dbus is a joke, it does not allow sending event to non-existing endpoint to be delivered later.) Every application using the standard mechanisms will get relaunched and activated as needed if it got shut down. It will receive a Bundle with state it managed to save and with original launch Intent. It can access an sqlite database to be made available via a restartable ContentProvider. SharedPreferences are also stored. Etc. In Linux world, there are no such de facto standards and what is most common is utterly broken. Same in Windows (only registry is persistent, it's not meant for data storage) and in OS X.
not2b 7y ago
The particular issue of locking up a system because there are too many tabs open in Firefox or Chrome should be fixed in the browser, because the browser is the program that is taking up all of the memory and is in the best position to know how to recover a lot of memory quickly while minimizing the loss of work for the user. One of the comments suggested having the kernel send a 'memory pressure' signal to all processes, and a process could catch the signal and do garbage collection or drop other storage. But even without such a feature, a program could do polling to determine that there is memory pressure, or do a check when a user asks to open a new tab, etc.
amyjess 7y ago
My experience for a long time has been that Linux is only usable with earlyoom. Running without it is a guaranteed way to end up having to power-cycle my system, often multiple times per day.
mighty_bander 7y ago
I managed to achieve this once, fresh from the world of Windows - missed breaking out of a loop while exploring Python(?) on Ubuntu 10 or so. Had to power cycle my damn machine.

214 points by stargrave 7y ago

What am I missing here? Why can't, as others have mentioned in other HN comments elsewhere, the OOM killer just get invoked when there's less than X amount of RAM left, and kill the highest-offending process? In my case, I would prefer that to anything else. Why does this page or that page matter?

the8472 7y ago
Because you're not truly out of o memory at that point. there are still pages that can be evicted. to invoke the oomkiller sooner you have to use heuristics to determine when a request could be satisfied in theory but not practically due to page thrashing. The oom killer only kicks in sometimes, e.g. when programs make truly egregious allocation requests. The benefit of having swap is that it turns things into a soft degradation since it's much easier for the system to start with swapping out rarely used pages. The gradual loss of performance makes it easier for the human to intervene compared to the cliff you encounter when it starts dropping shared code pages.
- wahern 7y ago
  > The oom killer only kicks in sometimes, e.g. when programs make truly egregious allocation requests. This is a myth. Allocation failures happen at least as much on small allocations as on big ones. In fact, I see OOMs every day and the vast majority of the time the trigger was a small allocation. For example, the kernel trying and failing to allocate a socket buffer. And that's really the root of the issue. You have a giant application with a 200GB committed working memory set doing important, critical work; and it gets shot down because some other process just tried to initiate an HTTP request. It's a ludicrous situation. And people defending Linux here by saying the same problem exists everywhere else are wishful apologists--the situation is absolutely not the same everywhere else. Even setting aside the issue of strict memory accounting--which, BTW, both Windows and Solaris are perfectly capable of doing, and do by default--Linux could still do dramatically better. Clearly there's some level of unreliability people are willing to put up with for the benefits of efficiency, but Linux blew past that equilibrium long ago.
  - the8472 7y ago
    E.g. == for example, other cases are permitted. What I am saying is that it only kicks in under some circumstances, not necessarily when one wants it to.
- tsimionescu 7y ago
  > Because you're not truly out of o memory at that point. there are still pages that can be evicted. No, you are truly out of RAM at that point - the amount of RAM all processes need exceeds the amount of memory the system has, and the user has indicated that no swapping should be done. Now, if the kernel truly wanted to handle this case gracefully by swapping disk-backed files, I think it should also tell the process scheduler about this, and enter a special mode where only processes whose code is currently resident would be allowed to run, until they hit a hard wait. This might prevent the thrashing behavior in many cases (assuming processes don't interact too much with memory-mapped files). Otherwise, every context switch initiated by the scheduler is likely to cause another page fault event.
  - the8472 7y ago
    > the amount of RAM all processes need exceeds the amount of memory the system has The word need does the heavy lifting here. Strictly speaking it does not. It CPU only needs a few data and instruction pages at a time to execute code. The former may, the latter frequently will be backed by memory-mapped files. If your program contains huge unicode lookup tables, megabytes of startup code and so on then it is perfectly reasonable to discard those pages. It would be wasteful to keep those resident, especially on memory-constrained devices. Not having swap is not the same as not wanting paging to happen. What the human wants is totally different from what the kernel needs to keep chugging along (at glacial pace). Bridging the gap is what is discussed further downstream in the mailing list linked by the article, and it's only possible based on previous work (PSI) that was added fairly recently.
    - mrob 7y ago
      Why is the kernel deciding if the memory is still needed or not? The kernel can only know how it is used, not why it is used. It can guess the purpose with heuristics, but only the applications know exactly why the memory is used. The correct solution is to disable overcommit, which will force programmers to make use of that superior knowledge to free memory when they no longer need it. We're in this crazy situation because somebody decided to sacrifice reliability to increase efficiency. Efficiency is easier to measure so they got away with it. Fixing the live-lock problem makes things slightly better, but it's not solving the underlying problem. Andries Brouwer's classic comment remains as relevant as ever: https://lwn.net/Articles/104185/ https://lwn.net/Articles/104185/
xenadu02 7y ago
Because simplistic policies like this can never satisfy everyone. On a server the process using the most memory might be the mail server so killing it is a bad idea... unless it normally never uses more than 2GB and for some reason it’s eating 14GB at the time. You have to define quality of service, set relative priorities within those categories, be able to correlate memory usage with load to detect abnormal spikes vs normal spikes, etc. For example a daemon’s memory usage is probably correlated with its open socket count. That in turn relies on a system much smarter than init scripts that understands the system is under memory load so re-starting a daemon that was just killed might not be appropriate (which gets into policy questions too). For interactive use things are even more complicated; you need a mechanism for the active application being used by the user to donate its ultra-high priority to any daemons or child processes it is talking to. If image_convert is running because I initiated it as the user then it should get more memory (and CPU/IO). If it is running because my desktop window manager is refreshing it’s icon cache then it should get its priority smashed hard - if the system is under pressure or burning battery it should even be killed and prevented from restarting temporarily. Who is going to do all the work in the kernel, userspace libraries, then get all the apps to properly tag their threads and requests with the correct priority? tl;dr: setting policy is hard and anyway it becomes the herding cats problem.
- pmontra 7y ago
  I think that all processes are needed on the typical server, monitoring included. If one often runs out of RAM, buy more physical or virtual RAM. Furthermore we probably have more than one server if we really care about the service they offer. One server crashes, another one spins up, the others copie with the load meanwhile. The option to kill processes is more interesting on personal machines, where sometimes adding RAM is not even an option. My anecdotal experience with a laptop without swap since 2014 (16 GB, then 32) is that I got near the limit a couple of times. I wouldn't mind if Linux killed any of the browsers, emacs (which is tiny nowadays), Thunderbird. They can recover. Maybe kill even databases. Leave me XOrg, the desktop environment and a terminal to investigate the problem.
  - AstralStorm 7y ago
posterboy 7y ago
Why can't the process itself decide how to handle what should be a null pointer return from malloc or mmap et al? In fact, firefox' and chromes example is the bane of my existence. The recovery feature itself takes a good amount of space. However efficient it may be, it's no use for private sessions. I'm aware of tab suspender add ons but never tried it. You can't tell me to clean up the ship and instead you burn it down and let it sink in the ocean? Is it my messy browsing (400 tabs can grow to 8gb quickly and it should always remain snappy, too, no eager unloading) or is that just symptomatic of a let it crash darwinism? It is indeed just as impossible for me to memorize all the pages, 90% wikipedia articles and related content, nothing important I'm sure. But it is utterly ridiculous that a gif which rendered would take hundreds of megabytes is a surefire way to crash the session that had been skimming the edge of 200 mb free resident memory left for weeks. I had resorted to polling 'free' frequently for display in a taskbar plugin. But sometimes over night something would hog memory and not free it, either a systems cron-job, an aberrant javascript (though no-script was rarely "temporarily" disabled) or a memory leak (same thing?). Of course a browser with leaks could not be expected to handle oom gracefully.
- lclarkmichalek 7y ago
  > Why can't the process itself decide how to handle what should be a null pointer return from malloc or mmap et al? At the point where the kernel knows that it should be returning a null pointer, you're probably already operating in a severely degraded state. You will be scanning the page table constantly, looking for pages to reclaim, wasting tons of CPU. You will have reclaimed all of the reclaimable pages you can reclaim, so io utilisation will be through the roof - calling a random function will cause a disk read to page the executable into memory. Your system will not be functional. And that's ignoring the reality that modern MMUs don't actually let the kernel know when all the memory is in use. If you want to handle memory pressure better for a given application, shove the application in a cgroup and use a user space oom killer. It's not possible for a program to react gracefully to a system OOM.
  - tremon 7y ago
    At the point where the kernel knows that it should be returning a null pointer, you're probably already operating in a severely degraded state That's only currently the case. It doesn't need to be. As per your example, you could just as easily provide an upper bound to the page table scan, and return NULL when enough pages can't be found in 100us. It's not possible for a program to react gracefully to a system OOM. A program should never receive a system OOM, the fact that Linux' memory accounting is so bad the system itself can run OOM is the problem. It is perfectly possible for a system kernel to execute in bounded memory, and never let programs claim parts of that. Linux just isn't designed that way.
mrob 7y ago
There's no reason the OOM killer can't be made more aggressive, and there are user-space implementations of that behavior. I use the "Early OOM Daemon"[0], which is packaged in Debian. I had problems with my system locking up under memory pressure before, but so far earlyoom has always managed to kill processes early enough to prevent this. [0] https://github.com/rfjakob/earlyoom https://github.com/rfjakob/earlyoom
deleted 7y ago
[deleted]

wahern 7y ago

> If you can't on your particular Unix, I'd actually say that your Unix is probably not letting you get full use out of your RAM. What use is "full use" if your system live locks!? This is similar logic to that of overcommit--more "efficient" use of RAM in most cases, with the tiny, insignificant cost of your processes randomly[1] being killed? What happens in practice is that people resort to over provisioning RAM anyhow. But even then that's no guarantee that your processes won't be OOM-killed. We're seeing precisely that in serious production environment--OOM killer shooting down processes even though there's plenty (e.g. >10%, tens of gigabytes) of unallocated memory because it can't evict the buffer cache quickly enough--where quickly enough is defined by some magic heuristics deep in the eviction code. [1] And before you say that it's not random but can be controlled... you really have no idea of the depth of the problem. Non-strict, overcommit-dependent logic is baked deep into the Linux kernel. There are plenty of ways to wedge the kernel where it will end up shooting down the processes that are supposed to be the most protected, sometimes shooting down much of the system. In the many cases people simply reboot the server entirely rather than wait around to see what the wreckage looks like. This is 1990s Windows revisited--reboot and move along, unreliability is simply the nature of computers....

the8472 7y ago
You can disable the oom killer if you think its heuristics are misfiring. > OOM killer shooting down processes even though there's plenty (e.g. >10%, tens of gigabytes) of unallocated memory because it can't evict the buffer cache quickly enough Is that due to dirty pages? Have you tried tweaking the dirty ratio to get dirty files flushed sooner? I believe there were also some improvements in the default behavior towards the end of the 4.x series that are supposed to result in more steady flushing.
- wahern 7y ago
  > You can disable the oom killer if you think its heuristics are misfiring. You can't completely disable the OOM killer. Some in-kernel allocations will trigger an OOM reap and kill regardless. Too many parts of Linux were written with the assumption of overcommit. Heck, as this issue shows even when there's technically free memory the OOM killer can still kick in. I had heard about these issues before but never saw that in practice--at least not often enough that I looked closely enough. I knew Google and Facebook and others have been relying on userspace OOM killers for years to improve reliability and consistency. But only after having recently joined a DevOps team at a large enterprise with large enterprise, big data customers did I begin to see the depth and breadth of the problem. Before getting into the hell-hole of helpless despair that is DevOps I primarily did systems programming. And, for the record, I always made sure to handle allocation failure in those languages where I could (C, Lua, etc) because as a software engineer it was always clear to me that, much as with security, an over reliance on policy-based mechanisms invariably resulted in the worst possible QoS. (Of course, in Linux even disabling overcommit is no guarantee that page fault won't trigger an OOM kill, but there are other techniques to improve reliability under load.) > Is that due to dirty pages? Yes, people have tweaked and turned those knobs and reduced incidence rate by maybe 20-30%. Yet nobody is sleeping better at night. Computing shouldn't be a black art. The irony is that overcommit and loose memory accounting more generally was originally intended to remove the necessity for professional system administrators to babysit a system to keep it humming along. But now we've come completely full circle. And it was entirely predictable. And before anyone says that cloud computing means you increase reliability by scaling horizontally, well "cloud scale" systems is exactly what are being run. But unless you massively over provision, and especially when certain tasks, even when split up across multiple nodes, can take minutes or even hours, an OOM kill incident can reverberate and cascade. Again, ironic because the whole idea of loose memory accounting is supposedly to obviate the need to over provision.

diamondo25 7y ago

Although I am not a fan of Apple, their Memory Pressure messages on iOS are really useful to prevent this. You get warned as a programmer to clean up your memory if possible, otherwise iOS shuts you down.

macdice 7y ago
Also AIX has SIGDANGER (best signal name ever). There is also "oomd", I think like the iOS thing, for Linux, in user space. Not sure how useful any of these things really are but wondering if FreeBSD should get one...
lclarkmichalek 7y ago
Which we now have on Linux :D https://facebookmicrosites.github.io/cgroup2/docs/pressure-metrics.html https://facebookmicrosites.github.io/cgroup2/docs/pressure-m...
gridlockd 7y ago
Yes, but as a result, memory-intensive applications (usually games) crash all the time, especially on older devices, which the App Store won't let you filter out. What are you going to anyway, just start deallocating the memory your application needs to actually work, so that you can display an "I'm sorry, we're out of memory" message? Might as well crash then. The amount of memory available also varies wildly from user to user and you can't put a memory requirement on an application. So what do people do? Disregard the issue. Even the best games on the App Store have tons of one star reviews by people with older devices. The flip side is that application that are prone to crash are pressured to optimize for relatively quick and painless restart.

popeye77 7y ago

I'd like to add that RHEL5 was more than tolerable in this aspect. Whatever kernel series it lived through, it seemed logical. We knew when RHEL5 systems started to swap that was our sign to A) look for long term usage growth that is beginning to bump some threshold somewhere B) someone turned a knob they shouldn't have C) a recent code change has caused a problem in the application stack Then RHEL6 came along, and it swapped all the time. Gone was our warning. The stats shows tens of gigabytes of cache engaged. WUT? How do we have tens of gigabytes of memory doing nothing? Before you could finish that thought, OOM killer was killing off programs due to memory pressure. WTF? The system is swallowing RAM to cache I/O, but couldn't spare a drop for programs? ...I could go on, but simply put, RHEL6 was garbage. And really I mean the RHEL6 kernels and the way that Red Hat tweaked whatever they did for it. RHEL7 was a little better, but still seeing echoes of the ugliness of RHEL6. RHEL5 was just a faded pleasant dream. The last 3 Fedoras on the other hand, the memory management seems like we're finally digging ourselves out of the nightmare. That nightmare lasted almost a full decade.... sheesh

trhway 7y ago

i routinely see these days systems without or with very low swap. It is like swap has become faux pas. That is especially strange giving the SSD drives available on the machines. Gradual degradation of service vs. the service sudden disappearance and/or stall or heavy overprovisioning and still ... Also comes to mind - while not generic swap - kind of edge case, a modern version of swap, ie. extending virtual memory space onto flash storage - Facebook replacement of some RAM with NVM https://research.fb.com/wp-content/uploads/2018/03/reducing-dram-footprint-with-nvm-in-facebook.pdf https://research.fb.com/wp-content/uploads/2018/03/reducing-...

IshKebab 7y ago
Yeah because swap on Linux doesn't really work. When it is needed your system grinds to a halt.
- Dylan16807 7y ago
  I strongly disagree with that characterization. At some amount of active memory use Linux will grind to a halt. This happens with or without swap. It's a problem of how it handles low memory situations. But if you have swap, this point comes much later. Often many gigabytes of unimportant data can be moved out of memory. If you can swap out 6 gigabytes without causing problems, and you only needed an extra 4 gigabytes of ram, then swap saves your day with almost no downside.
- trhway 7y ago
  i would venture a guess that probably you saw something like for example large JVM being partially swapped on HDD - in a situation like that it isn't swap per.se. which is at fault, it is the fact that JVM memory access pattern - randomly jumping all over - is extremely unsuitable for HDD based swap.
  - neel_k 7y ago
    Actually, there is a critically bad interaction between the way that garbage collection algorithms work and the LRU heuristic for swap. Tracing garbage collection algorithms work by doing a graph traversal that traces all the live pointers starting from the "root set" (ie, pointers in registers and on the stack) to find all the live objects. Now, note that (a) the root set are the hottest objects in memory, and (b) they are the ones that gc algorithms touch first. As a result, when you are doing a gc, the LRU (least-recently-used) heuristic for managing swap ends up putting all the hottest data in the program onto disk. This is pretty much the pessimal thing for the kernel to do, and can tank performance by triple-digit factors. However, this is entirely fixable! The gc and the kernel need to cooperate to figure out what is okay to push to disk or not. Matthew Hertz, Yi Feng and Emery Berger have a really nice PLDI 2005 paper, Garbage Collection without Paging, which shows how you can make gc performance hundreds of times better with just a little kernel assistance. However, because kernel developers do not program in garbage collected languages (they program in C, because that's what the kernel is written in), they don't understand garbage collection and are hostile to kernel changes to support it. Emery Berger told me they tried and failed to get the Linux kernel to support their changes, and people who worked on .NET told me they had similar experiences with the Windows kernel team.

lmilcin 7y ago

Well, the issue really is that applications are not being designed correctly. To create reliable applications you need to design them to work within limits of memory allocated to them. Unattended applications (like RDBMS or application container) should not allocate memory dynamically based on user input. By definition, reliable application will not fail because of external input. Allocating memory from OS cannot be done reliably unless that memory was set aside somehow. If the memory was not set aside then we say we are in overprovisioning situation and this means we accept that it might happen that a process wanting to allocate memory will fail to get it. So the solution (the simplest but not the only one) is to allocate the memory ahead of time for the maximum load the app can experience and to make a limit on the load (number of concurrent connections, etc.) so that this limit is never exceeded. Some systems do it halfway decently. For example Oracle will explicitly allocate all its memory spaces and then work within those. The OS really is a big heap of compromises. It does not know anything about the processes it runs but it is somehow expected to do the right thing. We see it is not behaving gracefully when memory runs out but the truth is, OS is built for the situation where it is being used mostly correctly. If memory is running out it means the user already made a mistake in desiging their use of resources (or didn't think about it at all) and there is not really much the OS can do to help it.

lclarkmichalek 7y ago
> Unattended applications (like RDBMS or application container) should not allocate memory dynamically based on user input. This is incredibly easy to say, and incredibly hard to do for any application of even moderate complexity. Getting people to accurately tune thread counts or concurrency limits is painful when working with diverse workloads, and getting a guaranteed upper bound on memory usage would be a lot harder. It's much more common to apply higher level techniques such as concurrency limits and load shedding to avoid OOMs and other resource starvation issues. The other thing we can do is reduce the impact of OOMs, and accept that they exist as a tail event. There is software running on your system that doesn't have hard memory limits, and you will always have to consider OOMs and other similar failure cases when ensuring your system is resilient. As long as we can prevent those tail events becoming a correlated failure across our fleet, we can get a pretty reliable system.

anticristi 7y ago

I feel that the problem is ill-posed: "At what amount of free RAM should we start freeing RAM by killing processes?" Maybe one should solve the problem: "At what eviction-induced latency should the OOM killer be invoked." Thanks to the infrastructure around latencytop, the latency might be available already. Of course, the never-ending dilemma of what process to kill is still there.

AstralStorm 7y ago
Preferably none. Saner APIs like Android will call low memory signals and start swapping out applications both ahead of time and as needed. It also has a relatively smart memory manager to detect unexpected vs expected spikes as well as slow leaks. POSIX has no such API. It was designed in a simple time.
- anticristi 7y ago
  Can you point to documentation on how Android deals with low-memory situations? My understanding is that: (A) They moved the problem to user-space. (B) Apps are supposed to constantly save the state they care about and expect to be killed whenever Android sees fit (or even at every app switch for developers).

known 7y ago

Unlike traditional swap, Android's Memory Manager kills inactive processes to free up memory

AstralStorm 7y ago
That's because the whole Android API and ecosystem was designed to make it possible. You cannot get away with that and POSIX applications. They have zero or bad session support. There's no global always available database to help you (Android has 4), nor event scheduling bus. (Dbus is a joke, it does not allow sending event to non-existing endpoint to be delivered later.) Every application using the standard mechanisms will get relaunched and activated as needed if it got shut down. It will receive a Bundle with state it managed to save and with original launch Intent. It can access an sqlite database to be made available via a restartable ContentProvider. SharedPreferences are also stored. Etc. In Linux world, there are no such de facto standards and what is most common is utterly broken. Same in Windows (only registry is persistent, it's not meant for data storage) and in OS X.

not2b 7y ago

The particular issue of locking up a system because there are too many tabs open in Firefox or Chrome should be fixed in the browser, because the browser is the program that is taking up all of the memory and is in the best position to know how to recover a lot of memory quickly while minimizing the loss of work for the user. One of the comments suggested having the kernel send a 'memory pressure' signal to all processes, and a process could catch the signal and do garbage collection or drop other storage. But even without such a feature, a program could do polling to determine that there is memory pressure, or do a check when a user asks to open a new tab, etc.

amyjess 7y ago

My experience for a long time has been that Linux is only usable with earlyoom. Running without it is a guaranteed way to end up having to power-cycle my system, often multiple times per day.

mighty_bander 7y ago

I managed to achieve this once, fresh from the world of Windows - missed breaking out of a loop while exploring Python(?) on Ubuntu 10 or so. Had to power cycle my damn machine.