15 ms·

Linux ate my RAM (2009)

245 points by StrauXX 3y ago

meepmorp 3y ago
[flagged]
julienpalard 3y ago
My advice in the situation when someone wants to "free RAM": "You bought it, better use it." It always felt strange that people buy lots of RAM but want it to be kept unused...
- freedomben 3y ago
  Well, usually you want to free it so you can use it for something else without hitting swap. At least that's my use case
  - lstodd 3y ago
    The whole point is that pagecache does not cause any swap hits. Oh my god, it's 2023 and we're still discussing this idea from 1970s. Is that so hard to grasp? No, stuff gets evicted from the cache long before you hit the swap, which is by the way measured by page swap-out/in rate and not by how much swap space is used, which is by itself a totally useless metric.
    - Dylan16807 3y ago
      > stuff gets evicted from the cache long before you hit the swap No...? I'm looking at a machine right now that has 3.7GB not swapped out and 1.2GB swapped out. Meanwhile the page cache has 26GB in it. Swapping can happen regardless of how big your page cache is. And how much you want swap to be used depends on use case. Sometimes you want to tune it up or down. In general the system will be good about dropping cache first, but it's not a guarantee. > measured by page swap-out/in rate and not by how much swap space is used Eh? I mean, the data got there somewhere. The rate was nonzero at some point despite a huge page cache. And usually when you want to specifically talk about the swap-out/in rate being too high, the term is "thrashing".
      - fsckboy 3y ago
        cached disk pages are not going to be swapped out, they're just freed (because these pages are already "out" in the same place swapland is) if your cached disk pages keep getting hit and are "recent", they're going to stay in, and your old untouched working pages are going to be swapped out, to make room either for new working pages because you've just loaded new programs or data, or to make room for more disk pages to be cached because your page cache accesses are "busier" than some of your working pages. you will swap out pages to make room for disk cache, but your cached disk pages will never be swapped out, they are just tossed (of course, after any dirty pages are written)
    - freedomben 3y ago
      > The whole point is that pagecache does not cause any swap hits. > Oh my god, it's 2023 and we're still discussing this idea from 1970s. > Is that so hard to grasp? No, stuff gets evicted from the cache long before you hit the swap, which is by the way measured by page swap-out/in rate and not by how much swap space is used, which is by itself a totally useless metric. Not everyone has been alive and into this stuff since the 1970s. That you and I know about this is irrelevant for the new people discovering it for the first time. There is always going to be a constant trickle in from new sources, for as long as it takes for the tech to go away. See relevant xkcd: https://xkcd.com/1053/ https://xkcd.com/1053/ But it's also worth pointing out that RAM/swap/page cache isn't always as simple as page cache out, RAM in. For example, this question[1] seems to indicate that things aren't as simple as you suggest. [1]: https://unix.stackexchange.com/questions/756990/why-does-my-system-heavily-use-swap-when-there-are-86-gb-of-ram-still-usable https://unix.stackexchange.com/questions/756990/why-does-my-...
      - lstodd 3y ago
        The page cache mechanism is very much alive. That's what we were discussing, is it not? I only lamented the fact that over 50 years the basics of how it works should have become common knowledge but did not. As for the link you provided, I do think I can get a system in a state like that, and that isn't even untrivial. To push firefox into swap, esp if you have just 8 gigs of it is.. simple. But it is not in any way a normal state of a system. Idk how the author got it in that state.
- outworlder 3y ago
  The issue is that they think they are reaching their system's capacity.
- jabroni_salad 3y ago
  Back when I played WoW I would occasionally run into issues with windows trying to put the game into memory compression, as opposed to basically any other process. It turned the game into a powerpoint. You could either get freaky with process explorer, or just keep some overhead so the system wouldn't try to do that. When I asked my guildies, they told me the 'default' for gaming is 16GB now, I was on 8 at the time. Pretty much every gamer will at some point tab out to process manager to see wtf the computer is trying to do and exactly zero of them will think to themselves "I'm so glad there is no wasted memory!"
  - spookie 3y ago
    For the 3rd paragraph, specifically: That's a fault with Windows not being clear enough with what is actually being in use, and what may be used and already there for a myriad of reasons. (edit: specified my intent on the reply)
- jstarfish 3y ago
  It's there for when you need to do something requiring that much memory. Your approach is like buying a giant house, becoming a hoarder, and trying to throw a party.
  - deleted 3y ago
    [deleted]
  - OrderlyTiamat 3y ago
    > Your approach is like buying a giant house, becoming a hoarder, and trying to throw a party. exactly, excepting that the items they're hoarding are occasionally very useful for making their day to day activities go faster. And the hoarder has the superpower that in the blink of an eye they can discard everything that's hoarded to make room for the party. Wait it isn't quite like a normal hoarder at all come to think of it!
- throwaway092323 3y ago
  Just because it's there, doesn't mean I want the same programs to use more.
- sznio 3y ago
  I just got a 64GB machine. It rarely sees much use, but I did go over 32GB a few times. Could've done away with less but I still have PTSD from all my applications crashing after I started Teams on my 16GB machine. On another note: Upgrading from i5-2500k to R7-5800X doesn't make Teams faster in any way.
skazazes 3y ago
Is this the reason Windows Task Manager seems to show Vmmem (WSL2) as gobbling up well more RAM then WSL seems to indicate is in use? I have more then enough RAM on my office workstation to just accept this, but on my personal gaming computer that moonlights as a dev machine, I run into issues and have to kill WSL from time to time.
- chabad360 3y ago
  No, that's because WSL (until v2/very recently) didn't properly release memory back to windows. This actually would cause docker to effectively leak memory really quickly.
- sp332 3y ago
  I think there is some conflict between the disk cache running inside WSL and the memory management outside. I tried turning up memory pressure in WSL but it didn't help. This does work but I have to run it manually from time to time: # sync; echo 3 > /proc/sys/vm/drop_caches
- praash 3y ago
  That's just one part of the issue - even after forcefully dropping Linux's caches, WSL has been unable to reclaim the memory back reliably. There has been a recent update that claims to finally fix this. You might find this package helpful: https://github.com/arkane-systems/wsl-drop-cache https://github.com/arkane-systems/wsl-drop-cache
  - Dylan16807 3y ago
    It's also really annoying that "drop caches" seems to be the only interface here. No way to simply limit it.
- deleted 3y ago
  [deleted]
- Zetobal 3y ago
  The worst offense of wsl2 is writing files to ram before copying it to the native filesystem unusable with lots of data.
mkhnews 3y ago
>> If applications want more memory, they just take it back from the disk cache. Q: If there is no swap configured, will a malloc() then take away clean page-cache pages ? Or does that happen only on page-in ?
- AnotherGoodName 3y ago
  malloc will take away from the disk cache. Fwiw without swap there isn't really any paging in or out (yes mmapped files technically still can but they are basically a special cased type of swap) so your question is hard to parse in this context. The disk cache is all about using unallocated memory and an allocation will reduce it. Paging is irrelevant here. Btw you should always enable swap. Without it you force all unused but allocated memory to live on physical RAM. Why would you want to do this? There's absolutely no benchmarks that show better performance with no swap. In fact it's almost always the opposite. Add some swap. Enjoy the performance boost! https://haydenjames.io/linux-performance-almost-always-add-swap-space/ https://haydenjames.io/linux-performance-almost-always-add-s...
  - dfox 3y ago
    I would say that for any modern unix implementation mmaped pages are quite significant, as all the read-only copies of libc code, other shared libraries and various mmaped data files (iconv tables, locales, terminfo, gettext catalogs…) are not exactly small.
    - robinsonb5 3y ago
      Which is why disabling swap in the hopes of preventing the system grinding to a halt on out-of-memory doesn't work, and actually makes things worse.
- dfox 3y ago
  In general, no and it will happen when there is something actually written to the page (which will cause a page fault and the kernel will have to somehow materialize the page). This works the same way regardless of how /proc/sys/vm/overcommit_memory is configured, the setting only affects how kernel tracks how much memory it is going to need in the future. (Obviously if we talk about malloc() this is a slight over-simplification as most malloc() implementations will write some kind of book-keeping structure and thus dirty some of the allocated pages) Whether swap is available is more or less irrelevant for this behavior. The only thing that swap changes is that kernel is then able to “clean” dirty anonymous pages by writing them out to swap.
_ea1k 3y ago
I once worked at a government job and took my computer into the IT department for an issue. I can't remember anything about the original issue. But I do vividly remember the help desk trying to figure out one last issue. Some process was consuming all my resources. They never could figure out why "System Idle Process" kept doing that.
- sweetjuly 3y ago
  A few years ago a similar issue cropped up on macOS where when the device was extremely hot (at least on intel), you'll see kernel_task seemingly using a ton of CPU time. What was actually happening is that the kernel was scheduling an idle thread which just put the CPU to sleep in an effort to lower the temperature beyond what it could achieve with fans and DVFS.
- justsomehnguy 3y ago
  Back in the day I was asked why it consumes 2 seconds of CPU Time each second. I couldn't answer that at time. Took a bit more years and understanding until I remembered that situation and it was obvious for me.
  - lynguist 3y ago
    What’s the answer? One second user time and one second kernel time? Or something completely different?
    - justsomehnguy 3y ago
      What @roelschroeven says. CPU Time is a metric of how much 'real time' was spent on the process, so a single thread running 100% time [on a single core with no HT] would give you a full 'real time' second. System idle 'consumes' all the idle CPU time so if the system is doing nothing it's get a second of CPU Time per execution thread. And there are multiple execution threads on almost anything later than 2005. On my T440 with i3-4010U (2 cores, 4 threads) the System Idle consumes ~3 seconds of CPU Time per second, because there are ton of shit running in the background so the system is never 100% idle.
flashback2199 3y ago
RAM isn't user friendly in Linux. Ubuntu Desktop is the most popular distro by far by Google Trends, but it doesn't even come with RAM compression set up out of the box, so as soon as you run out of memory the UI totally locks up until the task killer kills a process, which always takes minutes in my experience. Pop OS does come with RAM compression set up, which is Ubuntu based, but then you're stuck on xorg instead of Wayland right now, because they decided to make their own DE from scratch in Rust for some strange reason, which isn't available yet. You can set up RAM compression yourself, but when macOS and Windows both have it standard, coming to Linux as a newbie so you install Ubuntu Desktop and your whole system locks up as soon as you run out of physical RAM, it's really odd and unexpected. I'm not even sure who would want to run a desktop distro without RAM compression.
- yjftsjthsd-h 3y ago
  I'm pretty sure what you're calling RAM compression is swapping to zram, in which case the answer is that Some people prefer to not swap at all because that will still make things janky in comparison to just killing things when you're out of memory. (I would endorse earlyoom for that)
  - vlovich123 3y ago
    I’ve heard this position multiple times, and yet every single benchmark I’ve seen repeated by teams of engineers in multiple contexts fails to replicate this fear. Zswap really is something that should just always be enabled.
    - yjftsjthsd-h 3y ago
      How would you benchmark that?
      - vlovich123 3y ago
        For example, at Oculus they ran both performance benchmarks in a lab and collected feedback data from field telemetry. Now of course, it’s always possible some performance counter was overlooked / the degradation requires a specific workload to show, but the lack of ability to show any evidence of a difference implies that you probably are unlikely to see it given that the detractors were very vocal and engineering at big corps tends to be more stasis driven. I saw this also repeated at Apple (not Zswap since not Linux, but similar idea of compressing pages) and Android.
Tao3300 3y ago
I pity the fool who don't eat RAM.
- msla 3y ago
  I guess people around here are Too Young. This is a reference to a legitimate piece of Internet history: https://en.wikipedia.org/wiki/Ate_my_balls https://en.wikipedia.org/wiki/Ate_my_balls > "Ate my balls" is one of the earliest examples of an internet meme. It was widely shared in the late 1990s when adherents created web pages to depict a particular celebrity, fictional character, or other subject's zeal for eating testicles. Often, the site would consist of a humorous fictitious story or comic featuring edited photos about the titular individual; the photo editing was often crude and featured the character next to comic-book style speech in a thought balloon. > The fad was started in 1996 by Nehal Patel, a student at University of Illinois at Urbana-Champaign with a "Mr. T Ate My Balls" web page.
- meepmorp 3y ago
  It’s the first thing that popped into my head.
mjb 3y ago
There's some really interesting little details here. Linux, by default, is making the very reasonable assumption that the marginal cost of converting empty physical memory into caches and buffers is very near zero. This is fundamentally reasonable, because the cost of converting empty memory into used memory isn't really any cheaper than converting a clean cached page into used memory. It's a little more subtle when you take accounting into account, or when you think about dirty pages (which need to be written back to clear memory), or think about caches, but the core assumption is a very reasonable one. Except for on some multi-tenant infrastructure. Here, "empty" pages don't really exist. There's mostly not an empty page of memory kicking around waiting (like there is on client devices). Instead, nearly all the memory on the box is allocated, but each individual guest kernel doesn't know the full allocation. In this world, the assumption that the marginal cost of converting empty to full is zero is no longer true. There's some real cost. Projects like DAMON https://sjp38.github.io/post/damon/ https://sjp38.github.io/post/damon/ exist to handle this case, and similar cases where keeping empty memory rather than low-value cache is worse for the overall system. These kinds of systems aren't super common, especially on the client side, but aren't unusual in large-scale cloud services.
- mjb 3y ago
  The other interesting detail here is the memory sizing problem. If I can consume all my RAM with caches and buffers, how much RAM do I need? The answer (as always) depends on what you're optimizing for. For performance, bigger is better. For cost, energy, etc you're going to want some way to calculate whether adding more RAM (and so having bigger caches) is worth the cost, heat, power, etc. Gray and Putzolu's classic "The 5 minute rule for trading memory for disc accesses" (https://dl.acm.org/doi/pdf/10.1145/38713.38755 https://dl.acm.org/doi/pdf/10.1145/38713.38755) from 1987 is probably one of the most important CS systems papers of all time. In it, they lay out a way of thinking about memory and cache sizing by comparing the cost of holding cache to the cost of access (this isn't the first use of that line of thinking, but is a very influential statement of it). Back then, they found that storing 4kB in RAM for 5 minutes costs about the same as reading it back from storage. So if you're going to access something again within 5 minutes you should keep it around. The constants have change a lot (RAM is way cheaper, IOs are way cheaper, block sizes are typically bigger) since then, but the logic and way of thinking are largely timeless. The 5 minute rule is a quantitative way of thinking about the size of the working set, an idea that dates back at least to 1968 and Denning's "The working set model for program behavior" (https://dl.acm.org/doi/10.1145/363095.363141 https://dl.acm.org/doi/10.1145/363095.363141). Back to marginal costs - the marginal cost of converting empty RAM to cache is zero in the minute, but only because the full cost has been borne up front when the machine is purchased. It's not zero, just pre-paid.
bminor13 3y ago
Does anyone happen to have expertise/pointers on how ZFS' ARC interacts with Linux disk caching currently when using ZFS-on-Linux? It seems like the ARC space shows up as "used" despite being in a similar category of "made available if needed" - is that correct? Is data in the ARC double-cached by Linux's disk caching mentioned in the post? If so, is it possible to disable this double-caching somehow?
- MrDrMcCoy 3y ago
  ZFS ARC unfortunately does not integrate with the kernel file cache, so they step on each other a lot. ZFS does watch available system RAM and try to dynamically reduce its usage as memory pressure increases, but I've found its responsiveness for this to be far too slow. This combined with how ARC appears to just be an opaque block of RAM that cannot be reclaimed, I usually just set a hard limit on how big the ARC is allowed to get in the module load arguments and be done with it (at least for systems that are doing more than just storage).
  - drewg123 3y ago
    Is ARC really non-reclaimable on Linux? At least on FreeBSD, there is a kmem_cache_reap() that is called from the core kernel VM system's low memory handlers. Looking at the linux code in openzfs, it looks like there is an "spl_kmem_cache_reap_now()" function. Maybe the problem is the kernel dev's anti-ZFS stance, and it can't be hooked into the right place (eg, the kernel's VM low memory handling code)?
    - MrDrMcCoy 3y ago
      It's reclaimable, but opaque. The ARC just looks like used RAM rather than file cache, which throws off various means of accounting.
    - nubinetwork 3y ago
      echo 3 > /proc/sys/vm/drop_caches (Bear in mind that 3 is the most aggressive but other than exporting the pool, it's the only way to dump the cache, especially if you boot off ZFS)
therealmarv 3y ago
Windows and Mac use compressed RAM for many many years as standard. Yet on many Linux desktop you have to activate it (namely ZRAM). It solves the problem that a e.g. browser eats all your memory. It's much quicker than Swap and yet mostly unknown by many people who are running a Linux desktop. As mentioned by another user it's still not standard on Ubuntu desktop and I don't understand why.
- spookie 3y ago
  There are distributions that enable it by default, Fedora comes to mind.
- viraptor 3y ago
  > It solves the problem that a e.g. browser eats all your memory. It doesn't solve that. You get a little bit more headroom, but that's it. Not much ram is considered compressible anyway. On my Mac I'm barely reaching 10% of compressed memory anyway, so it doesn't make that much difference.
  - noisem4ker 3y ago
    Current RAM usage from my Windows 10 dev machine, as reported by Task Manager: > In use: 18028 MB > In use, compressed: 2718 MB > Compressed memory stores an estimated 9013 MB of data, saving the system 6294 MB of memory. That's not a small amount.
  - jowea 3y ago
    My experience is that zram saves quite a bit. I have another old laptop with 4GB where it's essential. Maybe it differs by program type? NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT /dev/zram0 lzo-rle 15,6G 1,9G 248,6M 418M 16 [SWAP]
    - viraptor 3y ago
      Yup, it will depend on your workload a lot. Worth testing of course!
kevin_nisbet 3y ago
When I worked in telco we used to run into this a lot. We'd demand standard alarms for things like memory leaks / out of memory conditions / high than normal memory usage, as to get 99.999% uptime we want to be paged when problems like this would occur. Except a bunch of platform did the extremely naive implementation and included recoverable memory in their alarm conditions. So inevitably someone would log in and grep the logs or copy some files to the system, and hit the alarm conditions. And there were some vendors who really didn't want to fix it, they would argue that recoverable memory is in use, so it should really be part of that alarm condition.
zaptrem 3y ago
> Disk caching makes the system much faster and more responsive! There are no downsides, except for confusing newbies. It does not take memory away from applications in any way, ever! No downsides except for massive data loss when the system suddenly loses power/a drive crashes and the massive theft of memory from host OSes (e.g., when using Windows Subsystem for Linux).
- nightfly 3y ago
  There is no data loss from disk caching.
  - zaptrem 3y ago
    Does disk caching not include write caching?
    - nightfly 3y ago
      Not in the context we're talking about
      - hifromwork 3y ago
        Are you sure? Dirty pages have to reside somewhere, so they are actually stuck in RAM until successfully written RAM. Linux will lie with straight face that dd to my 8gb pendrive finished successfully in a few seconds, so there may be non-trivial amounts of RAM involved here. I don't know enough of Linux internals to know if the writeback cache and read cache are the same object in the kernel, but they feel similar. Of course the real response is that without write cache (effectively adding fsync to every write) any modern linux system will grind to absolute halt and doing anything would be a challenge. So contray to GP's post, it's not reasonable to complain about it's existence.
        withinboredom 3y ago
        It's written asap to disk unless you configure it otherwise. The disk (or flash disk, in your case) has its own ram before actually flushing to physical storage (fsync, unless it lies, in which case there is probably an article about it here cough macs cough). So the kernel isn't lying to you, but your disk probably is.
__turbobrew__ 3y ago
> Disk cache can always be given back to applications immediately This is not true, there is a cost to freeing the cache pages and allocating them to the other program. I have seen some very regressive performance patterns around pages getting thrashed back and forth between programs and the page cache, especially in containers which are memory limited. You throw memory maps into the mix and things can get really bad really fast.
- blueflow 3y ago
  Can you elaborate further?
  - __turbobrew__ 3y ago
    There is a fast path and slow path to reclaiming cached pages. There are ways to measure the impact using ebpf probes. If you are interested in going deeper I recommend looking at the memory section of this book: https://www.brendangregg.com/blog/2020-07-15/systems-performance-2nd-edition.html https://www.brendangregg.com/blog/2020-07-15/systems-perform...
- mnw21cam 3y ago
  That's why Linux does keep a certain amount of RAM actually-free, so that it can hand over some pages immediately. If the amount of actually-free RAM goes below a certain amount, then it will pre-emptively evict a load of cache pages.
lxe 3y ago
This made me chase red herrings when debugging oom issues in production. Wish free would just remove the 'free' column and replaced it with 'available'.
- tetha 3y ago
  This is what we did in pretty much all of our monitoring some time ago. We ripped out most memory graphs except for "Total Memory" and "Available Memory" as well as memory pressure from the PSI metrics. And we placed alerts on available memory growing low, as well as pages being swapped in. Newer kernel opportunistically swap-out idle pages, but that's fine as long as you never see the path from disk to memory (swap-in). This has increased the quality of our memory monitoring by a lot.
drewg123 3y ago
I used to run a Linux workstation in the late 00's (sorry FreeBSD folks, I know, the shame...), and I ran a closed source PVR application on it. The memory access pattern was pretty much pessimal for my use of the box as a workstation. I'd use it from 7am -> 8/9pm every day, then when I'd walk away from the keyboard, I'd watch HD recordings (which could be 7GB or more per hour). Those would get cached in memory, and eventually my workstation stuff (emacs, xterms, firefox, thunderbird) would start to get paged out. In the mornings, it was painful to start using each application, as it waited forever to page in from a spinning disk. I eventually wrote an LD_PRELOAD for the DVR software that overloaded open, and added O_DIRECT (to tell the kernel not to cache the data). This totally solved my problem, and didn't impact my DVR usage at all.
- toast0 3y ago
  > I used to run a Linux workstation in the late 00's (sorry FreeBSD folks, I know, the shame...), and I ran a closed source PVR application on it. It's ok, no shame. But as I understand it, FreeBSD would prefer to throw out (clean) disk cache pages under memory pressure until somewhere around FreeBSD 11 +/- 1, where there were a few changes that combined to make things like you described likely to happen. Heavy I/O overnight might still have been enough, and I'm not going to test run an old OS version to check ;) I can't find the changes quickly, but IIRC, older FreeBSD didn't mark anonymous pages as inactive unless there was heavy memory pressure; when there was mild memory pressure, it would go through the page queue(s) and free clean disk pages and skip other page; only taking action on a second pass if the first pass didn't clean enough. This usually meant your program pages would stay in memory, but when you hit memory pressure, there would be a big pause to mark a lot of pages inactive, often too many pages, which would then get faulted back to active... Current FreeBSD marks pages inactive on a more consistent basis, which is nice because when there is memory pressure, chancws are there's already classified pages. But it can lead to anonymous pages getting swapped out in favor of disk pages as you described; it's all tunable, of course, but it was a kind of weird transition for me. After upgrading the OS, some of my heavy i/o machines saw rising swap usage running the same software as before; took a while to figure that out.
  - 3y ago
loktarogar 3y ago
This feels like a UX problem. If this is a normal and expected part of linux operation, it should be called out in the {T,G}UI.
- the8472 3y ago
  But it does. https://files.catbox.moe/l9je82.png https://files.catbox.moe/l9je82.png orange is the part used by caches
  - loktarogar 3y ago
    I'm not a linux user, so not an observation of experience. Just the existence of this website suggests to me that however it is being done right now could be made clearer somehow.
- nightfly 3y ago
  htop shows this
- gruez 3y ago
  It is. Windows does the same thing, but it's a non-issue because task manager makes it look like cached memory is free memory.
- bityard 3y ago
  1. You can't change `free` output, you'll break SO many scripts. 2. Most things which report memory usage in a user-friendly way _already_ do this in an obvious way. (Htop shows disk cache in a bar graph, but doesn't add it to the "used" counter.) 3. Should UX always compensate for some fraction of users' misunderstanding of how their OS kernel works? Or would it be better for them to ask the question and then be educated by the answer?
  - loktarogar 3y ago
    > Or would it be better for them to ask the question and then be educated by the answer? Good UX makes the question "why is linux using my unused RAM for disk caching" (a non pressing question) instead of "why is linux eating up all my RAM" (panic, stressful question)
neurostimulant 3y ago
> If your applications want more memory, they just take back a chunk that the disk cache borrowed. Disk cache can always be given back to applications immediately! You are not low on ram! I'm running RKE2 on my desktop and it'll start killing pods due to low memory pressure, even though the memory was only used for disk caching. I wonder if there is any way to make it stop doing that and instead only start killing pods if it's due to "real" low memory pressure.
- blueflow 3y ago
  Think about it: A processes executable code comes from a file. You will need the size of the executable available as disk cache or the program execution will cause heavy thrashing and I/O. So some part of it is "real" memory pressure.
  - neurostimulant 3y ago
    I also run an nfs server in the same machine, so after a period of heavy nfs use, most of the ram were eaten by the disk cache and rke2/kubernetes start having memory pressure taint. After a fresh restart with all pods running, the memory usage is below 10%, so I doubt the disk cache was full with executable files cache.
- okwhateverdude 3y ago
  Assuming Linux, oddly enough I came across this exact behavior[0] while researching resource management for an on-prem k8s cluster. Take a look at that thread for more info, but TL;DR, you need to actually finesse page cache constraints if you want avoid the behavior. You actually can have really fine grained control over page cache via cgroups v2[1][2] and systemd[3]. [0]: https://github.com/kubernetes/kubernetes/issues/43916 https://github.com/kubernetes/kubernetes/issues/43916 [1]: https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-interface-files https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-in... [2]: https://biriukov.dev/docs/page-cache/6-cgroup-v2-and-page-cache/ https://biriukov.dev/docs/page-cache/6-cgroup-v2-and-page-ca... [3]: https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html https://www.freedesktop.org/software/systemd/man/systemd.res...
jenadine 3y ago
How old is this website? It's from a time when a typical computer only had 1.5 G of ram.
- I_Am_Nous 3y ago
  The domain appears to have been registered 25 Apr 2009, and I remember seeing this quite a while ago. That would make sense for 1.5 G of RAM being typical. Glad it's still around :)
- Pathogen-David 3y ago
  Oldest copy on the Wayback Machine is from May 2009 https://web.archive.org/web/20090513043445/https://www.linuxatemyram.com/ https://web.archive.org/web/20090513043445/https://www.linux...
burnte 3y ago
Unused RAM is wasted RAM. Why people want to see GOBS of empty RAM boggles my mind.
- filchermcurr 3y ago
  I think the disconnect is not understanding how the RAM is used. If the average user looks and sees all of their RAM in use, they're going to think that there's no more room for the applications that they want to launch. They don't understand that what's cached will just get out of the way when the memory is actually needed. So they want to see free RAM, because that means it's free for their game or millions of tabs.
wingworks 3y ago
htop shows this (it's the orange/yellow bar in RAM)
- withinboredom 3y ago
  I recommend btop these days (https://github.com/aristocratos/btop https://github.com/aristocratos/btop)
  - wingworks 3y ago
    I just tried btop, it has crazy colours (nothing like screenshots) on my Mac Terminal app (Misterioso theme).
avgcorrection 3y ago
Trust me. If Linux really eats your RAM to the point of reaching an OOM state you will know. (This was of course because of having too many apps relative to my RAM. Not because of disk caching.) The OOM behavior is not pleasant for a desktop system.
- berkes 3y ago
  I've never encountered this with "many apps" starting to OOM, but many times with one process OOMing. That one will simply crash and everything else continues to run unharmed.
  - mbakke 3y ago
    What distribution are you using? IME, if a process grows out of control, Linux won't notice until the whole system is thrashing, at which point it's too late and it tries killing random things like browser tabs way before the offending process. In rare cases Linux might recover, but only because I hammered C-c the right place 30 minutes ago. In most cases a hard reboot is required (I left it overnight once, fans spinning at max, hoping the kernel would eventually accept my plea for help, but she had other priorities).
    - jltsiren 3y ago
      I guess OOM is more problematic on low-memory systems or when you have more than a nominal amount of swap. If you have enough memory that the desktop environment, browsers, and other processes that should keep running only use a small fraction of it, the OOM killer can pick a reasonable target reliably. A process that tries to allocate too much memory gets killed, and everything is robust and deterministic. I sometimes trigger OOM several times in an hour, for example when trying to find reasonable computational parameters for something.
      - Sakos 3y ago
        How much memory do you think is reasonable? I've had it happen to me with 16GB and even 32GB, where I never ever have this issue on Windows (unless for some reason I'm on a 2GB RAM system for God knows why). I wish people would stop defending pathological behavior that's broken for standard desktop use. What's wrong with wanting things to improve?
Scarbutt 3y ago
It's weird that their 'used' column is accounting for 'buff/cache' 'man free' states used Used or unavailable memory (calculated as total - available)
mkhnews 3y ago
Another question is about containers and memory limits. Does the page-cache count against my container memory limit ? And if so, then when I hit that limit from doing many reads, does the page-cache start taking from itself without OOM killer getting involved ?
- deleted 3y ago
  [deleted]
- defer 3y ago
  I also want to know this, but in reverse. I build older android (the OS) versions inside docker containers because they have dependencies on older glibc versions. This is a memory-heavy multi-threaded process and the OOM killer will kill build threads, making my build fail. However, there is plenty of available (but not free) memory in the docker host, but apparently not available in the container. If I drop caches on the host periodically, the build generally succeeds.
  - mkhnews 3y ago
    And perhaps k8s is a specific category to consider here. I've read and thought I've experienced where 'active' (as opposed to in-active) page-cache does count towards k8s mem limit.
- otterley 3y ago
  1. Pages cached by applications are charged to its container for the purpose of memory resource limits. 2. IME the kernel takes the container's memory limit into account when determining whether to allocate a page for cache. Caching, by itself, won't cause the container to exceed a memory limit.
mavhc 3y ago
My Linux ram problems are 1: Ubuntu default install, ends up with xwayland using 5GB ram. 2: when running out of ram it seems to default to crashing back to the logon screen
Karunamon 3y ago
Okay, if unused memory is wasted and there are truly no consequences for the "free" column reading zero, then why on a busy system do I get UI chugging and otherwise poor (bordering on unusable) performance under this condition that is immediately resolved by forcing the caches to drop and freeing multiple gigabytes? Whatever conceivable speedup there is from 12 GB of file cache as opposed to 11 is obliterated multiple times over from the time lost by having to do this dance, or worse, recovering after the oom killer wipes out my X session or browser.
- AnotherGoodName 3y ago
  >recovering after the oom killer wipes out my X session or browser. Perhaps you can share more details of what you're doing to force the cache to drop and what the side effects are exactly because an OOM can't be caused by the file cache since the total free memory available to applications remains the same. The whole point of the file cache is to use otherwise unallocated memory and give it up the moment it's needed. There should not be an OOM from this short of an OS bug or an over allocated virtualized system.
  - Karunamon 3y ago
    echo 3 > /proc/sys/vm/drop_caches Last time I ran into this was a couple of years ago on a stock Arch system. (Disabilities forced me back to Windows). Every time, the largest memory consumer was the web browser. Also every time, the system became nearly unresponsive due to swap thrashing (kswapd at the top of the CPU usage list, most of which was I/O wait). Last time I complained about this problem, someone suggested installing zram which did stop it from happening. However, this does not change the fact that there is some pathological failure case that contradicts the central thesis (not to mention, smug tone) of this website and makes searching for solutions to the problem infuriating.
- PhilipRoman 3y ago
  I find that task priorities in general are not strict enough under Linux. Try running a cpu heavy but very low priority task in background and it still manages to measurably affect the latency of the more important process. And this is just the CPU, not to mention disk utilization and network usage. I was too lazy to find a proper solution, so I just used mlockall after allocating a massive heap and pin the process to a core that is reserved only for this specific purpose. I think cgroups has very flexible tools for reserving system wide resources, but haven't had the time to test it yet.
HippoBaro 3y ago
I think the information there is valuable because questions about memory usage in Linux keep coming up. The answer: "don't worry about it," is probably a good starting point. The page claims things that are just really misleading, though. > There are no downsides, except for confusing newbies. False. Populating the page cache involves lots of memory copies. It pays off if what's written is read back many times; otherwise, it's a net loss. It also costs cycles and memory to keep track of all these pages and maintain usage statistics so we know what page should be kept and which can be discarded. Unfortunately, Linux makes quantifying that cost hard, so it is not well understood. > You can't disable disk caching. The only reason anyone ever wants to disable disk caching is because they think it takes memory away from their applications, which it doesn't! People do want that, and they do turn it off. It's probably the number one thing database people do because they want domain-specific caching in userland and use O_DIRECT to bypass the kernel caches altogether. If you don't, you end up caching things twice, which is efficient/redundant.
GuB-42 3y ago
I remember that being the case for early versions of Android, people were surprised all their RAM was used, and of course, we could find apps that "freed" the RAM, generally making things worse. And the response was similar: all that "used" RAM can be reclaimed at any time should an app need some, but in the meantime, the system (which is Linux) might as well use it. I think they "fixed" it in later versions. I don't know how, but I suspect they just changed the UI to stop people from complaining and downloading counterproductive apps. As usual in these situations, unless you really know what you are doing, let the system do its job, some of the best engineers with good knowledge of the internals have worked on it, you won't do better by looking at a single number and downloading random apps. For RAM in particular, because of the way virtual memory works, it is hard to get an idea of what is happening. There are caches, shared memory, mapped files, in-app allocators, etc...
gslin 3y ago
https://archive.ph/WmJHf https://archive.ph/WmJHf
- neonate 3y ago
  https://web.archive.org/web/20230928045611/https://www.linuxatemyram.com/ https://web.archive.org/web/20230928045611/https://www.linux...
404mm 3y ago
You can always just download more RAM. https://www.downloadmoreram.com https://www.downloadmoreram.com
simonblack 3y ago
Why all the fuss?? When you can download as much RAM as you want, any time you want. https://downloadmoreram.com/ https://downloadmoreram.com/
lnyng 3y ago
We published this paper "TMO: Transparent Memory Offloading in Datacenters" last year which covers some Linux memory management mechanisms that may be quite useful for providing reasonable estimations to application memory usage. We observed that the real memory footprint for applications depends on many factors: file access pattern, disk IO speed (especially if swap is enabled), ssd vs hdd, application latency sensitivity, etc. Instead of coming up with some overly complicated heuristic, we use the Linux kernel provided memory.pressure [0] metric via cgroup v2. It measures the amount of time spent waiting for memory (page fault etc). Then by slowly reclaiming memory from the application until its memory pressure hits some target (say 0.1%), we can claim that the steady state usage is the actual memory footprint. This may not be useful for PC but could be very useful for data center to track memory regression, and also to harvest disk swap without concerning too much about the cliff effect when the host runs out of memory and suddenly kernel pushes everything to swap space. [0] https://facebookmicrosites.github.io/cgroup2/docs/pressure-metrics.html https://facebookmicrosites.github.io/cgroup2/docs/pressure-m...
fuzztester 3y ago
Windows ate my hard disk (every year).
deleted 3y ago
[deleted]
thiag0 3y ago
Is the website down? I was reading and suddenly started to get timeout.

245 points by StrauXX 3y ago

[flagged]

My advice in the situation when someone wants to "free RAM": "You bought it, better use it." It always felt strange that people buy lots of RAM but want it to be kept unused...

freedomben 3y ago
Well, usually you want to free it so you can use it for something else without hitting swap. At least that's my use case
- lstodd 3y ago
  The whole point is that pagecache does not cause any swap hits. Oh my god, it's 2023 and we're still discussing this idea from 1970s. Is that so hard to grasp? No, stuff gets evicted from the cache long before you hit the swap, which is by the way measured by page swap-out/in rate and not by how much swap space is used, which is by itself a totally useless metric.
  - Dylan16807 3y ago
    > stuff gets evicted from the cache long before you hit the swap No...? I'm looking at a machine right now that has 3.7GB not swapped out and 1.2GB swapped out. Meanwhile the page cache has 26GB in it. Swapping can happen regardless of how big your page cache is. And how much you want swap to be used depends on use case. Sometimes you want to tune it up or down. In general the system will be good about dropping cache first, but it's not a guarantee. > measured by page swap-out/in rate and not by how much swap space is used Eh? I mean, the data got there somewhere. The rate was nonzero at some point despite a huge page cache. And usually when you want to specifically talk about the swap-out/in rate being too high, the term is "thrashing".
    - fsckboy 3y ago
      cached disk pages are not going to be swapped out, they're just freed (because these pages are already "out" in the same place swapland is) if your cached disk pages keep getting hit and are "recent", they're going to stay in, and your old untouched working pages are going to be swapped out, to make room either for new working pages because you've just loaded new programs or data, or to make room for more disk pages to be cached because your page cache accesses are "busier" than some of your working pages. you will swap out pages to make room for disk cache, but your cached disk pages will never be swapped out, they are just tossed (of course, after any dirty pages are written)
  - freedomben 3y ago
    > The whole point is that pagecache does not cause any swap hits. > Oh my god, it's 2023 and we're still discussing this idea from 1970s. > Is that so hard to grasp? No, stuff gets evicted from the cache long before you hit the swap, which is by the way measured by page swap-out/in rate and not by how much swap space is used, which is by itself a totally useless metric. Not everyone has been alive and into this stuff since the 1970s. That you and I know about this is irrelevant for the new people discovering it for the first time. There is always going to be a constant trickle in from new sources, for as long as it takes for the tech to go away. See relevant xkcd: https://xkcd.com/1053/ https://xkcd.com/1053/ But it's also worth pointing out that RAM/swap/page cache isn't always as simple as page cache out, RAM in. For example, this question[1] seems to indicate that things aren't as simple as you suggest. [1]: https://unix.stackexchange.com/questions/756990/why-does-my-system-heavily-use-swap-when-there-are-86-gb-of-ram-still-usable https://unix.stackexchange.com/questions/756990/why-does-my-...
    - lstodd 3y ago
      The page cache mechanism is very much alive. That's what we were discussing, is it not? I only lamented the fact that over 50 years the basics of how it works should have become common knowledge but did not. As for the link you provided, I do think I can get a system in a state like that, and that isn't even untrivial. To push firefox into swap, esp if you have just 8 gigs of it is.. simple. But it is not in any way a normal state of a system. Idk how the author got it in that state.
outworlder 3y ago
The issue is that they think they are reaching their system's capacity.
jabroni_salad 3y ago
Back when I played WoW I would occasionally run into issues with windows trying to put the game into memory compression, as opposed to basically any other process. It turned the game into a powerpoint. You could either get freaky with process explorer, or just keep some overhead so the system wouldn't try to do that. When I asked my guildies, they told me the 'default' for gaming is 16GB now, I was on 8 at the time. Pretty much every gamer will at some point tab out to process manager to see wtf the computer is trying to do and exactly zero of them will think to themselves "I'm so glad there is no wasted memory!"
- spookie 3y ago
  For the 3rd paragraph, specifically: That's a fault with Windows not being clear enough with what is actually being in use, and what may be used and already there for a myriad of reasons. (edit: specified my intent on the reply)
jstarfish 3y ago
It's there for when you need to do something requiring that much memory. Your approach is like buying a giant house, becoming a hoarder, and trying to throw a party.
- deleted 3y ago
  [deleted]
- OrderlyTiamat 3y ago
  > Your approach is like buying a giant house, becoming a hoarder, and trying to throw a party. exactly, excepting that the items they're hoarding are occasionally very useful for making their day to day activities go faster. And the hoarder has the superpower that in the blink of an eye they can discard everything that's hoarded to make room for the party. Wait it isn't quite like a normal hoarder at all come to think of it!
throwaway092323 3y ago
Just because it's there, doesn't mean I want the same programs to use more.
sznio 3y ago
I just got a 64GB machine. It rarely sees much use, but I did go over 32GB a few times. Could've done away with less but I still have PTSD from all my applications crashing after I started Teams on my 16GB machine. On another note: Upgrading from i5-2500k to R7-5800X doesn't make Teams faster in any way.

skazazes 3y ago

Is this the reason Windows Task Manager seems to show Vmmem (WSL2) as gobbling up well more RAM then WSL seems to indicate is in use? I have more then enough RAM on my office workstation to just accept this, but on my personal gaming computer that moonlights as a dev machine, I run into issues and have to kill WSL from time to time.

chabad360 3y ago
No, that's because WSL (until v2/very recently) didn't properly release memory back to windows. This actually would cause docker to effectively leak memory really quickly.
sp332 3y ago
I think there is some conflict between the disk cache running inside WSL and the memory management outside. I tried turning up memory pressure in WSL but it didn't help. This does work but I have to run it manually from time to time: # sync; echo 3 > /proc/sys/vm/drop_caches
praash 3y ago
That's just one part of the issue - even after forcefully dropping Linux's caches, WSL has been unable to reclaim the memory back reliably. There has been a recent update that claims to finally fix this. You might find this package helpful: https://github.com/arkane-systems/wsl-drop-cache https://github.com/arkane-systems/wsl-drop-cache
- Dylan16807 3y ago
  It's also really annoying that "drop caches" seems to be the only interface here. No way to simply limit it.
deleted 3y ago
[deleted]
Zetobal 3y ago
The worst offense of wsl2 is writing files to ram before copying it to the native filesystem unusable with lots of data.

mkhnews 3y ago

>> If applications want more memory, they just take it back from the disk cache. Q: If there is no swap configured, will a malloc() then take away clean page-cache pages ? Or does that happen only on page-in ?

AnotherGoodName 3y ago
malloc will take away from the disk cache. Fwiw without swap there isn't really any paging in or out (yes mmapped files technically still can but they are basically a special cased type of swap) so your question is hard to parse in this context. The disk cache is all about using unallocated memory and an allocation will reduce it. Paging is irrelevant here. Btw you should always enable swap. Without it you force all unused but allocated memory to live on physical RAM. Why would you want to do this? There's absolutely no benchmarks that show better performance with no swap. In fact it's almost always the opposite. Add some swap. Enjoy the performance boost! https://haydenjames.io/linux-performance-almost-always-add-swap-space/ https://haydenjames.io/linux-performance-almost-always-add-s...
- dfox 3y ago
  I would say that for any modern unix implementation mmaped pages are quite significant, as all the read-only copies of libc code, other shared libraries and various mmaped data files (iconv tables, locales, terminfo, gettext catalogs…) are not exactly small.
  - robinsonb5 3y ago
    Which is why disabling swap in the hopes of preventing the system grinding to a halt on out-of-memory doesn't work, and actually makes things worse.
dfox 3y ago
In general, no and it will happen when there is something actually written to the page (which will cause a page fault and the kernel will have to somehow materialize the page). This works the same way regardless of how /proc/sys/vm/overcommit_memory is configured, the setting only affects how kernel tracks how much memory it is going to need in the future. (Obviously if we talk about malloc() this is a slight over-simplification as most malloc() implementations will write some kind of book-keeping structure and thus dirty some of the allocated pages) Whether swap is available is more or less irrelevant for this behavior. The only thing that swap changes is that kernel is then able to “clean” dirty anonymous pages by writing them out to swap.

_ea1k 3y ago

I once worked at a government job and took my computer into the IT department for an issue. I can't remember anything about the original issue. But I do vividly remember the help desk trying to figure out one last issue. Some process was consuming all my resources. They never could figure out why "System Idle Process" kept doing that.

sweetjuly 3y ago
A few years ago a similar issue cropped up on macOS where when the device was extremely hot (at least on intel), you'll see kernel_task seemingly using a ton of CPU time. What was actually happening is that the kernel was scheduling an idle thread which just put the CPU to sleep in an effort to lower the temperature beyond what it could achieve with fans and DVFS.
justsomehnguy 3y ago
Back in the day I was asked why it consumes 2 seconds of CPU Time each second. I couldn't answer that at time. Took a bit more years and understanding until I remembered that situation and it was obvious for me.
- lynguist 3y ago
  What’s the answer? One second user time and one second kernel time? Or something completely different?
  - justsomehnguy 3y ago
    What @roelschroeven says. CPU Time is a metric of how much 'real time' was spent on the process, so a single thread running 100% time [on a single core with no HT] would give you a full 'real time' second. System idle 'consumes' all the idle CPU time so if the system is doing nothing it's get a second of CPU Time per execution thread. And there are multiple execution threads on almost anything later than 2005. On my T440 with i3-4010U (2 cores, 4 threads) the System Idle consumes ~3 seconds of CPU Time per second, because there are ton of shit running in the background so the system is never 100% idle.

flashback2199 3y ago

RAM isn't user friendly in Linux. Ubuntu Desktop is the most popular distro by far by Google Trends, but it doesn't even come with RAM compression set up out of the box, so as soon as you run out of memory the UI totally locks up until the task killer kills a process, which always takes minutes in my experience. Pop OS does come with RAM compression set up, which is Ubuntu based, but then you're stuck on xorg instead of Wayland right now, because they decided to make their own DE from scratch in Rust for some strange reason, which isn't available yet. You can set up RAM compression yourself, but when macOS and Windows both have it standard, coming to Linux as a newbie so you install Ubuntu Desktop and your whole system locks up as soon as you run out of physical RAM, it's really odd and unexpected. I'm not even sure who would want to run a desktop distro without RAM compression.

yjftsjthsd-h 3y ago
I'm pretty sure what you're calling RAM compression is swapping to zram, in which case the answer is that Some people prefer to not swap at all because that will still make things janky in comparison to just killing things when you're out of memory. (I would endorse earlyoom for that)
- vlovich123 3y ago
  I’ve heard this position multiple times, and yet every single benchmark I’ve seen repeated by teams of engineers in multiple contexts fails to replicate this fear. Zswap really is something that should just always be enabled.
  - yjftsjthsd-h 3y ago
    How would you benchmark that?
    - vlovich123 3y ago
      For example, at Oculus they ran both performance benchmarks in a lab and collected feedback data from field telemetry. Now of course, it’s always possible some performance counter was overlooked / the degradation requires a specific workload to show, but the lack of ability to show any evidence of a difference implies that you probably are unlikely to see it given that the detractors were very vocal and engineering at big corps tends to be more stasis driven. I saw this also repeated at Apple (not Zswap since not Linux, but similar idea of compressing pages) and Android.

Tao3300 3y ago

I pity the fool who don't eat RAM.

msla 3y ago
I guess people around here are Too Young. This is a reference to a legitimate piece of Internet history: https://en.wikipedia.org/wiki/Ate_my_balls https://en.wikipedia.org/wiki/Ate_my_balls > "Ate my balls" is one of the earliest examples of an internet meme. It was widely shared in the late 1990s when adherents created web pages to depict a particular celebrity, fictional character, or other subject's zeal for eating testicles. Often, the site would consist of a humorous fictitious story or comic featuring edited photos about the titular individual; the photo editing was often crude and featured the character next to comic-book style speech in a thought balloon. > The fad was started in 1996 by Nehal Patel, a student at University of Illinois at Urbana-Champaign with a "Mr. T Ate My Balls" web page.
meepmorp 3y ago
It’s the first thing that popped into my head.

mjb 3y ago

There's some really interesting little details here. Linux, by default, is making the very reasonable assumption that the marginal cost of converting empty physical memory into caches and buffers is very near zero. This is fundamentally reasonable, because the cost of converting empty memory into used memory isn't really any cheaper than converting a clean cached page into used memory. It's a little more subtle when you take accounting into account, or when you think about dirty pages (which need to be written back to clear memory), or think about caches, but the core assumption is a very reasonable one. Except for on some multi-tenant infrastructure. Here, "empty" pages don't really exist. There's mostly not an empty page of memory kicking around waiting (like there is on client devices). Instead, nearly all the memory on the box is allocated, but each individual guest kernel doesn't know the full allocation. In this world, the assumption that the marginal cost of converting empty to full is zero is no longer true. There's some real cost. Projects like DAMON https://sjp38.github.io/post/damon/ https://sjp38.github.io/post/damon/ exist to handle this case, and similar cases where keeping empty memory rather than low-value cache is worse for the overall system. These kinds of systems aren't super common, especially on the client side, but aren't unusual in large-scale cloud services.

mjb 3y ago
The other interesting detail here is the memory sizing problem. If I can consume all my RAM with caches and buffers, how much RAM do I need? The answer (as always) depends on what you're optimizing for. For performance, bigger is better. For cost, energy, etc you're going to want some way to calculate whether adding more RAM (and so having bigger caches) is worth the cost, heat, power, etc. Gray and Putzolu's classic "The 5 minute rule for trading memory for disc accesses" (https://dl.acm.org/doi/pdf/10.1145/38713.38755 https://dl.acm.org/doi/pdf/10.1145/38713.38755) from 1987 is probably one of the most important CS systems papers of all time. In it, they lay out a way of thinking about memory and cache sizing by comparing the cost of holding cache to the cost of access (this isn't the first use of that line of thinking, but is a very influential statement of it). Back then, they found that storing 4kB in RAM for 5 minutes costs about the same as reading it back from storage. So if you're going to access something again within 5 minutes you should keep it around. The constants have change a lot (RAM is way cheaper, IOs are way cheaper, block sizes are typically bigger) since then, but the logic and way of thinking are largely timeless. The 5 minute rule is a quantitative way of thinking about the size of the working set, an idea that dates back at least to 1968 and Denning's "The working set model for program behavior" (https://dl.acm.org/doi/10.1145/363095.363141 https://dl.acm.org/doi/10.1145/363095.363141). Back to marginal costs - the marginal cost of converting empty RAM to cache is zero in the minute, but only because the full cost has been borne up front when the machine is purchased. It's not zero, just pre-paid.

bminor13 3y ago

Does anyone happen to have expertise/pointers on how ZFS' ARC interacts with Linux disk caching currently when using ZFS-on-Linux? It seems like the ARC space shows up as "used" despite being in a similar category of "made available if needed" - is that correct? Is data in the ARC double-cached by Linux's disk caching mentioned in the post? If so, is it possible to disable this double-caching somehow?

MrDrMcCoy 3y ago
ZFS ARC unfortunately does not integrate with the kernel file cache, so they step on each other a lot. ZFS does watch available system RAM and try to dynamically reduce its usage as memory pressure increases, but I've found its responsiveness for this to be far too slow. This combined with how ARC appears to just be an opaque block of RAM that cannot be reclaimed, I usually just set a hard limit on how big the ARC is allowed to get in the module load arguments and be done with it (at least for systems that are doing more than just storage).
- drewg123 3y ago
  Is ARC really non-reclaimable on Linux? At least on FreeBSD, there is a kmem_cache_reap() that is called from the core kernel VM system's low memory handlers. Looking at the linux code in openzfs, it looks like there is an "spl_kmem_cache_reap_now()" function. Maybe the problem is the kernel dev's anti-ZFS stance, and it can't be hooked into the right place (eg, the kernel's VM low memory handling code)?
  - MrDrMcCoy 3y ago
    It's reclaimable, but opaque. The ARC just looks like used RAM rather than file cache, which throws off various means of accounting.
  - nubinetwork 3y ago
    echo 3 > /proc/sys/vm/drop_caches (Bear in mind that 3 is the most aggressive but other than exporting the pool, it's the only way to dump the cache, especially if you boot off ZFS)

therealmarv 3y ago

Windows and Mac use compressed RAM for many many years as standard. Yet on many Linux desktop you have to activate it (namely ZRAM). It solves the problem that a e.g. browser eats all your memory. It's much quicker than Swap and yet mostly unknown by many people who are running a Linux desktop. As mentioned by another user it's still not standard on Ubuntu desktop and I don't understand why.

spookie 3y ago
There are distributions that enable it by default, Fedora comes to mind.
viraptor 3y ago
> It solves the problem that a e.g. browser eats all your memory. It doesn't solve that. You get a little bit more headroom, but that's it. Not much ram is considered compressible anyway. On my Mac I'm barely reaching 10% of compressed memory anyway, so it doesn't make that much difference.
- noisem4ker 3y ago
  Current RAM usage from my Windows 10 dev machine, as reported by Task Manager: > In use: 18028 MB > In use, compressed: 2718 MB > Compressed memory stores an estimated 9013 MB of data, saving the system 6294 MB of memory. That's not a small amount.
- jowea 3y ago
  My experience is that zram saves quite a bit. I have another old laptop with 4GB where it's essential. Maybe it differs by program type? NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT /dev/zram0 lzo-rle 15,6G 1,9G 248,6M 418M 16 [SWAP]
  - viraptor 3y ago
    Yup, it will depend on your workload a lot. Worth testing of course!

kevin_nisbet 3y ago

When I worked in telco we used to run into this a lot. We'd demand standard alarms for things like memory leaks / out of memory conditions / high than normal memory usage, as to get 99.999% uptime we want to be paged when problems like this would occur. Except a bunch of platform did the extremely naive implementation and included recoverable memory in their alarm conditions. So inevitably someone would log in and grep the logs or copy some files to the system, and hit the alarm conditions. And there were some vendors who really didn't want to fix it, they would argue that recoverable memory is in use, so it should really be part of that alarm condition.

zaptrem 3y ago

> Disk caching makes the system much faster and more responsive! There are no downsides, except for confusing newbies. It does not take memory away from applications in any way, ever! No downsides except for massive data loss when the system suddenly loses power/a drive crashes and the massive theft of memory from host OSes (e.g., when using Windows Subsystem for Linux).

nightfly 3y ago
There is no data loss from disk caching.
- zaptrem 3y ago
  Does disk caching not include write caching?
  - nightfly 3y ago
    Not in the context we're talking about
    - hifromwork 3y ago
      Are you sure? Dirty pages have to reside somewhere, so they are actually stuck in RAM until successfully written RAM. Linux will lie with straight face that dd to my 8gb pendrive finished successfully in a few seconds, so there may be non-trivial amounts of RAM involved here. I don't know enough of Linux internals to know if the writeback cache and read cache are the same object in the kernel, but they feel similar. Of course the real response is that without write cache (effectively adding fsync to every write) any modern linux system will grind to absolute halt and doing anything would be a challenge. So contray to GP's post, it's not reasonable to complain about it's existence.
      - withinboredom 3y ago
        It's written asap to disk unless you configure it otherwise. The disk (or flash disk, in your case) has its own ram before actually flushing to physical storage (fsync, unless it lies, in which case there is probably an article about it here cough macs cough). So the kernel isn't lying to you, but your disk probably is.

__turbobrew__ 3y ago

> Disk cache can always be given back to applications immediately This is not true, there is a cost to freeing the cache pages and allocating them to the other program. I have seen some very regressive performance patterns around pages getting thrashed back and forth between programs and the page cache, especially in containers which are memory limited. You throw memory maps into the mix and things can get really bad really fast.

blueflow 3y ago
Can you elaborate further?
- __turbobrew__ 3y ago
  There is a fast path and slow path to reclaiming cached pages. There are ways to measure the impact using ebpf probes. If you are interested in going deeper I recommend looking at the memory section of this book: https://www.brendangregg.com/blog/2020-07-15/systems-performance-2nd-edition.html https://www.brendangregg.com/blog/2020-07-15/systems-perform...
mnw21cam 3y ago
That's why Linux does keep a certain amount of RAM actually-free, so that it can hand over some pages immediately. If the amount of actually-free RAM goes below a certain amount, then it will pre-emptively evict a load of cache pages.

lxe 3y ago

This made me chase red herrings when debugging oom issues in production. Wish free would just remove the 'free' column and replaced it with 'available'.

tetha 3y ago
This is what we did in pretty much all of our monitoring some time ago. We ripped out most memory graphs except for "Total Memory" and "Available Memory" as well as memory pressure from the PSI metrics. And we placed alerts on available memory growing low, as well as pages being swapped in. Newer kernel opportunistically swap-out idle pages, but that's fine as long as you never see the path from disk to memory (swap-in). This has increased the quality of our memory monitoring by a lot.

drewg123 3y ago

I used to run a Linux workstation in the late 00's (sorry FreeBSD folks, I know, the shame...), and I ran a closed source PVR application on it. The memory access pattern was pretty much pessimal for my use of the box as a workstation. I'd use it from 7am -> 8/9pm every day, then when I'd walk away from the keyboard, I'd watch HD recordings (which could be 7GB or more per hour). Those would get cached in memory, and eventually my workstation stuff (emacs, xterms, firefox, thunderbird) would start to get paged out. In the mornings, it was painful to start using each application, as it waited forever to page in from a spinning disk. I eventually wrote an LD_PRELOAD for the DVR software that overloaded open, and added O_DIRECT (to tell the kernel not to cache the data). This totally solved my problem, and didn't impact my DVR usage at all.

toast0 3y ago
> I used to run a Linux workstation in the late 00's (sorry FreeBSD folks, I know, the shame...), and I ran a closed source PVR application on it. It's ok, no shame. But as I understand it, FreeBSD would prefer to throw out (clean) disk cache pages under memory pressure until somewhere around FreeBSD 11 +/- 1, where there were a few changes that combined to make things like you described likely to happen. Heavy I/O overnight might still have been enough, and I'm not going to test run an old OS version to check ;) I can't find the changes quickly, but IIRC, older FreeBSD didn't mark anonymous pages as inactive unless there was heavy memory pressure; when there was mild memory pressure, it would go through the page queue(s) and free clean disk pages and skip other page; only taking action on a second pass if the first pass didn't clean enough. This usually meant your program pages would stay in memory, but when you hit memory pressure, there would be a big pause to mark a lot of pages inactive, often too many pages, which would then get faulted back to active... Current FreeBSD marks pages inactive on a more consistent basis, which is nice because when there is memory pressure, chancws are there's already classified pages. But it can lead to anonymous pages getting swapped out in favor of disk pages as you described; it's all tunable, of course, but it was a kind of weird transition for me. After upgrading the OS, some of my heavy i/o machines saw rising swap usage running the same software as before; took a while to figure that out.
- 3y ago

loktarogar 3y ago

This feels like a UX problem. If this is a normal and expected part of linux operation, it should be called out in the {T,G}UI.

the8472 3y ago
But it does. https://files.catbox.moe/l9je82.png https://files.catbox.moe/l9je82.png orange is the part used by caches
- loktarogar 3y ago
  I'm not a linux user, so not an observation of experience. Just the existence of this website suggests to me that however it is being done right now could be made clearer somehow.
nightfly 3y ago
htop shows this
gruez 3y ago
It is. Windows does the same thing, but it's a non-issue because task manager makes it look like cached memory is free memory.
bityard 3y ago
1. You can't change `free` output, you'll break SO many scripts. 2. Most things which report memory usage in a user-friendly way _already_ do this in an obvious way. (Htop shows disk cache in a bar graph, but doesn't add it to the "used" counter.) 3. Should UX always compensate for some fraction of users' misunderstanding of how their OS kernel works? Or would it be better for them to ask the question and then be educated by the answer?
- loktarogar 3y ago
  > Or would it be better for them to ask the question and then be educated by the answer? Good UX makes the question "why is linux using my unused RAM for disk caching" (a non pressing question) instead of "why is linux eating up all my RAM" (panic, stressful question)

neurostimulant 3y ago

> If your applications want more memory, they just take back a chunk that the disk cache borrowed. Disk cache can always be given back to applications immediately! You are not low on ram! I'm running RKE2 on my desktop and it'll start killing pods due to low memory pressure, even though the memory was only used for disk caching. I wonder if there is any way to make it stop doing that and instead only start killing pods if it's due to "real" low memory pressure.

blueflow 3y ago
Think about it: A processes executable code comes from a file. You will need the size of the executable available as disk cache or the program execution will cause heavy thrashing and I/O. So some part of it is "real" memory pressure.
- neurostimulant 3y ago
  I also run an nfs server in the same machine, so after a period of heavy nfs use, most of the ram were eaten by the disk cache and rke2/kubernetes start having memory pressure taint. After a fresh restart with all pods running, the memory usage is below 10%, so I doubt the disk cache was full with executable files cache.
okwhateverdude 3y ago
Assuming Linux, oddly enough I came across this exact behavior[0] while researching resource management for an on-prem k8s cluster. Take a look at that thread for more info, but TL;DR, you need to actually finesse page cache constraints if you want avoid the behavior. You actually can have really fine grained control over page cache via cgroups v2[1][2] and systemd[3]. [0]: https://github.com/kubernetes/kubernetes/issues/43916 https://github.com/kubernetes/kubernetes/issues/43916 [1]: https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-interface-files https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-in... [2]: https://biriukov.dev/docs/page-cache/6-cgroup-v2-and-page-cache/ https://biriukov.dev/docs/page-cache/6-cgroup-v2-and-page-ca... [3]: https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html https://www.freedesktop.org/software/systemd/man/systemd.res...

jenadine 3y ago

How old is this website? It's from a time when a typical computer only had 1.5 G of ram.

I_Am_Nous 3y ago
The domain appears to have been registered 25 Apr 2009, and I remember seeing this quite a while ago. That would make sense for 1.5 G of RAM being typical. Glad it's still around :)
Pathogen-David 3y ago
Oldest copy on the Wayback Machine is from May 2009 https://web.archive.org/web/20090513043445/https://www.linuxatemyram.com/ https://web.archive.org/web/20090513043445/https://www.linux...

burnte 3y ago

Unused RAM is wasted RAM. Why people want to see GOBS of empty RAM boggles my mind.

filchermcurr 3y ago
I think the disconnect is not understanding how the RAM is used. If the average user looks and sees all of their RAM in use, they're going to think that there's no more room for the applications that they want to launch. They don't understand that what's cached will just get out of the way when the memory is actually needed. So they want to see free RAM, because that means it's free for their game or millions of tabs.

wingworks 3y ago

htop shows this (it's the orange/yellow bar in RAM)

withinboredom 3y ago
I recommend btop these days (https://github.com/aristocratos/btop https://github.com/aristocratos/btop)
- wingworks 3y ago
  I just tried btop, it has crazy colours (nothing like screenshots) on my Mac Terminal app (Misterioso theme).

avgcorrection 3y ago

Trust me. If Linux really eats your RAM to the point of reaching an OOM state you will know. (This was of course because of having too many apps relative to my RAM. Not because of disk caching.) The OOM behavior is not pleasant for a desktop system.

berkes 3y ago
I've never encountered this with "many apps" starting to OOM, but many times with one process OOMing. That one will simply crash and everything else continues to run unharmed.
- mbakke 3y ago
  What distribution are you using? IME, if a process grows out of control, Linux won't notice until the whole system is thrashing, at which point it's too late and it tries killing random things like browser tabs way before the offending process. In rare cases Linux might recover, but only because I hammered C-c the right place 30 minutes ago. In most cases a hard reboot is required (I left it overnight once, fans spinning at max, hoping the kernel would eventually accept my plea for help, but she had other priorities).
  - jltsiren 3y ago
    I guess OOM is more problematic on low-memory systems or when you have more than a nominal amount of swap. If you have enough memory that the desktop environment, browsers, and other processes that should keep running only use a small fraction of it, the OOM killer can pick a reasonable target reliably. A process that tries to allocate too much memory gets killed, and everything is robust and deterministic. I sometimes trigger OOM several times in an hour, for example when trying to find reasonable computational parameters for something.
    - Sakos 3y ago
      How much memory do you think is reasonable? I've had it happen to me with 16GB and even 32GB, where I never ever have this issue on Windows (unless for some reason I'm on a 2GB RAM system for God knows why). I wish people would stop defending pathological behavior that's broken for standard desktop use. What's wrong with wanting things to improve?

Scarbutt 3y ago

It's weird that their 'used' column is accounting for 'buff/cache' 'man free' states used Used or unavailable memory (calculated as total - available)

mkhnews 3y ago

Another question is about containers and memory limits. Does the page-cache count against my container memory limit ? And if so, then when I hit that limit from doing many reads, does the page-cache start taking from itself without OOM killer getting involved ?

deleted 3y ago
[deleted]
defer 3y ago
I also want to know this, but in reverse. I build older android (the OS) versions inside docker containers because they have dependencies on older glibc versions. This is a memory-heavy multi-threaded process and the OOM killer will kill build threads, making my build fail. However, there is plenty of available (but not free) memory in the docker host, but apparently not available in the container. If I drop caches on the host periodically, the build generally succeeds.
- mkhnews 3y ago
  And perhaps k8s is a specific category to consider here. I've read and thought I've experienced where 'active' (as opposed to in-active) page-cache does count towards k8s mem limit.
otterley 3y ago
1. Pages cached by applications are charged to its container for the purpose of memory resource limits. 2. IME the kernel takes the container's memory limit into account when determining whether to allocate a page for cache. Caching, by itself, won't cause the container to exceed a memory limit.

mavhc 3y ago

My Linux ram problems are 1: Ubuntu default install, ends up with xwayland using 5GB ram. 2: when running out of ram it seems to default to crashing back to the logon screen

Karunamon 3y ago

Okay, if unused memory is wasted and there are truly no consequences for the "free" column reading zero, then why on a busy system do I get UI chugging and otherwise poor (bordering on unusable) performance under this condition that is immediately resolved by forcing the caches to drop and freeing multiple gigabytes? Whatever conceivable speedup there is from 12 GB of file cache as opposed to 11 is obliterated multiple times over from the time lost by having to do this dance, or worse, recovering after the oom killer wipes out my X session or browser.

AnotherGoodName 3y ago
>recovering after the oom killer wipes out my X session or browser. Perhaps you can share more details of what you're doing to force the cache to drop and what the side effects are exactly because an OOM can't be caused by the file cache since the total free memory available to applications remains the same. The whole point of the file cache is to use otherwise unallocated memory and give it up the moment it's needed. There should not be an OOM from this short of an OS bug or an over allocated virtualized system.
- Karunamon 3y ago
  echo 3 > /proc/sys/vm/drop_caches Last time I ran into this was a couple of years ago on a stock Arch system. (Disabilities forced me back to Windows). Every time, the largest memory consumer was the web browser. Also every time, the system became nearly unresponsive due to swap thrashing (kswapd at the top of the CPU usage list, most of which was I/O wait). Last time I complained about this problem, someone suggested installing zram which did stop it from happening. However, this does not change the fact that there is some pathological failure case that contradicts the central thesis (not to mention, smug tone) of this website and makes searching for solutions to the problem infuriating.
PhilipRoman 3y ago
I find that task priorities in general are not strict enough under Linux. Try running a cpu heavy but very low priority task in background and it still manages to measurably affect the latency of the more important process. And this is just the CPU, not to mention disk utilization and network usage. I was too lazy to find a proper solution, so I just used mlockall after allocating a massive heap and pin the process to a core that is reserved only for this specific purpose. I think cgroups has very flexible tools for reserving system wide resources, but haven't had the time to test it yet.

HippoBaro 3y ago

I think the information there is valuable because questions about memory usage in Linux keep coming up. The answer: "don't worry about it," is probably a good starting point. The page claims things that are just really misleading, though. > There are no downsides, except for confusing newbies. False. Populating the page cache involves lots of memory copies. It pays off if what's written is read back many times; otherwise, it's a net loss. It also costs cycles and memory to keep track of all these pages and maintain usage statistics so we know what page should be kept and which can be discarded. Unfortunately, Linux makes quantifying that cost hard, so it is not well understood. > You can't disable disk caching. The only reason anyone ever wants to disable disk caching is because they think it takes memory away from their applications, which it doesn't! People do want that, and they do turn it off. It's probably the number one thing database people do because they want domain-specific caching in userland and use O_DIRECT to bypass the kernel caches altogether. If you don't, you end up caching things twice, which is efficient/redundant.

GuB-42 3y ago

I remember that being the case for early versions of Android, people were surprised all their RAM was used, and of course, we could find apps that "freed" the RAM, generally making things worse. And the response was similar: all that "used" RAM can be reclaimed at any time should an app need some, but in the meantime, the system (which is Linux) might as well use it. I think they "fixed" it in later versions. I don't know how, but I suspect they just changed the UI to stop people from complaining and downloading counterproductive apps. As usual in these situations, unless you really know what you are doing, let the system do its job, some of the best engineers with good knowledge of the internals have worked on it, you won't do better by looking at a single number and downloading random apps. For RAM in particular, because of the way virtual memory works, it is hard to get an idea of what is happening. There are caches, shared memory, mapped files, in-app allocators, etc...

gslin 3y ago

https://archive.ph/WmJHf https://archive.ph/WmJHf

neonate 3y ago
https://web.archive.org/web/20230928045611/https://www.linuxatemyram.com/ https://web.archive.org/web/20230928045611/https://www.linux...

404mm 3y ago

You can always just download more RAM. https://www.downloadmoreram.com https://www.downloadmoreram.com

simonblack 3y ago

Why all the fuss?? When you can download as much RAM as you want, any time you want. https://downloadmoreram.com/ https://downloadmoreram.com/

lnyng 3y ago

We published this paper "TMO: Transparent Memory Offloading in Datacenters" last year which covers some Linux memory management mechanisms that may be quite useful for providing reasonable estimations to application memory usage. We observed that the real memory footprint for applications depends on many factors: file access pattern, disk IO speed (especially if swap is enabled), ssd vs hdd, application latency sensitivity, etc. Instead of coming up with some overly complicated heuristic, we use the Linux kernel provided memory.pressure [0] metric via cgroup v2. It measures the amount of time spent waiting for memory (page fault etc). Then by slowly reclaiming memory from the application until its memory pressure hits some target (say 0.1%), we can claim that the steady state usage is the actual memory footprint. This may not be useful for PC but could be very useful for data center to track memory regression, and also to harvest disk swap without concerning too much about the cliff effect when the host runs out of memory and suddenly kernel pushes everything to swap space. [0] https://facebookmicrosites.github.io/cgroup2/docs/pressure-metrics.html https://facebookmicrosites.github.io/cgroup2/docs/pressure-m...

fuzztester 3y ago

Windows ate my hard disk (every year).

deleted 3y ago

[deleted]

thiag0 3y ago

Is the website down? I was reading and suddenly started to get timeout.