12 ms·

Linux ate my RAM

194 points by wallzz 13y ago

jaseemabid 13y ago
Lovely! This domain could iframe that SO question :)
- daGrevis 13y ago
  What question?
  - Schiphol 13y ago
    Why, that one, of course.
    - lucasnemeth 13y ago
      I've thought the answer to that one was 42.
      - Samuel_Michon 13y ago
        ITT: reddit-style comments that don’t get downvoted. I’m all for it. http://i.imgur.com/lrlFq0y.jpg http://i.imgur.com/lrlFq0y.jpg
daGrevis 13y ago
The page seems to be down. Here's how it looked: http://web.archive.org/web/20131011053519/http://www.linuxatemyram.com/ http://web.archive.org/web/20131011053519/http://www.linuxat...
makomk 13y ago
This is actually wrong. For some reason (probably because of the way it's implemented internally) Linux includes shared memory allocations in the "cached" figure. If you calculate the amount of free memory using "free -m" in the way this web page advises, you'll think that memory is free when it really isn't - unlike true cache, shared memory cannot be discarded when the space is needed for something else. I've actually had buggy desktop applications use several gigabytes of RAM as shared memory, so this can make a big difference.
- appleflaxen 13y ago
  > For some reason (probably because of the way it's implemented internally) (this is tautologically true)
  - cma 13y ago
    he means the rationale behind developing it that way was due to the way other stuff is laid out internally, not "the number shows up because the existing code codes for it and causes it to show up".
- gue5t 13y ago
  Linux also counts files on tmpfs as part of the cache number--it's really annoying that no number in top can serve to show you a decent approximation of how many page-size malloc()s you can do before OOM (with overhead and fragmentation and so on, of course it's infeasible to get an exact number).
  - marcosdumay 13y ago
    It's worse. No line of /proc/meminfo has that value, and since not all of the cache can be instantly discarded (I didn't know that), I don't see any combination of them that would help. That sentence: > There are no downsides, except for confusing newbies. Just isn't true. There is the downside that we can't know how much memory is available.
    - makomk 13y ago
      Yeah, and obviously some (though not all) forms of shared memory are actually implemented using tmpfs behind the scenes which makes life even more interesting. I think the problem I encountered involved SysV shared memory which isn't backed by tmpfs and does show up in /proc/meminfo, whereas other types may not.
    - mitchty 13y ago
      Can't you use something like this: https://raw.github.com/pixelb/ps_mem/master/ps_mem.py https://raw.github.com/pixelb/ps_mem/master/ps_mem.py I use that to help determine "least incorrect" ram usage on linux. It is ok, maybe not perfect but it at least lets you get a better view on memory.
jbogp 13y ago
Also HN ate your server apparently.
Morgawr 13y ago
Best way to check how much ram is actually free is to use a higher level tool like a task manager (from various DEs) or a command line tool like htop, they report the actual memory usage of processes
v-yadli 13y ago
https://www.kernel.org/doc/Documentation/sysctl/vm.txt https://www.kernel.org/doc/Documentation/sysctl/vm.txt You can drop cache by sending a byte into drop_cache.
- mprovost 13y ago
  But why would you do that? (The only time I've ever used that is when benchmarking and you need a clean cache between runs). Otherwise the kernel will just empty it as needed when you run a program.
  - jrkatz 13y ago
    I can't answer for everyone, but I can tell you why I have a cron job dropping my cache every several minutes: The application I develop consumes most of the ram on my machine when running, and takes several minutes to rebuild and start up to begin with. When most of my memory is being used for the cache, this process takes several minutes longer, because I'm making millions upon millions of calls for more memory -- and each one has to get some of that cached memory back for itself. If I simply drop the cache all at once, every minute, it takes a split second. If I shrink it over a million increments, it takes around a minute. Even typing that I feel I must be doing something wrong; and yet, it worked.
    - deathanatos 13y ago
      Your app is making more calls to malloc when there is memory being used by disk cache, or you mean something else by "millions upon millions of calls for more memory"? You control hope many times you call malloc. Just call it once with what you need.
  - v-yadli 13y ago
    First, it's not why. It's that this is possible. Please don't say "this is impossible because this doesn't make sense". Second, I agree with you. Normally people don't need to do this unless they hate their hard drives. :-) But there's some special cases, for example when the disk is "volatile", or when you want a huge chunk of continuous physical address and you want to tell the OS to stay away and don't touch any pages inside. PS: An interesting thing is that Windows people always seek for a way to enlarge disk cache. It's fine on server versions, but on desktop versions such as Windows 7, if I don't remember wrong, disk cache is capped at ~4GB(not sure if globally or per file). Oh well.
Damogran6 13y ago
FWIW, it's relevant to OS X too (even Mavericks)...'sudo purge' will reclaim some of the resources.
- sounds 13y ago
  OS X does not behave exactly in this manner, but is indeed confusing. Linux never needs a 'purge' or equivalent. Doing a 'drop_cache' only slows the system slightly because nothing is cached. 'sudo purge' can prevent OS X from using swap – i.e. running out of RAM even though there is RAM it could reclaim. This behavior has been tweaked but not fixed in Mavericks.
  - Zr40 13y ago
    Actually, purge is completely unrelated to swap. Purge forces disk cache to be flushed to disk, and can be used to approximate a cold disk buffer cache scenario for performance analysis. Some small amount of swap usage (compared to total RAM) is perfectly normal, by the way. More cache, for instance, is a better use of RAM than avoiding swap usage by keeping never-accessed pages present.
free652 13y ago
Windows loves to cache and "prefetch" apps, I still need to figure out how to disable prefetching since I have a very fast SSD.
- csmuk 13y ago
  Why? All it does is make it go faster.
  - TBastiani 13y ago
    And drain your battery for no reason.
    - hansbogert 13y ago
      Do you have any proof of that? "Free" memory uses as much energy as "used" memory, there is no distinction in SD|DDR-RAM. And if the OS is smart, it only has to keep a simple index of memory pages still relevant. I'd argue that this has a measurable battery-hit.
      - jeltz 13y ago
        Reading from the SSD to laod it into RAM does drain battery. No idea if it is enough to matter though.
      - runako 13y ago
        You're only accounting for the RAM power. Accessing the SSD costs power, too.
        tedunangst 13y ago
        Then I think I'd rather Windows prefetched my programs while I'm plugged in so it doesn't have to do so when I'm on battery.
        tanzam75 13y ago
        > You're only accounting for the RAM power. Accessing the SSD costs power, too. A typical modern SSD reads at 500 MiB/s, consumes 3 watts when reading, and 0.3 watts when idle. A stick of DDR3 RAM consumes about 3 watts, just sitting idle. The power consumption from prefetching is minor compared to that of the RAM.
jheriko 13y ago
right. so there is a bug in linux. gotcha.
- annnnd 13y ago
  No. Well, not technically... There is a UX bug in Linux. Not the only one at that, but they don't count much in Linux-land.
  - hansbogert 13y ago
    Just a difference in semantics -- I mean, c'mon UX in a utility like top? which is probably specified in some Unix convention anyway. Pure BSD Unix do this as well in their own way.
- CmonDev 13y ago
  Free is used, black is white, truth is a lie, lie is a truth.
- gmuslera 13y ago
  Leaving your memory with no use at all would be the bug. It is a free resource that could be used to improve system performance at basically no cost (the disk is already read, the memory is still available immediately for any app that request it, etc), so it is a good default to let linux do it. But could be cases where it could be side effects of it (i.e. running in a VM could make less memory available for the host, and it could be caching disk already) so you could consider to disable it sometimes.
- jheriko 13y ago
  it confuses users. its a bug. thank you for the -4 points... :)
caw 13y ago
Wow, I can't believe how many times I've had to explain this to people as they log on to shared VNC servers. "I thought this had 16G of RAM free, I asked for a 16G machine!". "It does, stop trying to read top!" Now I'll just point them to this website.
pearjuice 13y ago
Then you should http://downloadmoreram.com http://downloadmoreram.com
- blowski 13y ago
  Haha, that's brilliant. From their Twitter account: > Have an iPhone 3? Download RAM twice to make it a iPhone 5. https://twitter.com/DownloadMoreRam/status/385026260008587264 https://twitter.com/DownloadMoreRam/status/38502626000858726...
- 10098 13y ago
  Crap, I wanted to post that!
- puppypilgrim 13y ago
  Brilliant. I love it.
- unimpressive 13y ago
  When my circle of friends found that site it became an in-joke. I'd never heard of it, and eventually they came up to me and asked "Can you download more RAM?". I don't think they were expecting my answer: "Sure you can, just use plan 9."
nraynaud 13y ago
that's funny, like avery complex or "smart" algorithm, if it works it's cool, but when something doesn't work, it makes it very hard to pinpoint a culprit. I'm more and more inclined to either use stupider algorithm that at least I can explain to the user, incremental stuff where they can check intermediate results or taking into account the fact that this algorithm has to be conveyed to users in one way or another since inception. I guess in this case a measure of the swap activity would be a good indicator that the current limiting factor is the RAM, but it also has to be credible. Somehow having 16G written on the box of the ram and 7G written in the "available ram" of a software makes everything more credible that saying "I have a pressure of 1.5madeupunit on the ram front". Human factors, again.
- lamontcg 13y ago
  I've been doing performance analysis and debugging as part of being a unix/linux sysadmin now for something like 20 years. And even though I have many well-cherished anecdotes about explaining the VM free list to software architects who were the go-to guys for the vision of our entire global platform, complete with PhDs, I still don't want to go back to the bad old days without a shared buffer cache. There's a reason why that's there, and you don't want to go back to 1992-era memory management. It also makes perfect sense from the perspective of writing the software. The free list really is the list of VM pages that have been free'd and that has a specific meaning, and changing that name internally would be terrible. You just need to understand how the machine actually works.
dancecodes 13y ago
Cant load page while. But Linux RAM problems can be in desktop when something wrong with video systems (X, drivers, compositing, etc...) Check first: xorg, video drivers, mesa. Sometime java apps makes big swaps. Check your linux distro also. Try learning Gentoo.
andmarios 13y ago
This is the worst piece of Linux advice ever. I use Linux for many years and every time I had a memory leak or a process using too much memory, every google search about the problem would lead to some "guru" giving this advice. This explanation which has been replicated thousands of times on every linux forum on the planet, prevents people from solving real problems. It started as a useful piece of information but nowadays it is spam.
- jlgreco 13y ago
  How does this understanding prevent people from solving real problems, simply by virtue of it being commonly cited? I don't understand your complaint.
  - marcosdumay 13y ago
    I understand this is a complaint about Google. It won't help you find pages that cite a problem when a diffrent, but similar problem is too popular. I've seen that happening several times, but I'm not the GP, so I may still be missunderstanding him.
    - andmarios 13y ago
      Yes, this is exactly what I mean. Maybe I should've describe it as noise rather than spam. But it isn't only a Google search issue. Every time you will ask a question about a memory problem in Linux, everybody will give you this answer and classify you as newbie. When you explain them that this isn't the case, they will simply leave the thread/question and nobody would ever bother reading your problem again. People just go in denial mode because of this common knowledge, as Linux can't have memory leaks or any memory related problem.
      - jlgreco 13y ago
        Just give more information right off the bat. Don't say "XYZ is chewing up lots of RAM"; the best case scenario if you say that is they have to respond back asking for more information. Give that information to them in the first place "XYZ starts at XXXXMB shared and XXMB resident. Over five minutes it steadily climbs to XXXXMB resident. Shared remains constant at XXXMB." Doing this, I can't say I've ever been brushed off using this page.
deleted 13y ago
[deleted]
hippich 13y ago
That's how I check true free memory available: root@joe-pc:~# free -m total used free shared buffers cached Mem: 32057 30347 1710 0 1312 21568 -/+ buffers/cache: 7466 24591 Swap: 0 0 0 root@joe-pc:~# sync; echo 3 > /proc/sys/vm/drop_caches root@joe-pc:~# free -m total used free shared buffers cached Mem: 32057 5628 26429 0 4 891 -/+ buffers/cache: 4732 27325 Swap: 0 0 0 root@joe-pc:~#
- dice 13y ago
  Yikes! Hopefully not on production systems. I've had fileservers which needed 60+GB of dentry cache to serve up files in a reasonable time, something like that could bring the site to a crawl for 10 minutes or more.
  - hippich 13y ago
    Good point. I am doing it on my laptop and server without load. So this is not a big problem in my cases.
bediger4000 13y ago
How bizarre. In the 1993-94 time frame, one of the big talking points about Windows NT was that it was the only OS with an integrated file system cache. That wasn't true even at that time, as Solaris also had such a cache, but such is the nature of propaganda. I'm tickled that what once was a big "technical" advantage becomes a major source of confusion. Presumably, OSes of Windows NT heritage have retained file system caching, which offers a huge performance benefit. Why don't Windows users have confusion over this issue? Doesn't Windows offer an easy way to check or is the whole thing presented clearly, or is Windows so complicated not many people get this far? I'll also draw a parallel to DUI - without a breath-o-meter, DUI is a lot like driving while distracted. It's impossible to quantify, and therefore not that much of a crime. With a breath-o-meter, or with "top", it's possible to put a number on drunkeness or lack of free memory, and the whole thing becomes a problem.
- chiph 13y ago
  From what I can tell, Windows does not show the disk cache allocation in Task Manager. So even though all the RAM is likely in use, that is hidden from the user. You can probably see it in PerfMon, as that is less "typical user" oriented.
  - rplnt 13y ago
    It's not in the graph but it's there in the "Cached" field.
  - tracker1 13y ago
    In windows 7, take "available" and subtract "free", that's your cache usage. Windows Vista started using RAM very aggressively for caching, including pre-filling cache with frequently used pages (apps/data) from disk... ReadyBoost was basically an extension of this...
- Lagged2Death 13y ago
  Why don't Windows users have confusion over this issue? Windows' Task Manager has generally counted memory occupied by the disk cache as "available."
nly 13y ago
Forget the kernel, someone really needs to do chromeatemyram.com
FlyingAvatar 13y ago
I hate the invention of "burstable" RAM on VPSes for this reason. Eventually your IO cache grows until it has most of your guaranteed RAM. Now every new process is playing a game of Russian roulette with your hypervisor to see if it can actually allocate into your "burstable" RAM what the kernel is reporting is freely available.
- vajrabum 13y ago
  Actually this is a general issue when running multiple Linux VMs under a hypervisor. In that case you don't want them filling up RAM with prefetch data from disk because that's likely to cause contention and reduce page sharing between VMs.
rdtsc 13y ago
A quick practical answer is to use htop and look at the memory bar. * Green = resident memory used (probably what you want most of the time) * Blue = buffers * Yellow = cache
- ChuckMcM 13y ago
  +1 for htop, it not only shows excellent memory stats in an easy to absorb way but it also shows CPU cores usage. Definitely one of my "wtf is going on?" tools.
- hisham_hm 13y ago
  htop author here. It was nice seeing the table in the article with the memory split in three colors and thinking "well, but that's how I already split it in htop, isn't it?". The "Linux calls it" column should probably be called "top calls it"...
  - rdtsc 13y ago
    Thank you for writing htop, it is very useful tool. I just about fell off my chair when I realized I could click with my mouse on the columns to sort them!
    - gregf 13y ago
      I always forget the key bindings for this, you just changed my life!
      - rdtsc 13y ago
        It was accidental too. I was just showing my coworker something and I have a bad habit of highlighting stuff I am reading and randomly clicking on text, and I noticed that column was selected to be sorted. It blew my mind, this was in an SSH session logged into a remote server.
- sciurus 13y ago
  I'll give a pitch for atop as well. Here's a screenshot of it showing systemwide and per-process memory information: http://www.atoptool.nl/images/screenshots/memory.png http://www.atoptool.nl/images/screenshots/memory.png Here's a case study of using it to observe a memory leak: http://www.atoptool.nl/download/case_leakage.pdf http://www.atoptool.nl/download/case_leakage.pdf
usamec 13y ago
Ehm. But sometimes stupid linux kernel prefers moving something usefull to swap and caching some shit instead of keeping stuff in ram and not doing caching. So linux sometimes eats your RAM (but in different way).
- Amadou 13y ago
  For situations where you have a time-critical app, you can pin the app's memory into RAM with the mlock() family of functions. There may be a command-line utility that lets you pin an arbitrary running a process too, I haven't looked for it on linux.
jared314 13y ago
Previous Discussions: https://news.ycombinator.com/item?id=3698652 https://news.ycombinator.com/item?id=3698652 (1 year ago) https://news.ycombinator.com/item?id=6368562 https://news.ycombinator.com/item?id=6368562 (2 months ago) https://news.ycombinator.com/item?id=1831322 https://news.ycombinator.com/item?id=1831322 (3 years ago, no comments)
jvdh 13y ago
So basically this is the inverse of swap. Using memory to make disk reads/writes faster, instead of using disk to make more memory. If Linux is consistently doing this, why would you still need a swap partition?
- ableal 13y ago
  > why would you still need a swap partition? You don't, but if something decides to grab memory, the OOM-killer will kill processes abruptly. Swap may allow the system to limp along until there's some solution. It's not like 10 or 20GB HD these days matter much either.
  - perlgeek 13y ago
    Though of course some people prefer the OOM-killer to kick in fast instead of the whole system being unresponsive for about an hour or so. That's why my desktop and laptop systems don't have swap enabled.
  - rosser 13y ago
    Would you rather consume physical pages of RAM by storing less recently used programs and data, or by caching more recently used disk pages? Myself, I'd generally prefer to evict the older stuff to swap in order to cache the newer stuff. But maybe that's just me.
    - ableal 13y ago
      Quite a while ago, HP-UX (v9, if memory serves) got this balance wrong - the system was keeping files in RAM, and kicking active processes to swap. Much gnashing of teeth ensued until HP fixed that ...
- Tuna-Fish 13y ago
  Because you can (and usually do) have areas of memory and disk that are more or less active, meaning it can make sense to evict some ram pages to disk while still caching other disk pages.
- 13y ago
AsymetricCom 13y ago
ITT tech startups learn about Lunix.
talles 13y ago
Or you can just http://downloadmoreram.com/ http://downloadmoreram.com/ (joke)
fit2rule 13y ago
Another thing, you know how sometimes a USB stick needs some time to un-mount? This non-sync'ed data (say, a copy of a big file) eventually has to be written to the physical disk, and through the little pipe, takes time. Find out how much of this "dirty" buffer you have yet to physically write to the device, while giving the kernel 'hints' to sync the dirty blocks quickly, with: while true; do sync && grep Dirty /proc/meminfo; done Lots of gold to be mined from /proc/meminfo .. bonus points for wiring it up to gnuplot so you can get a real picture of the dynamics of the Linux kernel .. ! :)
- voltagex_ 13y ago
  This seems a bit magic, what does it do? (stuck in Windows-land at the moment)
  - lcrs 13y ago
    I do this a lot, on machines which are annoyingly vendor-tweaked to allow a lot of dirty pages. Copying files to USB sticks happens instantly, but the unmount then takes an age - like, 20 minutes. This is a way to get a measure of the progress of the unmount. I generally have a sleep in there as well though ;)
Sami_Lehtinen 13y ago
I think it's still disinformative. Disk cache also uses RAM which is allocted and reserved by applications, of course data is swapped out in this situation. o you'll need more swap, but it isn't a bad thing at all. Even OS/2 did this a long time ago, nothing new at all.
xacaxulu 13y ago
I just love the website design :-) It makes me happy.
richo 13y ago
Not strictly accurate, in the "Why would you want to do this" regard. Ages ago I was hacking linux onto an ARM soc that claimed to have memory it didn't (it wasn't aware of the display's DMA segment). It was near the top of the address space, so generally speaking it didn't cause too many problems, but until I could debug it I needed to keep memory usage low, wchih meant disabling the disk cache. What a shitshow that was :/
dmourati 13y ago
I always run Ganglia on my servers. You can see memory used, memory shared, memory cached, memory buffered, and memory swapped all over time as well as total in-core memory. This has saved me (and my team) from ever having to misinterpret command line output. Takeaway, Linux is smart about using memory, probably smarter than you. http://ganglia.sourceforge.net/ http://ganglia.sourceforge.net/

Linux ate my RAM

194 points by wallzz 13y ago

jaseemabid 13y ago

Lovely! This domain could iframe that SO question :)

daGrevis 13y ago
What question?
- Schiphol 13y ago
  Why, that one, of course.
  - lucasnemeth 13y ago
    I've thought the answer to that one was 42.
    - Samuel_Michon 13y ago
      ITT: reddit-style comments that don’t get downvoted. I’m all for it. http://i.imgur.com/lrlFq0y.jpg http://i.imgur.com/lrlFq0y.jpg

daGrevis 13y ago

The page seems to be down. Here's how it looked: http://web.archive.org/web/20131011053519/http://www.linuxatemyram.com/ http://web.archive.org/web/20131011053519/http://www.linuxat...

makomk 13y ago

This is actually wrong. For some reason (probably because of the way it's implemented internally) Linux includes shared memory allocations in the "cached" figure. If you calculate the amount of free memory using "free -m" in the way this web page advises, you'll think that memory is free when it really isn't - unlike true cache, shared memory cannot be discarded when the space is needed for something else. I've actually had buggy desktop applications use several gigabytes of RAM as shared memory, so this can make a big difference.

appleflaxen 13y ago
> For some reason (probably because of the way it's implemented internally) (this is tautologically true)
- cma 13y ago
  he means the rationale behind developing it that way was due to the way other stuff is laid out internally, not "the number shows up because the existing code codes for it and causes it to show up".
gue5t 13y ago
Linux also counts files on tmpfs as part of the cache number--it's really annoying that no number in top can serve to show you a decent approximation of how many page-size malloc()s you can do before OOM (with overhead and fragmentation and so on, of course it's infeasible to get an exact number).
- marcosdumay 13y ago
  It's worse. No line of /proc/meminfo has that value, and since not all of the cache can be instantly discarded (I didn't know that), I don't see any combination of them that would help. That sentence: > There are no downsides, except for confusing newbies. Just isn't true. There is the downside that we can't know how much memory is available.
  - makomk 13y ago
    Yeah, and obviously some (though not all) forms of shared memory are actually implemented using tmpfs behind the scenes which makes life even more interesting. I think the problem I encountered involved SysV shared memory which isn't backed by tmpfs and does show up in /proc/meminfo, whereas other types may not.
  - mitchty 13y ago
    Can't you use something like this: https://raw.github.com/pixelb/ps_mem/master/ps_mem.py https://raw.github.com/pixelb/ps_mem/master/ps_mem.py I use that to help determine "least incorrect" ram usage on linux. It is ok, maybe not perfect but it at least lets you get a better view on memory.

jbogp 13y ago

Also HN ate your server apparently.

Morgawr 13y ago

Best way to check how much ram is actually free is to use a higher level tool like a task manager (from various DEs) or a command line tool like htop, they report the actual memory usage of processes

v-yadli 13y ago

https://www.kernel.org/doc/Documentation/sysctl/vm.txt https://www.kernel.org/doc/Documentation/sysctl/vm.txt You can drop cache by sending a byte into drop_cache.

mprovost 13y ago
But why would you do that? (The only time I've ever used that is when benchmarking and you need a clean cache between runs). Otherwise the kernel will just empty it as needed when you run a program.
- jrkatz 13y ago
  I can't answer for everyone, but I can tell you why I have a cron job dropping my cache every several minutes: The application I develop consumes most of the ram on my machine when running, and takes several minutes to rebuild and start up to begin with. When most of my memory is being used for the cache, this process takes several minutes longer, because I'm making millions upon millions of calls for more memory -- and each one has to get some of that cached memory back for itself. If I simply drop the cache all at once, every minute, it takes a split second. If I shrink it over a million increments, it takes around a minute. Even typing that I feel I must be doing something wrong; and yet, it worked.
  - deathanatos 13y ago
    Your app is making more calls to malloc when there is memory being used by disk cache, or you mean something else by "millions upon millions of calls for more memory"? You control hope many times you call malloc. Just call it once with what you need.
- v-yadli 13y ago
  First, it's not why. It's that this is possible. Please don't say "this is impossible because this doesn't make sense". Second, I agree with you. Normally people don't need to do this unless they hate their hard drives. :-) But there's some special cases, for example when the disk is "volatile", or when you want a huge chunk of continuous physical address and you want to tell the OS to stay away and don't touch any pages inside. PS: An interesting thing is that Windows people always seek for a way to enlarge disk cache. It's fine on server versions, but on desktop versions such as Windows 7, if I don't remember wrong, disk cache is capped at ~4GB(not sure if globally or per file). Oh well.

Damogran6 13y ago

FWIW, it's relevant to OS X too (even Mavericks)...'sudo purge' will reclaim some of the resources.

sounds 13y ago
OS X does not behave exactly in this manner, but is indeed confusing. Linux never needs a 'purge' or equivalent. Doing a 'drop_cache' only slows the system slightly because nothing is cached. 'sudo purge' can prevent OS X from using swap – i.e. running out of RAM even though there is RAM it could reclaim. This behavior has been tweaked but not fixed in Mavericks.
- Zr40 13y ago
  Actually, purge is completely unrelated to swap. Purge forces disk cache to be flushed to disk, and can be used to approximate a cold disk buffer cache scenario for performance analysis. Some small amount of swap usage (compared to total RAM) is perfectly normal, by the way. More cache, for instance, is a better use of RAM than avoiding swap usage by keeping never-accessed pages present.

free652 13y ago

Windows loves to cache and "prefetch" apps, I still need to figure out how to disable prefetching since I have a very fast SSD.

csmuk 13y ago
Why? All it does is make it go faster.
- TBastiani 13y ago
  And drain your battery for no reason.
  - hansbogert 13y ago
    Do you have any proof of that? "Free" memory uses as much energy as "used" memory, there is no distinction in SD|DDR-RAM. And if the OS is smart, it only has to keep a simple index of memory pages still relevant. I'd argue that this has a measurable battery-hit.
    - jeltz 13y ago
      Reading from the SSD to laod it into RAM does drain battery. No idea if it is enough to matter though.
    - runako 13y ago
      You're only accounting for the RAM power. Accessing the SSD costs power, too.
      - tedunangst 13y ago
        Then I think I'd rather Windows prefetched my programs while I'm plugged in so it doesn't have to do so when I'm on battery.
      - tanzam75 13y ago
        > You're only accounting for the RAM power. Accessing the SSD costs power, too. A typical modern SSD reads at 500 MiB/s, consumes 3 watts when reading, and 0.3 watts when idle. A stick of DDR3 RAM consumes about 3 watts, just sitting idle. The power consumption from prefetching is minor compared to that of the RAM.

jheriko 13y ago

right. so there is a bug in linux. gotcha.

annnnd 13y ago
No. Well, not technically... There is a UX bug in Linux. Not the only one at that, but they don't count much in Linux-land.
- hansbogert 13y ago
  Just a difference in semantics -- I mean, c'mon UX in a utility like top? which is probably specified in some Unix convention anyway. Pure BSD Unix do this as well in their own way.
CmonDev 13y ago
Free is used, black is white, truth is a lie, lie is a truth.
gmuslera 13y ago
Leaving your memory with no use at all would be the bug. It is a free resource that could be used to improve system performance at basically no cost (the disk is already read, the memory is still available immediately for any app that request it, etc), so it is a good default to let linux do it. But could be cases where it could be side effects of it (i.e. running in a VM could make less memory available for the host, and it could be caching disk already) so you could consider to disable it sometimes.
jheriko 13y ago
it confuses users. its a bug. thank you for the -4 points... :)

caw 13y ago

Wow, I can't believe how many times I've had to explain this to people as they log on to shared VNC servers. "I thought this had 16G of RAM free, I asked for a 16G machine!". "It does, stop trying to read top!" Now I'll just point them to this website.

pearjuice 13y ago

Then you should http://downloadmoreram.com http://downloadmoreram.com

blowski 13y ago
Haha, that's brilliant. From their Twitter account: > Have an iPhone 3? Download RAM twice to make it a iPhone 5. https://twitter.com/DownloadMoreRam/status/385026260008587264 https://twitter.com/DownloadMoreRam/status/38502626000858726...
10098 13y ago
Crap, I wanted to post that!
puppypilgrim 13y ago
Brilliant. I love it.
unimpressive 13y ago
When my circle of friends found that site it became an in-joke. I'd never heard of it, and eventually they came up to me and asked "Can you download more RAM?". I don't think they were expecting my answer: "Sure you can, just use plan 9."

nraynaud 13y ago

that's funny, like avery complex or "smart" algorithm, if it works it's cool, but when something doesn't work, it makes it very hard to pinpoint a culprit. I'm more and more inclined to either use stupider algorithm that at least I can explain to the user, incremental stuff where they can check intermediate results or taking into account the fact that this algorithm has to be conveyed to users in one way or another since inception. I guess in this case a measure of the swap activity would be a good indicator that the current limiting factor is the RAM, but it also has to be credible. Somehow having 16G written on the box of the ram and 7G written in the "available ram" of a software makes everything more credible that saying "I have a pressure of 1.5madeupunit on the ram front". Human factors, again.

lamontcg 13y ago
I've been doing performance analysis and debugging as part of being a unix/linux sysadmin now for something like 20 years. And even though I have many well-cherished anecdotes about explaining the VM free list to software architects who were the go-to guys for the vision of our entire global platform, complete with PhDs, I still don't want to go back to the bad old days without a shared buffer cache. There's a reason why that's there, and you don't want to go back to 1992-era memory management. It also makes perfect sense from the perspective of writing the software. The free list really is the list of VM pages that have been free'd and that has a specific meaning, and changing that name internally would be terrible. You just need to understand how the machine actually works.

dancecodes 13y ago

Cant load page while. But Linux RAM problems can be in desktop when something wrong with video systems (X, drivers, compositing, etc...) Check first: xorg, video drivers, mesa. Sometime java apps makes big swaps. Check your linux distro also. Try learning Gentoo.

andmarios 13y ago

This is the worst piece of Linux advice ever. I use Linux for many years and every time I had a memory leak or a process using too much memory, every google search about the problem would lead to some "guru" giving this advice. This explanation which has been replicated thousands of times on every linux forum on the planet, prevents people from solving real problems. It started as a useful piece of information but nowadays it is spam.

jlgreco 13y ago
How does this understanding prevent people from solving real problems, simply by virtue of it being commonly cited? I don't understand your complaint.
- marcosdumay 13y ago
  I understand this is a complaint about Google. It won't help you find pages that cite a problem when a diffrent, but similar problem is too popular. I've seen that happening several times, but I'm not the GP, so I may still be missunderstanding him.
  - andmarios 13y ago
    Yes, this is exactly what I mean. Maybe I should've describe it as noise rather than spam. But it isn't only a Google search issue. Every time you will ask a question about a memory problem in Linux, everybody will give you this answer and classify you as newbie. When you explain them that this isn't the case, they will simply leave the thread/question and nobody would ever bother reading your problem again. People just go in denial mode because of this common knowledge, as Linux can't have memory leaks or any memory related problem.
    - jlgreco 13y ago
      Just give more information right off the bat. Don't say "XYZ is chewing up lots of RAM"; the best case scenario if you say that is they have to respond back asking for more information. Give that information to them in the first place "XYZ starts at XXXXMB shared and XXMB resident. Over five minutes it steadily climbs to XXXXMB resident. Shared remains constant at XXXMB." Doing this, I can't say I've ever been brushed off using this page.

deleted 13y ago

[deleted]

hippich 13y ago

That's how I check true free memory available: root@joe-pc:~# free -m total used free shared buffers cached Mem: 32057 30347 1710 0 1312 21568 -/+ buffers/cache: 7466 24591 Swap: 0 0 0 root@joe-pc:~# sync; echo 3 > /proc/sys/vm/drop_caches root@joe-pc:~# free -m total used free shared buffers cached Mem: 32057 5628 26429 0 4 891 -/+ buffers/cache: 4732 27325 Swap: 0 0 0 root@joe-pc:~#

dice 13y ago
Yikes! Hopefully not on production systems. I've had fileservers which needed 60+GB of dentry cache to serve up files in a reasonable time, something like that could bring the site to a crawl for 10 minutes or more.
- hippich 13y ago
  Good point. I am doing it on my laptop and server without load. So this is not a big problem in my cases.

bediger4000 13y ago

How bizarre. In the 1993-94 time frame, one of the big talking points about Windows NT was that it was the only OS with an integrated file system cache. That wasn't true even at that time, as Solaris also had such a cache, but such is the nature of propaganda. I'm tickled that what once was a big "technical" advantage becomes a major source of confusion. Presumably, OSes of Windows NT heritage have retained file system caching, which offers a huge performance benefit. Why don't Windows users have confusion over this issue? Doesn't Windows offer an easy way to check or is the whole thing presented clearly, or is Windows so complicated not many people get this far? I'll also draw a parallel to DUI - without a breath-o-meter, DUI is a lot like driving while distracted. It's impossible to quantify, and therefore not that much of a crime. With a breath-o-meter, or with "top", it's possible to put a number on drunkeness or lack of free memory, and the whole thing becomes a problem.

chiph 13y ago
From what I can tell, Windows does not show the disk cache allocation in Task Manager. So even though all the RAM is likely in use, that is hidden from the user. You can probably see it in PerfMon, as that is less "typical user" oriented.
- rplnt 13y ago
  It's not in the graph but it's there in the "Cached" field.
- tracker1 13y ago
  In windows 7, take "available" and subtract "free", that's your cache usage. Windows Vista started using RAM very aggressively for caching, including pre-filling cache with frequently used pages (apps/data) from disk... ReadyBoost was basically an extension of this...
Lagged2Death 13y ago
Why don't Windows users have confusion over this issue? Windows' Task Manager has generally counted memory occupied by the disk cache as "available."

nly 13y ago

Forget the kernel, someone really needs to do chromeatemyram.com

FlyingAvatar 13y ago

I hate the invention of "burstable" RAM on VPSes for this reason. Eventually your IO cache grows until it has most of your guaranteed RAM. Now every new process is playing a game of Russian roulette with your hypervisor to see if it can actually allocate into your "burstable" RAM what the kernel is reporting is freely available.

vajrabum 13y ago
Actually this is a general issue when running multiple Linux VMs under a hypervisor. In that case you don't want them filling up RAM with prefetch data from disk because that's likely to cause contention and reduce page sharing between VMs.

rdtsc 13y ago

A quick practical answer is to use htop and look at the memory bar. * Green = resident memory used (probably what you want most of the time) * Blue = buffers * Yellow = cache

ChuckMcM 13y ago
+1 for htop, it not only shows excellent memory stats in an easy to absorb way but it also shows CPU cores usage. Definitely one of my "wtf is going on?" tools.
hisham_hm 13y ago
htop author here. It was nice seeing the table in the article with the memory split in three colors and thinking "well, but that's how I already split it in htop, isn't it?". The "Linux calls it" column should probably be called "top calls it"...
- rdtsc 13y ago
  Thank you for writing htop, it is very useful tool. I just about fell off my chair when I realized I could click with my mouse on the columns to sort them!
  - gregf 13y ago
    I always forget the key bindings for this, you just changed my life!
    - rdtsc 13y ago
      It was accidental too. I was just showing my coworker something and I have a bad habit of highlighting stuff I am reading and randomly clicking on text, and I noticed that column was selected to be sorted. It blew my mind, this was in an SSH session logged into a remote server.
sciurus 13y ago
I'll give a pitch for atop as well. Here's a screenshot of it showing systemwide and per-process memory information: http://www.atoptool.nl/images/screenshots/memory.png http://www.atoptool.nl/images/screenshots/memory.png Here's a case study of using it to observe a memory leak: http://www.atoptool.nl/download/case_leakage.pdf http://www.atoptool.nl/download/case_leakage.pdf

usamec 13y ago

Ehm. But sometimes stupid linux kernel prefers moving something usefull to swap and caching some shit instead of keeping stuff in ram and not doing caching. So linux sometimes eats your RAM (but in different way).

Amadou 13y ago
For situations where you have a time-critical app, you can pin the app's memory into RAM with the mlock() family of functions. There may be a command-line utility that lets you pin an arbitrary running a process too, I haven't looked for it on linux.

jared314 13y ago

Previous Discussions: https://news.ycombinator.com/item?id=3698652 https://news.ycombinator.com/item?id=3698652 (1 year ago) https://news.ycombinator.com/item?id=6368562 https://news.ycombinator.com/item?id=6368562 (2 months ago) https://news.ycombinator.com/item?id=1831322 https://news.ycombinator.com/item?id=1831322 (3 years ago, no comments)

jvdh 13y ago

So basically this is the inverse of swap. Using memory to make disk reads/writes faster, instead of using disk to make more memory. If Linux is consistently doing this, why would you still need a swap partition?

ableal 13y ago
> why would you still need a swap partition? You don't, but if something decides to grab memory, the OOM-killer will kill processes abruptly. Swap may allow the system to limp along until there's some solution. It's not like 10 or 20GB HD these days matter much either.
- perlgeek 13y ago
  Though of course some people prefer the OOM-killer to kick in fast instead of the whole system being unresponsive for about an hour or so. That's why my desktop and laptop systems don't have swap enabled.
- rosser 13y ago
  Would you rather consume physical pages of RAM by storing less recently used programs and data, or by caching more recently used disk pages? Myself, I'd generally prefer to evict the older stuff to swap in order to cache the newer stuff. But maybe that's just me.
  - ableal 13y ago
    Quite a while ago, HP-UX (v9, if memory serves) got this balance wrong - the system was keeping files in RAM, and kicking active processes to swap. Much gnashing of teeth ensued until HP fixed that ...
Tuna-Fish 13y ago
Because you can (and usually do) have areas of memory and disk that are more or less active, meaning it can make sense to evict some ram pages to disk while still caching other disk pages.
13y ago

AsymetricCom 13y ago

ITT tech startups learn about Lunix.

talles 13y ago

Or you can just http://downloadmoreram.com/ http://downloadmoreram.com/ (joke)

fit2rule 13y ago

Another thing, you know how sometimes a USB stick needs some time to un-mount? This non-sync'ed data (say, a copy of a big file) eventually has to be written to the physical disk, and through the little pipe, takes time. Find out how much of this "dirty" buffer you have yet to physically write to the device, while giving the kernel 'hints' to sync the dirty blocks quickly, with: while true; do sync && grep Dirty /proc/meminfo; done Lots of gold to be mined from /proc/meminfo .. bonus points for wiring it up to gnuplot so you can get a real picture of the dynamics of the Linux kernel .. ! :)

voltagex_ 13y ago
This seems a bit magic, what does it do? (stuck in Windows-land at the moment)
- lcrs 13y ago
  I do this a lot, on machines which are annoyingly vendor-tweaked to allow a lot of dirty pages. Copying files to USB sticks happens instantly, but the unmount then takes an age - like, 20 minutes. This is a way to get a measure of the progress of the unmount. I generally have a sleep in there as well though ;)

Sami_Lehtinen 13y ago

I think it's still disinformative. Disk cache also uses RAM which is allocted and reserved by applications, of course data is swapped out in this situation. o you'll need more swap, but it isn't a bad thing at all. Even OS/2 did this a long time ago, nothing new at all.

xacaxulu 13y ago

I just love the website design :-) It makes me happy.

richo 13y ago

Not strictly accurate, in the "Why would you want to do this" regard. Ages ago I was hacking linux onto an ARM soc that claimed to have memory it didn't (it wasn't aware of the display's DMA segment). It was near the top of the address space, so generally speaking it didn't cause too many problems, but until I could debug it I needed to keep memory usage low, wchih meant disabling the disk cache. What a shitshow that was :/

dmourati 13y ago

I always run Ganglia on my servers. You can see memory used, memory shared, memory cached, memory buffered, and memory swapped all over time as well as total in-core memory. This has saved me (and my team) from ever having to misinterpret command line output. Takeaway, Linux is smart about using memory, probably smarter than you. http://ganglia.sourceforge.net/ http://ganglia.sourceforge.net/