7 ms·

Okay, I'm one of those devs. What is this virtual/resident set size thing you're talking about? EDIT: Thank you for all the helpful responses!

ivan78 15y ago
Well, if you don't understand it, you must read some basic book about modern computer architecture. Short answer: It is possible to reserve memory address space without assigning actual physical memory. As your program runs it can dynamically assign physical memory pages to its address space when it needs it. Again - you must read a book if you want to understand it.
- ivan78 15y ago
  Why downvoting? It may be not the best explanation, but IMO the closest to the way a "developer" should understand it. On Linux a virtual memory is committed with mmap call. Check this for details: http://stackoverflow.com/questions/2782628/any-way-to-reserve-but-not-commit-memory-in-linux http://stackoverflow.com/questions/2782628/any-way-to-reserv... This is basically how shared libraries and other shared objects are attached to your process.
  - henrikschroder 15y ago
    > Why downvoting? > you must read a book if you want to understand it. "Go read a book" is not a very constructive or helpful answer to a question that - as it turns out - could be answered with a brief comment.
    - onemoreact 15y ago
      It can be when a short description completely and accurately describes what's going on, but the person creating the comment needed to read a few books on the subject to actually understand both what's happening and why due to other issues. EX: How does virtual memory get mapped to L1 cache?
  - ivan78 15y ago
    to henrikschroder: I expected that this reaction was caused by mentioning a book. It is my deepest belief that questions like memory management can not be answered with brief comment. You must read at least a basic book if you want to understand it in depth.
Steve_Baker 15y ago
The virtual size of a program includes all shared objects (shared libraries, shared copy on write memory pages, the read only executable pages and other shared memory) that a process uses in addition to the memory that only that particular process is using (its resident set size.) Shared objects like shared libraries can be mapped into memory by multiple processes and thus don't use any additional physical memory for each additional process that uses it. The RSS thus usually indicates the amount of heap and stack a process is using that is unique to it.
- mgedmin 15y ago
  RSS also includes shared memory pages (those that are currently resident).
illumin8 15y ago
Short and overly simple answer: If a program allocates 2GB of RAM, it might not need to use it all right away. The kernel pretends like it just gave the program 2GB of RAM, but it doesn't really need to flush all those unused pages of cache yet. It can still use them for other things like disk cache, and when the program tries to access those pages then the kernel will dynamically page them in and out as needed. This explains why a program might have 2GB virtual but only 128MB is resident in actual physical memory pages. This is overly simple and there are a lot of nuances such as shared memory segments, etc.
- adavies42 15y ago
  then there's over-commit, e.g. "you want to malloc 200% of the ram? here you go! enjoy! oh wait, you wrote to the second half? have fun finding the power button, sucker!" can someone explain to me why so many distros, including "enterprise" server stuff, ship with /proc/sys/vm/overcommit_memory set to 1?
  - aliguori 15y ago
    Most software cannot cope with malloc failing. A little swapping is at least recoverable. A failed malloc almost guarantees that your application will crash--sometimes in a very unpleasant way.
    - adavies42 15y ago
      ime the main alternative is the box crashing, frequently without leaving behind enough information to know what went wrong. at least if the app crashes you have a pretty good idea who was incompetent.
maushu 15y ago
I'm a developer too and this is what I understand (might be wrong): - Virtual memory is basically "abstract memory" that is linked (mapped) to RAM or HDD, this means that by accessing this "memory" you might actually be accessing the HDD and, because of this, larger than physical ram. - Virtual set size is the allocated virtual memory (the above) to the process. - Resident set size is the allocated physical (RAM) memory to the process. - Shared memory is memory that is shared on multiple processes, meaning that if you have 10 processes using 10mb of resident memory each and have 2mb of shared memory each, the total of resident memory used is not 100mb but 82mb.
- scott_s 15y ago
  Virtual memory is actually more subtle than that. At a mechanical level, virtual memory is permission from the operating system to use addresses in your address space. It is so called because, as you point out, it allows us to separate the concept of "memory for a process" from "physical memory on a chip." The reason I further refine the concept is that allocating virtual memory does not allocate actual memory. Let's look at an example: void* addr = mmap(NULL, 10 * 1024 * 1024, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); That call allocates 10 MB of virtual memory. But there is no physical memory backing any of it. All that has happened is that the operating system has now said "Okay, starting at the address I return to you, you can now access 10 MB of memory. I will do all of the work of making sure physical memory backs the virtual memory when you access it." That is, once I try to access the memory, it will trigger a page fault, and the OS will find a page in physical memory to back my virtual memory. But until that happens, no memory - not in RAM, not on disk - backs the virtual memory.
- qwe123_troll 15y ago
  No, not at all. Virtual memory isn't really memory at all, it's just an entry in a table which marks addresses as accessible to your process. It's up to the OS to decide whether or not these addresses correspond to anything physical or not. In Linux (and every other modern OS, AFAIK) the OS allocates physical memory on-the-fly, only when your program actually starts reading or writing addresses in the virtual memory table. Thus, on a 64-bit machine you can allocate thousands of gigabytes of "virtual memory" without using any real memory at all.
sciurus 15y ago
There's a good overview of this in the first four pages of http://www.atoptool.nl/download/case_leakage.pdf http://www.atoptool.nl/download/case_leakage.pdf The rest of that document shows how to use atop to monitor per-process memory usage and identify a memory leak.
yread 15y ago
On Windows it's called committed memory http://blogs.technet.com/b/markrussinovich/archive/2008/11/17/3155406.aspx#Committed_Memory http://blogs.technet.com/b/markrussinovich/archive/2008/11/1... (the article doesn't have an anchor there... but you know what I mean)

ivan78 15y ago

Well, if you don't understand it, you must read some basic book about modern computer architecture. Short answer: It is possible to reserve memory address space without assigning actual physical memory. As your program runs it can dynamically assign physical memory pages to its address space when it needs it. Again - you must read a book if you want to understand it.

ivan78 15y ago
Why downvoting? It may be not the best explanation, but IMO the closest to the way a "developer" should understand it. On Linux a virtual memory is committed with mmap call. Check this for details: http://stackoverflow.com/questions/2782628/any-way-to-reserve-but-not-commit-memory-in-linux http://stackoverflow.com/questions/2782628/any-way-to-reserv... This is basically how shared libraries and other shared objects are attached to your process.
- henrikschroder 15y ago
  > Why downvoting? > you must read a book if you want to understand it. "Go read a book" is not a very constructive or helpful answer to a question that - as it turns out - could be answered with a brief comment.
  - onemoreact 15y ago
    It can be when a short description completely and accurately describes what's going on, but the person creating the comment needed to read a few books on the subject to actually understand both what's happening and why due to other issues. EX: How does virtual memory get mapped to L1 cache?
- ivan78 15y ago
  to henrikschroder: I expected that this reaction was caused by mentioning a book. It is my deepest belief that questions like memory management can not be answered with brief comment. You must read at least a basic book if you want to understand it in depth.

Steve_Baker 15y ago

The virtual size of a program includes all shared objects (shared libraries, shared copy on write memory pages, the read only executable pages and other shared memory) that a process uses in addition to the memory that only that particular process is using (its resident set size.) Shared objects like shared libraries can be mapped into memory by multiple processes and thus don't use any additional physical memory for each additional process that uses it. The RSS thus usually indicates the amount of heap and stack a process is using that is unique to it.

mgedmin 15y ago
RSS also includes shared memory pages (those that are currently resident).

illumin8 15y ago

Short and overly simple answer: If a program allocates 2GB of RAM, it might not need to use it all right away. The kernel pretends like it just gave the program 2GB of RAM, but it doesn't really need to flush all those unused pages of cache yet. It can still use them for other things like disk cache, and when the program tries to access those pages then the kernel will dynamically page them in and out as needed. This explains why a program might have 2GB virtual but only 128MB is resident in actual physical memory pages. This is overly simple and there are a lot of nuances such as shared memory segments, etc.

adavies42 15y ago
then there's over-commit, e.g. "you want to malloc 200% of the ram? here you go! enjoy! oh wait, you wrote to the second half? have fun finding the power button, sucker!" can someone explain to me why so many distros, including "enterprise" server stuff, ship with /proc/sys/vm/overcommit_memory set to 1?
- aliguori 15y ago
  Most software cannot cope with malloc failing. A little swapping is at least recoverable. A failed malloc almost guarantees that your application will crash--sometimes in a very unpleasant way.
  - adavies42 15y ago
    ime the main alternative is the box crashing, frequently without leaving behind enough information to know what went wrong. at least if the app crashes you have a pretty good idea who was incompetent.

maushu 15y ago

I'm a developer too and this is what I understand (might be wrong): - Virtual memory is basically "abstract memory" that is linked (mapped) to RAM or HDD, this means that by accessing this "memory" you might actually be accessing the HDD and, because of this, larger than physical ram. - Virtual set size is the allocated virtual memory (the above) to the process. - Resident set size is the allocated physical (RAM) memory to the process. - Shared memory is memory that is shared on multiple processes, meaning that if you have 10 processes using 10mb of resident memory each and have 2mb of shared memory each, the total of resident memory used is not 100mb but 82mb.

scott_s 15y ago
Virtual memory is actually more subtle than that. At a mechanical level, virtual memory is permission from the operating system to use addresses in your address space. It is so called because, as you point out, it allows us to separate the concept of "memory for a process" from "physical memory on a chip." The reason I further refine the concept is that allocating virtual memory does not allocate actual memory. Let's look at an example: void* addr = mmap(NULL, 10 * 1024 * 1024, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); That call allocates 10 MB of virtual memory. But there is no physical memory backing any of it. All that has happened is that the operating system has now said "Okay, starting at the address I return to you, you can now access 10 MB of memory. I will do all of the work of making sure physical memory backs the virtual memory when you access it." That is, once I try to access the memory, it will trigger a page fault, and the OS will find a page in physical memory to back my virtual memory. But until that happens, no memory - not in RAM, not on disk - backs the virtual memory.
qwe123_troll 15y ago
No, not at all. Virtual memory isn't really memory at all, it's just an entry in a table which marks addresses as accessible to your process. It's up to the OS to decide whether or not these addresses correspond to anything physical or not. In Linux (and every other modern OS, AFAIK) the OS allocates physical memory on-the-fly, only when your program actually starts reading or writing addresses in the virtual memory table. Thus, on a 64-bit machine you can allocate thousands of gigabytes of "virtual memory" without using any real memory at all.

sciurus 15y ago

There's a good overview of this in the first four pages of http://www.atoptool.nl/download/case_leakage.pdf http://www.atoptool.nl/download/case_leakage.pdf The rest of that document shows how to use atop to monitor per-process memory usage and identify a memory leak.

yread 15y ago

On Windows it's called committed memory http://blogs.technet.com/b/markrussinovich/archive/2008/11/17/3155406.aspx#Committed_Memory http://blogs.technet.com/b/markrussinovich/archive/2008/11/1... (the article doesn't have an anchor there... but you know what I mean)