4 ms·

This article correctly states that committed memory is that in use + memory that's being paged out. Now why would you want to know the committed memory over the

This article correctly states that committed memory is that in use + memory that's being paged out. Now why would you want to know the committed memory over the actual physical RAM in use? I can trivially create an app that memory maps a massive file and will show several GB of committed memory. This won't be in use of course, memory mapping files so that the OS will page in/out as required is intentional. Those GB of committed memory aren't something you should care about. I'd be scared if someone looked at the committed memory use of a program that correctly uses mmap and caused someone to exclaim "OMG this is uses TB of RAM!". Task Manager is doing the right thing here. It's showing you want's actually paged in and in use right now.

quotemstr 8y ago
> committed memory is that in use + memory that's being paged out That's not actually true though. You can see an increase in a process's commit charge without anything new being written to the pagefile. Commit is a check that the kernel writes to applications; you're confusing that check for cash in a wallet. You can also have commit without any virtual address space to blame for it through section handles or other tricks. It's complicated.
- AGoodName 8y ago
  I'm not going to dispute that but just want to highlight it doesn't change the fact that committed RAM can show as extremely high just by working with memory mapped files. Memory mapping a GB log file for example will absolutely show GB's of committed memory but in reality you'll only have the last page in actual physical RAM.
  - quotemstr 8y ago
    Right. When you memory-map a file, what you've essentially done is add a temporary new pagefile to the system (your mapped file), and when you work with memory backed by that file, it's no different from working with "anonymous" memory backed by the system-wide pagefile.
    - nothrabannosir 8y ago
      Aside: I have to say this is a genius perspective on mmap, and I now finally understand why the same syscall is used for both. I never understood the link. Thank you.
- _cs2017_ 8y ago
  I understand commit can happen without physical page file being written to. But you say commit can happen even without any increase in virtual address space usage? That seems strange, could you explain how it can happen?
  - quotemstr 8y ago
    Change a PAGE_READONLY mapping to a PAGE_WRITECOPY one. The commit charge is billed at VirtualProtect time, and the protection can fail if you run out of commit. The kernel doesn't have to commit anything for a PAGE_READONLY mapping because all pages in such a mapping are guaranteed to be clean and trivially evictable. Not so once you introduce the possibility of COW faults. Reserving a big range of address space and committing it a little bit at a time is a very common pattern. Another thing: create a 1GB section. Map it, and fill it up. Unmap it. Map it again. What you wrote is still there. Between the map and remap, you have commit without corresponding address space.
dijit 8y ago
OOOOOOO, something I actually know! > Now why would you want to know the committed memory over the actual physical RAM in use? Because in Windows, committed size is relative to physical size. You can commit a lot more than RAM, but watch your page-file grow. malloc() can fail on Windows for this reason. This is not the same on Linux or any of the BSD's I've tried. :) I experienced/discovered this August last year.. sometimes understanding a lot about Linux can make you blind to the architectural differences Windows has. I wrote some cross-platform CPP to show it[0] [0]: https://gist.github.com/dijit/cb2caa1a40d48e03613f5af0e518d626 https://gist.github.com/dijit/cb2caa1a40d48e03613f5af0e518d6...
- AGoodName 8y ago
  This doesn't occur if you memory map a file though (barring certain flags that you can set as stated by quotemstr below) You can legitimately have Windows stating many GB's of committed RAM without actually using that RAM and it's not using the systems pagefile/swap. It's also common for this to occur. Pretty much every program capable of opening large files (GB+) in a non-sequential fashion does this.
  - dijit 8y ago
    Hm, maybe I wasn't clear. It's not actually /using/ the memory when it's committed. But the sum of your committed memory across all applications must exist in some form on the host system. So, for example it's a common performance optimisation to double the amount of allocated space when you grow anything in C++, what this means is that you're not actually using the space yet but malloc() and zeroing is kinda slow. So, you have 128MB of ram which is your programs address space and you just doubled your array from 75MB to 150MB, well, that extra 22MB must exist. Even though you're only using 76MB.. even if the OS shows it as free. (which, it will) Thems the rules, and I promise you that I have thoroughly tested this; as it was causing a really nice crash on my servers even though we had more than 50% of memory "free"
    - AGoodName 8y ago
      Memory Mapped files do exist in some form or another though. As the file itself. That's the point of memory mapping files. You can right now memory map every file on your computer. That's TB of files. There will be no physical RAM usage and no swap file usage unless you start to actually work with those files (at which point they will be paged in). This will show as 'committed memory' in task manager. Your example above isn't memory mapping files. It's just allocating RAM. That does have to exist in RAM or the swap file. But that's not what 'committed memory' above shows. Which is the whole point. The column the article is telling people to use is misleading.
gameswithgo 8y ago
>Task Manager is doing the right thing here I had a problem on my windows 10 pc for a long time, where I would clearly be running out of ram, task manager would show 100% usage, everything getting slow. However if I added up all of the processes using ram, it was nowhere close. So something was using up ram that task manager gave me no visibility into. I had to download obscure tools and do some guesswork to figure it out. It would have been really nice if Task Manager could just report it in the first place (it turned out to be a network card driver with a memory leak in it)
- fphhotchips 8y ago
  Yep, this is particularly evident if you use Virtualbox a lot - the memory use isn't in the actual process but in one of the drivers. Has also happened to me with a VPN drivers. Best tool I've seen to start finding where it's going is RAMmap [1]. 1. https://docs.microsoft.com/en-us/sysinternals/downloads/rammap https://docs.microsoft.com/en-us/sysinternals/downloads/ramm...

quotemstr 8y ago

> committed memory is that in use + memory that's being paged out That's not actually true though. You can see an increase in a process's commit charge without anything new being written to the pagefile. Commit is a check that the kernel writes to applications; you're confusing that check for cash in a wallet. You can also have commit without any virtual address space to blame for it through section handles or other tricks. It's complicated.

AGoodName 8y ago
I'm not going to dispute that but just want to highlight it doesn't change the fact that committed RAM can show as extremely high just by working with memory mapped files. Memory mapping a GB log file for example will absolutely show GB's of committed memory but in reality you'll only have the last page in actual physical RAM.
- quotemstr 8y ago
  Right. When you memory-map a file, what you've essentially done is add a temporary new pagefile to the system (your mapped file), and when you work with memory backed by that file, it's no different from working with "anonymous" memory backed by the system-wide pagefile.
  - nothrabannosir 8y ago
    Aside: I have to say this is a genius perspective on mmap, and I now finally understand why the same syscall is used for both. I never understood the link. Thank you.
_cs2017_ 8y ago
I understand commit can happen without physical page file being written to. But you say commit can happen even without any increase in virtual address space usage? That seems strange, could you explain how it can happen?
- quotemstr 8y ago
  Change a PAGE_READONLY mapping to a PAGE_WRITECOPY one. The commit charge is billed at VirtualProtect time, and the protection can fail if you run out of commit. The kernel doesn't have to commit anything for a PAGE_READONLY mapping because all pages in such a mapping are guaranteed to be clean and trivially evictable. Not so once you introduce the possibility of COW faults. Reserving a big range of address space and committing it a little bit at a time is a very common pattern. Another thing: create a 1GB section. Map it, and fill it up. Unmap it. Map it again. What you wrote is still there. Between the map and remap, you have commit without corresponding address space.

dijit 8y ago

OOOOOOO, something I actually know! > Now why would you want to know the committed memory over the actual physical RAM in use? Because in Windows, committed size is relative to physical size. You can commit a lot more than RAM, but watch your page-file grow. malloc() can fail on Windows for this reason. This is not the same on Linux or any of the BSD's I've tried. :) I experienced/discovered this August last year.. sometimes understanding a lot about Linux can make you blind to the architectural differences Windows has. I wrote some cross-platform CPP to show it[0] [0]: https://gist.github.com/dijit/cb2caa1a40d48e03613f5af0e518d626 https://gist.github.com/dijit/cb2caa1a40d48e03613f5af0e518d6...

AGoodName 8y ago
This doesn't occur if you memory map a file though (barring certain flags that you can set as stated by quotemstr below) You can legitimately have Windows stating many GB's of committed RAM without actually using that RAM and it's not using the systems pagefile/swap. It's also common for this to occur. Pretty much every program capable of opening large files (GB+) in a non-sequential fashion does this.
- dijit 8y ago
  Hm, maybe I wasn't clear. It's not actually /using/ the memory when it's committed. But the sum of your committed memory across all applications must exist in some form on the host system. So, for example it's a common performance optimisation to double the amount of allocated space when you grow anything in C++, what this means is that you're not actually using the space yet but malloc() and zeroing is kinda slow. So, you have 128MB of ram which is your programs address space and you just doubled your array from 75MB to 150MB, well, that extra 22MB must exist. Even though you're only using 76MB.. even if the OS shows it as free. (which, it will) Thems the rules, and I promise you that I have thoroughly tested this; as it was causing a really nice crash on my servers even though we had more than 50% of memory "free"
  - AGoodName 8y ago
    Memory Mapped files do exist in some form or another though. As the file itself. That's the point of memory mapping files. You can right now memory map every file on your computer. That's TB of files. There will be no physical RAM usage and no swap file usage unless you start to actually work with those files (at which point they will be paged in). This will show as 'committed memory' in task manager. Your example above isn't memory mapping files. It's just allocating RAM. That does have to exist in RAM or the swap file. But that's not what 'committed memory' above shows. Which is the whole point. The column the article is telling people to use is misleading.

gameswithgo 8y ago

>Task Manager is doing the right thing here I had a problem on my windows 10 pc for a long time, where I would clearly be running out of ram, task manager would show 100% usage, everything getting slow. However if I added up all of the processes using ram, it was nowhere close. So something was using up ram that task manager gave me no visibility into. I had to download obscure tools and do some guesswork to figure it out. It would have been really nice if Task Manager could just report it in the first place (it turned out to be a network card driver with a memory leak in it)

fphhotchips 8y ago
Yep, this is particularly evident if you use Virtualbox a lot - the memory use isn't in the actual process but in one of the drivers. Has also happened to me with a VPN drivers. Best tool I've seen to start finding where it's going is RAMmap [1]. 1. https://docs.microsoft.com/en-us/sysinternals/downloads/rammap https://docs.microsoft.com/en-us/sysinternals/downloads/ramm...