4 ms·

In theory, a system without overcommit can run just as efficiently, IF you have reserved huge amounts of swap space. As long as swap is available, the OS can do

In theory, a system without overcommit can run just as efficiently, IF you have reserved huge amounts of swap space. As long as swap is available, the OS can do all the same COW and efficiency tricks. It’s not the physically backed memory is the limiting factor, it’s RAM+swap

dekhn 5y ago
no, the kernel maintains other allocated objects (dentry, inode caches) that it can't swap out. Under memory pressure, those get dropped before application pages. See https://unix.stackexchange.com/questions/17936/setting-proc-sys-vm-drop-caches-to-clear-cache https://unix.stackexchange.com/questions/17936/setting-proc-... and https://unix.stackexchange.com/questions/111893/how-long-do-dentries-stay-in-the-dcache https://unix.stackexchange.com/questions/111893/how-long-do-... I've found the linux memory system to be far too complicated to understand for quite some time, compared to what's documented in, for example, The Design and Implementation of The FreeBSD Operating System, for a more comprehensible system.
- joosters 5y ago
  Those objects must exist with and without overcommit, I don’t understand why they must make one less efficient than the other. (I’m talking in general here - not Linux specifically)
  - dekhn 5y ago
    dentry caches exist with and without overcommit, but you get higher cache hit rates with overcommit, because you flush them less recently. Depending on workload, this can matter a lot. It mattered more in the time of hard drives.
    - joosters 5y ago
      I’m sorry, but I’m still not following… why must a cache be (in theory) flushed more often without overcommit?
      - dekhn 5y ago
        if overcommit is disabled, then the system drops its internal caches (dentry cache, pagecache) to satisfy a memory allocation. Since most applications don't touch pages they allocate, that means the OS could have avoided dropping the caches. Since it did drop the caches, other parts of the system will then have to do more work to reconstruct the caches (looking up an dentry explicitly, or loading a page from disk instead of RAM). Everything I'm describing is about a busy server with heterogenous workloads of specific types.
        wongarsu 5y ago
        I think what joosters is trying to say that instead of overcommitting a system could just allocate memory directly in swap. Both creating the page in swap and swapping it in on first write can be mostly optimized away apart from the bookkeeping, but you guarantee that you always have enough RAM+Swap to provide what you have committed to.
        josefx 5y ago
        > if overcommit is disabled, then the system drops its internal caches (dentry cache, pagecache) to satisfy a memory allocation. Couldn't the system just reserve the pages for future use by the application but still use them for caching until the application actually tries to use them?
        yc12340 5y ago
        This sounds like it would cause LRU caches to lose their LRU properties...
        josefx 5y ago
        It doesn't need to reserve a specific page during allocation, just ensure that the number of reserved pages is smaller than the total amount of pages. I would expect at least read only caches to mostly behave like they currently do on a system with over commit enabled.

dekhn 5y ago

no, the kernel maintains other allocated objects (dentry, inode caches) that it can't swap out. Under memory pressure, those get dropped before application pages. See https://unix.stackexchange.com/questions/17936/setting-proc-sys-vm-drop-caches-to-clear-cache https://unix.stackexchange.com/questions/17936/setting-proc-... and https://unix.stackexchange.com/questions/111893/how-long-do-dentries-stay-in-the-dcache https://unix.stackexchange.com/questions/111893/how-long-do-... I've found the linux memory system to be far too complicated to understand for quite some time, compared to what's documented in, for example, The Design and Implementation of The FreeBSD Operating System, for a more comprehensible system.

joosters 5y ago
Those objects must exist with and without overcommit, I don’t understand why they must make one less efficient than the other. (I’m talking in general here - not Linux specifically)
- dekhn 5y ago
  dentry caches exist with and without overcommit, but you get higher cache hit rates with overcommit, because you flush them less recently. Depending on workload, this can matter a lot. It mattered more in the time of hard drives.
  - joosters 5y ago
    I’m sorry, but I’m still not following… why must a cache be (in theory) flushed more often without overcommit?
    - dekhn 5y ago
      if overcommit is disabled, then the system drops its internal caches (dentry cache, pagecache) to satisfy a memory allocation. Since most applications don't touch pages they allocate, that means the OS could have avoided dropping the caches. Since it did drop the caches, other parts of the system will then have to do more work to reconstruct the caches (looking up an dentry explicitly, or loading a page from disk instead of RAM). Everything I'm describing is about a busy server with heterogenous workloads of specific types.
      - wongarsu 5y ago
        I think what joosters is trying to say that instead of overcommitting a system could just allocate memory directly in swap. Both creating the page in swap and swapping it in on first write can be mostly optimized away apart from the bookkeeping, but you guarantee that you always have enough RAM+Swap to provide what you have committed to.
      - josefx 5y ago
        > if overcommit is disabled, then the system drops its internal caches (dentry cache, pagecache) to satisfy a memory allocation. Couldn't the system just reserve the pages for future use by the application but still use them for caching until the application actually tries to use them?
        yc12340 5y ago
        This sounds like it would cause LRU caches to lose their LRU properties...
        josefx 5y ago
        It doesn't need to reserve a specific page during allocation, just ensure that the number of reserved pages is smaller than the total amount of pages. I would expect at least read only caches to mostly behave like they currently do on a system with over commit enabled.