4 ms·

There's a reason nobody does this: RAM is expensive. Disabling overcommit on your typical server workload will waste a great deal of it. TFA completely ignores

• wmf 10mo ago
  If you have enough swap there's no waste.
  • jcalvinowens 10mo ago
    Wasted swap is still waste, and the swapping costs cycles.
    • wmf 10mo ago
      With overcommit off the swap isn't used; it's only necessary for accounting purposes. I agree that it's a waste of disk space.
• toast0 10mo ago
  Even if you disable overcommit, I don't think you will get pages assigned when you allocate. If your allocations don't trigger an allocation failure, you should get the same behavior with respect to disk cache using otherwise unused pages. The difference is that you'll fail allocations, where there's a reasonable interface for errors, rather than failing at demand paging when writing to previously unused pages where there's not a good interface. Of course, there are many software patterns where excessive allocations are made without any intent of touching most of the pages; that's fine with overcommit, but it will lead to allocation failures when you disable overcommit. Disabling overcommit does make fork in a large process tricky; I don't think the rant about redis in the article is totally on target; fork to persist is a pretty good solution, copy on write is a reasonable cost to pay while dumping the data to disk and then it returns to normal when the dump is done. But without overcommit, it doubles the memory commitment while the dump is running, and that's likely to cause issues if redis is large relative to memory and that's worth checking for and warning about. The linked jemalloc issue seems like it could be problematic too, but I only skimmed; seems like that's worth warning about as well. For the fork path, it might be nice if you could request overcommit in certain circumstances... fork but only commit X% rather than the whole memory space.
  • jcalvinowens 10mo ago
    You're correct it doesn't prefault the mappings, but that's irrelevant: it accounts them as allocated, and a later allocation which goes over the limit will immediately fail. Remember, the limit is artificial and defined by the user with overcommit=2, by overcommit_ratio and user_reserve_kbytes. Using overcommit=2 necessarily wastes RAM (renders a larger portion of it unusable).
    • loeg 10mo ago
      If the overcommit ratio is 1, there is no portion rendered unusable? This seems to contradict your "necessarily" wastes RAM claim?
      • jcalvinowens 10mo ago
        Read the comment again, that wasn't the only one I mentioned.
        • loeg 10mo ago
          Please point out what you're talking about, because the comment is short and I read it fully multiple times now.
    • toast0 10mo ago
      > Using overcommit=2 necessarily wastes RAM (renders a larger portion of it unusable). The RAM is not unusable, it will be used. Some portion of ram may be unallocatable, but that doesn't mean it's wasted. There's a tradeoff. With overcommit disabled, you will get allocation failure rather than OOM killer. But you'll likely get allocation failures at memory pressure below that needed to trigger the OOM killer. And if you're running a wide variety of software, you'll run into problems because overcommit is the mainstream default for Linux, so many things are only widely tested with it enabled.
      • jcalvinowens 10mo ago
• loeg 10mo ago
  How does disabling overcommit waste RAM?
  • jcalvinowens 10mo ago
    Because userspace rarely actually faults in all the pages it allocates.
    • loeg 10mo ago
      Surely the source of the waste here is the userspace program not using the memory it allocated, rather than whether or not the kernel overcommits memory. Attributing this to overcommit behavior is invalid.
      • jcalvinowens 10mo ago
        Obviously. But all programs do that and have done it forever, it's literally the very reason overcommit exists.
        • userbinator 10mo ago
          Only the poorly-written ones, which are unfortunately the majority of them.
      • nickelpro 10mo ago
        Reading COW memory doesn't cause a fault. It doesn't mean unused literally. And even if it's not COW, there's nothing wrong or inefficient about opportunistically allocating pages ahead of time to avoid syscall latency. Or mmapping files and deciding halfway through you don't need the whole thing. There are plenty of reasons overcommit is the default.
      • PunchyHamster 10mo ago
        The waste comes with asterisk That "waste" (that overcommit turns into "not a waste") means you can do far less allocations, with overcommit you can just allocate a bunch of memory and use it gradually, instead of having to do malloc() every time you need a bit of memory and free() every time you get rid of it. You'd also increase memory fragmentation that way possibly hitting performance. It's also pretty much required for GCed languages to work sensibly