4 ms·

Another example would be the JVM’s new low-latency GC, ZGC, that does some smart addressing tricks to store some information in pointer addresses themselves, ef

by kaba0 2y ago

Another example would be the JVM’s new low-latency GC, ZGC, that does some smart addressing tricks to store some information in pointer addresses themselves, effectively tripling the actual memory usage when naively looked at through virt memory.

layer8 2y ago
Such applications could handle this explicitly, e.g. with mmap and mprotect. In principle there is no reason for such use-cases to rely on implicit overcommit behavior. Which also isn't present on Windows.
- kaba0 2y ago
  It might actually be using that, I’m not intimately familiar with the topic.
  - layer8 2y ago
    Yes, I would expect that as well. Implicit overcommit is really only necessary for fork and general backwards compatibility.
- jart 2y ago
  That's because Windows forces you to use special userspace libraries that account for memory and let you create "reservations" for large regions, without actually populating entries in the MMU (i.e. PML4t). Linux doesn't have that. You create reservations for large memory regions by allocating virtual memory, which of course doesn't become real (resident) until you use it. That's overcommit. But since there's no userspace libraries like WIN32 forced upon you on Linux, there's really no way for the kernel to know your intent when it comes to memory, because the abstractions for memory on Linux are so simple. Windows has a ridiculous byzantine system, which some people think is better. But people are really missing the point when they try to make an apples-to-apples comparison between the two systems of memory management. It's not a question of (a) overcommit good, vs. (b) overcommit bad. Anyone who thinks either is a fool. Linux is simply a system that requires a smart system administrator who understands your use cases and can tune sysctl accordingly when you want to push beyond the normal boundaries.
  - layer8 2y ago
    It seems to me that the prot parameter and/or MAP_NORESERVE would indicate the intent.
ackfoobar 2y ago
BTW the newer Generational ZGC is not using multi-mapped memory any more.

by kaba0 2y ago

layer8 2y ago

Such applications could handle this explicitly, e.g. with mmap and mprotect. In principle there is no reason for such use-cases to rely on implicit overcommit behavior. Which also isn't present on Windows.

kaba0 2y ago
It might actually be using that, I’m not intimately familiar with the topic.
- layer8 2y ago
  Yes, I would expect that as well. Implicit overcommit is really only necessary for fork and general backwards compatibility.
jart 2y ago
That's because Windows forces you to use special userspace libraries that account for memory and let you create "reservations" for large regions, without actually populating entries in the MMU (i.e. PML4t). Linux doesn't have that. You create reservations for large memory regions by allocating virtual memory, which of course doesn't become real (resident) until you use it. That's overcommit. But since there's no userspace libraries like WIN32 forced upon you on Linux, there's really no way for the kernel to know your intent when it comes to memory, because the abstractions for memory on Linux are so simple. Windows has a ridiculous byzantine system, which some people think is better. But people are really missing the point when they try to make an apples-to-apples comparison between the two systems of memory management. It's not a question of (a) overcommit good, vs. (b) overcommit bad. Anyone who thinks either is a fool. Linux is simply a system that requires a smart system administrator who understands your use cases and can tune sysctl accordingly when you want to push beyond the normal boundaries.
- layer8 2y ago
  It seems to me that the prot parameter and/or MAP_NORESERVE would indicate the intent.

ackfoobar 2y ago

BTW the newer Generational ZGC is not using multi-mapped memory any more.