3 ms·

Correct me if I'm wrong Linux only supports KSM (memory-deduping) between processes when doing it between VMs, as QEMU provides information to the kernel to per

Correct me if I'm wrong Linux only supports KSM (memory-deduping) between processes when doing it between VMs, as QEMU provides information to the kernel to perform it.

yjftsjthsd-h 2y ago
https://www.kernel.org/doc/html/latest/admin-guide/mm/ksm.html https://www.kernel.org/doc/html/latest/admin-guide/mm/ksm.ht... > KSM was originally developed for use with KVM (where it was known as Kernel Shared Memory), to fit more virtual machines into physical memory, by sharing the data common between them. But it can be useful to any application which generates many instances of the same data Although... > KSM only operates on those areas of address space which an application has advised to be likely candidates for merging, by using the madvise(2) system call
- pradn 2y ago
  I wonder if you could just madvise the entire address space. My hunch is that it's for performance reasons only - fewer pages to scan, hash, and de-duplicate.
  - MrDrMcCoy 2y ago
    You can. There are custom kernel builds that do this, as well as shim loaders that do that madvise call. However, those are considerably less practical than the recent systemd feature that allows you to do it per unit. If applied to the system scope, it effectively covers the whole system. However, there are caveats: 1. You need a much better newer systemd than your distro likely packages. 2. KSM dedupe scans aren't free. Your system can spend its time doing that scan or it can spend its doing work with duplicate pages. Only relatively idle or highly homogenous systems would be free of the penalty. 3. For applications, especially statically linked ones, the duplicated code is not super likely to fall along page boundaries, thus the actual detectable duplication will be relatively low. That said, it's still great for densely deploying a high traffic microservice that's more memory than CPU bound.

Correct me if I'm wrong Linux only supports KSM (memory-deduping) between processes when doing it between VMs, as QEMU provides information to the kernel to perform it.

yjftsjthsd-h 2y ago

https://www.kernel.org/doc/html/latest/admin-guide/mm/ksm.html https://www.kernel.org/doc/html/latest/admin-guide/mm/ksm.ht... > KSM was originally developed for use with KVM (where it was known as Kernel Shared Memory), to fit more virtual machines into physical memory, by sharing the data common between them. But it can be useful to any application which generates many instances of the same data Although... > KSM only operates on those areas of address space which an application has advised to be likely candidates for merging, by using the madvise(2) system call

pradn 2y ago
I wonder if you could just madvise the entire address space. My hunch is that it's for performance reasons only - fewer pages to scan, hash, and de-duplicate.
- MrDrMcCoy 2y ago
  You can. There are custom kernel builds that do this, as well as shim loaders that do that madvise call. However, those are considerably less practical than the recent systemd feature that allows you to do it per unit. If applied to the system scope, it effectively covers the whole system. However, there are caveats: 1. You need a much better newer systemd than your distro likely packages. 2. KSM dedupe scans aren't free. Your system can spend its time doing that scan or it can spend its doing work with duplicate pages. Only relatively idle or highly homogenous systems would be free of the penalty. 3. For applications, especially statically linked ones, the duplicated code is not super likely to fall along page boundaries, thus the actual detectable duplication will be relatively low. That said, it's still great for densely deploying a high traffic microservice that's more memory than CPU bound.