3 ms·

This is usually below the level of abstraction I am working on. I have questions. Before madvise, did people simply assume that a memory page is always 4kiB in

This is usually below the level of abstraction I am working on. I have questions. Before madvise, did people simply assume that a memory page is always 4kiB in size and built that assumption into so many programs? Is that why many programs break? Why did they assume that? Was there at least something like "size of int" or so around before madvise? And if so, why did they not use that?

karatinversion 4y ago
The article has some examples of breakage, and these are mostly using an allocator other than the system allocator which hard codes a 4k page size (Go, Chrome, jemalloc) at least for x86 and arm. Why did they assume that? 4k pages are a feature of the memory management unit of the cpu. Optional support for large pages came to x86 with pentium in the mid-1990’s. Presumably all x86 cpus out there today have large page support, but the assumption of the 4k default is deeply ingrained.
- kris_wayton 4y ago
  The Redis example is a little different: "fork() e.g. Redis: Calling fork marks all of the process's pages as copy-on-write. Then when a single byte on a page is modified, the page must be copied. Redis uses fork to create a read-only "snapshot" of memory, when writing a checkpoint to disk." So prior, that fork forced only a rewrite of some collection of 4kb pages. Afterwards, obviously much larger rewrites.
  - ignoramous 4y ago
    Interestingly a common pattern: fork (and no exec) is also what Android does [0] to not only speed up Kotlin / Java app start-up but save on allocation of shared dex/bytecode (a couple dozen megabytes); code: https://archive.is/GMka3 https://archive.is/GMka3 [0] The state of ASLR on Android 5 (2015), https://archive.is/ADx65 https://archive.is/ADx65 (copperhead.co).
- jasone 4y ago
  jemalloc does not assume a particular page size; it configures page size at build time. By default the configured page size is the actual page size (usually 4KiB), but optionally it's some power-of-two multiple of the actual page size. In the early days of jemalloc the page size was queried at startup rather than burned into the binary, but that dynamic flexibility was never of any use in practice, and it inhibited compiler optimizations.
feffe 4y ago
I think most applications don't have any dependency towards a specific page size. They use malloc (C) or new (C++) to allocate memory which does not expose this constraint. You need to care if using mmap directly to map files or other resources into the virtual memory address space. The default page size can be queried using for example sysconf() on Linux. I guess something like garbage collectors in language run-times would also use mmap directly as it's most likely to side step malloc/new. An application would normally not use madvise, unless also using mmap for some special purpose. It depends on the CPU architecture how flexible it is with different page sizes. For example, from what I recall, MIPS was extremely flexible and allowed any even power of two size for any TLB entry. x86_64 only support three different page sizes, 4 kB, 2 MB and 1 GB and there are limitations wrt the number of TLB entries that can be used for the larger page sizes. So, yea, there are bound to be regressions if just trying to switch to 2 MB as a default but I think it should be doable. Not all archs use 4 kB to begin with.
zajio1am 4y ago
You can call sysconf(_SC_PAGESIZE) to get page size. Problem with transparent huge pages is that you get regular page size (4k) from sysconf(), but then OS sometimes use 2M instead. Just switching to larger regular page size (e.g. 64k) on platforms that support it would not have problems associated with THP.
- kzrdude 4y ago
  Is that a performance problem or a correctness problem?
  - zajio1am 4y ago
    We had issue where memory consumption was several times higher with THP enabled. When you hit OOM, such performance problem became correctness problem.

karatinversion 4y ago

The article has some examples of breakage, and these are mostly using an allocator other than the system allocator which hard codes a 4k page size (Go, Chrome, jemalloc) at least for x86 and arm. Why did they assume that? 4k pages are a feature of the memory management unit of the cpu. Optional support for large pages came to x86 with pentium in the mid-1990’s. Presumably all x86 cpus out there today have large page support, but the assumption of the 4k default is deeply ingrained.

kris_wayton 4y ago
The Redis example is a little different: "fork() e.g. Redis: Calling fork marks all of the process's pages as copy-on-write. Then when a single byte on a page is modified, the page must be copied. Redis uses fork to create a read-only "snapshot" of memory, when writing a checkpoint to disk." So prior, that fork forced only a rewrite of some collection of 4kb pages. Afterwards, obviously much larger rewrites.
- ignoramous 4y ago
  Interestingly a common pattern: fork (and no exec) is also what Android does [0] to not only speed up Kotlin / Java app start-up but save on allocation of shared dex/bytecode (a couple dozen megabytes); code: https://archive.is/GMka3 https://archive.is/GMka3 [0] The state of ASLR on Android 5 (2015), https://archive.is/ADx65 https://archive.is/ADx65 (copperhead.co).
jasone 4y ago
jemalloc does not assume a particular page size; it configures page size at build time. By default the configured page size is the actual page size (usually 4KiB), but optionally it's some power-of-two multiple of the actual page size. In the early days of jemalloc the page size was queried at startup rather than burned into the binary, but that dynamic flexibility was never of any use in practice, and it inhibited compiler optimizations.

feffe 4y ago

I think most applications don't have any dependency towards a specific page size. They use malloc (C) or new (C++) to allocate memory which does not expose this constraint. You need to care if using mmap directly to map files or other resources into the virtual memory address space. The default page size can be queried using for example sysconf() on Linux. I guess something like garbage collectors in language run-times would also use mmap directly as it's most likely to side step malloc/new. An application would normally not use madvise, unless also using mmap for some special purpose. It depends on the CPU architecture how flexible it is with different page sizes. For example, from what I recall, MIPS was extremely flexible and allowed any even power of two size for any TLB entry. x86_64 only support three different page sizes, 4 kB, 2 MB and 1 GB and there are limitations wrt the number of TLB entries that can be used for the larger page sizes. So, yea, there are bound to be regressions if just trying to switch to 2 MB as a default but I think it should be doable. Not all archs use 4 kB to begin with.

zajio1am 4y ago

You can call sysconf(_SC_PAGESIZE) to get page size. Problem with transparent huge pages is that you get regular page size (4k) from sysconf(), but then OS sometimes use 2M instead. Just switching to larger regular page size (e.g. 64k) on platforms that support it would not have problems associated with THP.

kzrdude 4y ago
Is that a performance problem or a correctness problem?
- zajio1am 4y ago
  We had issue where memory consumption was several times higher with THP enabled. When you hit OOM, such performance problem became correctness problem.