3 ms·

Isn't `madvise` useful if you want to use Huge Pages?

scottlamb 4y ago
If you're hoping file-backed mmaps from any filesystem other than tmpfs/shmem will use huge pages, you will be disappointed. :-( See the following note at https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html https://www.kernel.org/doc/html/latest/admin-guide/mm/transh... > Currently THP only works for anonymous memory mappings and tmpfs/shmem.
- the8472 4y ago
  Not sure if that's true any longer on recent kernels. There have been some changes with folios that I think enable huge pages in the page cache. $ cat /proc/<firefox process>/smaps [...] 7efca62e3000-7efcaa13d000 r-xp 00ae3000 00:18 75354786 /usr/lib/libLLVM-15.so Size: 63848 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 58420 kB Pss: 19213 kB Pss_Dirty: 0 kB Shared_Clean: 56372 kB Shared_Dirty: 0 kB Private_Clean: 2048 kB Private_Dirty: 0 kB Referenced: 58420 kB Anonymous: 0 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB FilePmdMapped: 57344 kB <==== huge pages Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 1 VmFlags: rd ex mr mw me sd
  - scottlamb 4y ago
    Exciting if true! I see https://docs.kernel.org/filesystems/proc.html https://docs.kernel.org/filesystems/proc.html describes FilePmdMapped as "Page cache mapped into userspace with huge pages", consistent with what you are saying. I don't fully understand the distinction between that and FileHugePages: "Memory used for filesystem data (page cache) allocated with huge pages". I wouldn't think it'd be possible to map it into userspace as huge pages if the kernel hasn't allocated it as contiguous physical memory (and consistently aligned with the userspace virtual addresses), so there's something I'm missing. What kernel version did that output come from? Do you happen to know if Firefox did anything special to set that up? What filesystem type is this?
    - scottlamb 4y ago
      Huh. I did a little digging through kernel source. There's been a CONFIG_READ_ONLY_THP_FOR_FS since 2019. It's still marked as experimental and isn't enabled on the precompiled kernel I'm using (with Ubuntu 22.10). Is that option set on your kernel, or is this something else? I also see something about MADV_COLLAPSE that supposedly supports file-backed pages. [1] [1] https://lwn.net/Articles/913363/ https://lwn.net/Articles/913363/
      - the8472 4y ago
        Yeah, it's enabled here $ zgrep 'CONFIG_READ_ONLY_THP_FOR_FS' /proc/config.gz CONFIG_READ_ONLY_THP_FOR_FS=y > What kernel version did that output come from? kernel 6.2.8-arch1-1 > What filesystem type is this? btrfs > I don't fully understand the distinction between that and FileHugePages: "Memory used for filesystem data (page cache) allocated with huge pages". Probably pages in the page cache that aren't mapped into a user process. Which is what happens when you read()
- leni536 4y ago
  You don't actually need transparent huge pages, but maybe you can work with explicit huge pages.
  - scottlamb 4y ago
    Is there a working way to request explicit huge pages for a file-backed mmap on ext4/xfs/btrfs? I'm not aware of it.
    - leni536 4y ago
      I have no idea. I was pointing out that the transparent huge pages documentation is not authorative for requesting huge pages explicitly with mmap. I didn't find any documentation that would indicate that explicit huge pages didn't work with on-disk filesystems, but sure enough, it doesn't seem to work on ext4.

If you're hoping file-backed mmaps from any filesystem other than tmpfs/shmem will use huge pages, you will be disappointed. :-( See the following note at https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html https://www.kernel.org/doc/html/latest/admin-guide/mm/transh... > Currently THP only works for anonymous memory mappings and tmpfs/shmem.

the8472 4y ago
Not sure if that's true any longer on recent kernels. There have been some changes with folios that I think enable huge pages in the page cache. $ cat /proc/<firefox process>/smaps [...] 7efca62e3000-7efcaa13d000 r-xp 00ae3000 00:18 75354786 /usr/lib/libLLVM-15.so Size: 63848 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 58420 kB Pss: 19213 kB Pss_Dirty: 0 kB Shared_Clean: 56372 kB Shared_Dirty: 0 kB Private_Clean: 2048 kB Private_Dirty: 0 kB Referenced: 58420 kB Anonymous: 0 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB FilePmdMapped: 57344 kB <==== huge pages Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 1 VmFlags: rd ex mr mw me sd
- scottlamb 4y ago
  Exciting if true! I see https://docs.kernel.org/filesystems/proc.html https://docs.kernel.org/filesystems/proc.html describes FilePmdMapped as "Page cache mapped into userspace with huge pages", consistent with what you are saying. I don't fully understand the distinction between that and FileHugePages: "Memory used for filesystem data (page cache) allocated with huge pages". I wouldn't think it'd be possible to map it into userspace as huge pages if the kernel hasn't allocated it as contiguous physical memory (and consistently aligned with the userspace virtual addresses), so there's something I'm missing. What kernel version did that output come from? Do you happen to know if Firefox did anything special to set that up? What filesystem type is this?
  - scottlamb 4y ago
    Huh. I did a little digging through kernel source. There's been a CONFIG_READ_ONLY_THP_FOR_FS since 2019. It's still marked as experimental and isn't enabled on the precompiled kernel I'm using (with Ubuntu 22.10). Is that option set on your kernel, or is this something else? I also see something about MADV_COLLAPSE that supposedly supports file-backed pages. [1] [1] https://lwn.net/Articles/913363/ https://lwn.net/Articles/913363/
    - the8472 4y ago
      Yeah, it's enabled here $ zgrep 'CONFIG_READ_ONLY_THP_FOR_FS' /proc/config.gz CONFIG_READ_ONLY_THP_FOR_FS=y > What kernel version did that output come from? kernel 6.2.8-arch1-1 > What filesystem type is this? btrfs > I don't fully understand the distinction between that and FileHugePages: "Memory used for filesystem data (page cache) allocated with huge pages". Probably pages in the page cache that aren't mapped into a user process. Which is what happens when you read()
leni536 4y ago
You don't actually need transparent huge pages, but maybe you can work with explicit huge pages.
- scottlamb 4y ago
  Is there a working way to request explicit huge pages for a file-backed mmap on ext4/xfs/btrfs? I'm not aware of it.
  - leni536 4y ago
    I have no idea. I was pointing out that the transparent huge pages documentation is not authorative for requesting huge pages explicitly with mmap. I didn't find any documentation that would indicate that explicit huge pages didn't work with on-disk filesystems, but sure enough, it doesn't seem to work on ext4.