3 ms·

It's surprising that we have stuck with 4 kB pages on x86 since the 386, even though computers have ~10 000x as much memory now (4 MB -> 32GB).

b9a2cab5 5y ago
Indeed, the M1/A14 on mobile has larger pages which lets them have more effective TLB coverage with a smaller cache. In some applications this can boost performance by double digit % (which you can simulate by enabling large pages on x86).
TD-Linux 5y ago
Linux still has a lot of assumptions baked into the page size. Power9 and some aarch64 systems have 16kB pages, but occasionally you run into some corner cases - for example, you can't mount a btrfs partition created on a x86 machine on a power9 one because the btrfs page size must be >= the mmu page size.
- classichasclass 5y ago
  64K pages, actually. Also, POWER9 and aarch64 are perfectly capable of running with 4K pages, but not everything does. My desktop POWER9 with Fedora is a 64K page system, but Void PPC runs the same hardware with a 4K page.
- diegocg 5y ago
  That's the fault of btrfs (ext4 or xfs handle these cases just fine), and even btrfs is getting support for it in the latest releases
  - josteink 5y ago
    While you may be technically correct, I guess from a practical point of view, OP is correct in that page-size still matters if you trying to run Linux on "unconventional" platforms.
delsarto 5y ago
I wrote a thesis on this in 2008 (Transparent large-page support for Itanium Linux https://ts.data61.csiro.au/publications/theses_public/08/Wienand:me.pdf https://ts.data61.csiro.au/publications/theses_public/08/Wie...) and Matthew Wilcox was already involved in the area then. I admire his persistence, and have certainly have not kept up in the state of the art. Itanium had probably the greatest ability to select page size, probably more than any other architecture (?). On x86-64 you really only have 2mb or 4k to work with in a general purpose situation. It was difficult to show the benefits of managing all the different page sizes and, as this notes, re/writing everything to be aware of page sizes effectively. Those who had really big workloads that benefited from huge pinned mappings didn't really care that much either. It made the work hard to find traction at the time.
- brandmeyer 5y ago
  There is a series of page sizes that steps in increments of 5 bits per width. 8k/256k/8M therefore have regular join points in the address space, the first of which is at 8G. Does superpage management get easier when each superpage is composed of only 32x of the next size down? When I first stumbled on this idea, it seemed like it would have many more opportunities for forming intermediate-sized superpages.
toast0 5y ago
You can also do large pages, 2MB or 1GB or whatever the obscenely large page size is for 5-level paging on latest systems. 2MB vs 4kB isn't quite the same ratio as 4MB -> 32GB, but it's still a lot less pages to cache in the TLB, and it's not too big to manage when you need to copy on write or swap out (or compress with zram) and whatever else needs to be done at the page level.
rwmj 5y ago
RHEL used 64k page size on aarch64 for a while. I believe we have switched back to 4k. It caused some problems, from memory: * Blow-ups in various kernel data structures. There was some virtio code which was allocating N pages per driver queue. * Problems with GPUs, either the driver or the firmware assumed 4k pages. (Edit: This actually affected Power, not ARM, but the issue is caused by page size: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/3ZBQBUN67RG6U5QHOTBHEXAKR77S3NIC/ https://lists.fedoraproject.org/archives/list/devel@lists.fe...) * Filesystems make assumptions about page size versus block size. * Processes generally take more RAM, with RAM wasted because of internal fragmentation.
- dmw_ng 5y ago
  I bet this would also screw masses of user code making assumptions about optimal size, or simply designed/optimized for a smaller size. LMDB was the first thing that came to mind
- cesarb 5y ago
  > RHEL used 64k page size on aarch64 for a while. I believe we have switched back to 4k. Last I checked, at least CentOS 8 (which should be the same as RHEL 8) is still using 64k pages (search at https://git.centos.org/rpms/kernel/blob/c8/f/SOURCES/kernel-aarch64.config https://git.centos.org/rpms/kernel/blob/c8/f/SOURCES/kernel-... for CONFIG_ARM64_64K_PAGES=y). And AFAIK, to access the maximum amount of physical memory in AARCH64 (52-bit physical addresses, instead of 48 bit physical addresses), you must use 64k pages. Since RHEL is normally used on servers, it makes sense to want to be able to access huge amounts of physical memory; that's probably the true reason (or even the sole reason) RHEL uses 64k pages on AARCH64.
  - Hello71 5y ago
    AMD64 supports only 48-bit addresses, and is still the most popular server ISA. You have: 2^48 byte You want: tebibyte 2^48 byte = 256 tebibyte 2^48 byte = (1 / 0.00390625) tebibyte how many servers do you have with more than 256 TiB of RAM?
DudeInBasement 5y ago
A lot of peripherals have 4kB address space. So it becomes complicated if you change to 16kB as you'll be mapping in more than you bargained for.

b9a2cab5 5y ago

Indeed, the M1/A14 on mobile has larger pages which lets them have more effective TLB coverage with a smaller cache. In some applications this can boost performance by double digit % (which you can simulate by enabling large pages on x86).

TD-Linux 5y ago

Linux still has a lot of assumptions baked into the page size. Power9 and some aarch64 systems have 16kB pages, but occasionally you run into some corner cases - for example, you can't mount a btrfs partition created on a x86 machine on a power9 one because the btrfs page size must be >= the mmu page size.

classichasclass 5y ago
64K pages, actually. Also, POWER9 and aarch64 are perfectly capable of running with 4K pages, but not everything does. My desktop POWER9 with Fedora is a 64K page system, but Void PPC runs the same hardware with a 4K page.
diegocg 5y ago
That's the fault of btrfs (ext4 or xfs handle these cases just fine), and even btrfs is getting support for it in the latest releases
- josteink 5y ago
  While you may be technically correct, I guess from a practical point of view, OP is correct in that page-size still matters if you trying to run Linux on "unconventional" platforms.

delsarto 5y ago

I wrote a thesis on this in 2008 (Transparent large-page support for Itanium Linux https://ts.data61.csiro.au/publications/theses_public/08/Wienand:me.pdf https://ts.data61.csiro.au/publications/theses_public/08/Wie...) and Matthew Wilcox was already involved in the area then. I admire his persistence, and have certainly have not kept up in the state of the art. Itanium had probably the greatest ability to select page size, probably more than any other architecture (?). On x86-64 you really only have 2mb or 4k to work with in a general purpose situation. It was difficult to show the benefits of managing all the different page sizes and, as this notes, re/writing everything to be aware of page sizes effectively. Those who had really big workloads that benefited from huge pinned mappings didn't really care that much either. It made the work hard to find traction at the time.

brandmeyer 5y ago
There is a series of page sizes that steps in increments of 5 bits per width. 8k/256k/8M therefore have regular join points in the address space, the first of which is at 8G. Does superpage management get easier when each superpage is composed of only 32x of the next size down? When I first stumbled on this idea, it seemed like it would have many more opportunities for forming intermediate-sized superpages.

toast0 5y ago

You can also do large pages, 2MB or 1GB or whatever the obscenely large page size is for 5-level paging on latest systems. 2MB vs 4kB isn't quite the same ratio as 4MB -> 32GB, but it's still a lot less pages to cache in the TLB, and it's not too big to manage when you need to copy on write or swap out (or compress with zram) and whatever else needs to be done at the page level.

rwmj 5y ago

RHEL used 64k page size on aarch64 for a while. I believe we have switched back to 4k. It caused some problems, from memory: * Blow-ups in various kernel data structures. There was some virtio code which was allocating N pages per driver queue. * Problems with GPUs, either the driver or the firmware assumed 4k pages. (Edit: This actually affected Power, not ARM, but the issue is caused by page size: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/3ZBQBUN67RG6U5QHOTBHEXAKR77S3NIC/ https://lists.fedoraproject.org/archives/list/devel@lists.fe...) * Filesystems make assumptions about page size versus block size. * Processes generally take more RAM, with RAM wasted because of internal fragmentation.

dmw_ng 5y ago
I bet this would also screw masses of user code making assumptions about optimal size, or simply designed/optimized for a smaller size. LMDB was the first thing that came to mind
cesarb 5y ago
> RHEL used 64k page size on aarch64 for a while. I believe we have switched back to 4k. Last I checked, at least CentOS 8 (which should be the same as RHEL 8) is still using 64k pages (search at https://git.centos.org/rpms/kernel/blob/c8/f/SOURCES/kernel-aarch64.config https://git.centos.org/rpms/kernel/blob/c8/f/SOURCES/kernel-... for CONFIG_ARM64_64K_PAGES=y). And AFAIK, to access the maximum amount of physical memory in AARCH64 (52-bit physical addresses, instead of 48 bit physical addresses), you must use 64k pages. Since RHEL is normally used on servers, it makes sense to want to be able to access huge amounts of physical memory; that's probably the true reason (or even the sole reason) RHEL uses 64k pages on AARCH64.
- Hello71 5y ago
  AMD64 supports only 48-bit addresses, and is still the most popular server ISA. You have: 2^48 byte You want: tebibyte 2^48 byte = 256 tebibyte 2^48 byte = (1 / 0.00390625) tebibyte how many servers do you have with more than 256 TiB of RAM?

DudeInBasement 5y ago

A lot of peripherals have 4kB address space. So it becomes complicated if you change to 16kB as you'll be mapping in more than you bargained for.