5 ms·

Prepare your apps for Google Play's 16 KB page size compatibility requirement

88 points by ingve 1y ago

londons_explore 1y ago
Letting userspace know the page size was IMO a design mistake. Imagine a world where the page size is secret to userspace. Anything that needs page size alignment will be chosen by the kernel. That in turn allows mixed page size, variable page size, heirarchical pages, etc.
- Dylan16807 1y ago
  It's hard to hide all the details, especially when you involve varied permissions over different chunks of memory.
- duskwuff 1y ago
  It's unavoidable. APIs like mprotect() operate on pages; there's no way to hide that from them.
  - viraptor 1y ago
    Also various performance characteristics graphs will have steps around the multiples of page sizes. People would find out anyway. (Although maybe the runtime detection is not a bad idea)
    - londons_explore 1y ago
      Just like cache size. But when someone releases a new CPU with a larger or smaller cache, all old software continues to work. Secret page size would offer the same benefit.
      - dataflow 1y ago
        Cache line size would be a better example.
        Tuna-Fish 1y ago
        Yup. And like with page sizes, the big problems with cache line sizes is not when people design things against a specific line size, but the (much more numerous) cases where they design something that only works well for a fixed size but they don't even know that, because they literally never thought about it, and it worked fine because every machine they used was similar. It doesn't matter what apis you were provided or what analysis you did, you don't know your software works with different hardware until you test it on it.
  - londons_explore 1y ago
    you can simulate arbitrary sized mprotect() by having the kernel do the closest it can using the hardware, and then any pages crossing the boundary will be handled by page faults. The performance hit should be small as long as most mprotect regions are large (which they typically are).
    - dzaima 1y ago
      That'd fail horrifically for guard pages around a main region where the whole used region is reasonably expected to be used. Considering that it'd be literally completely entirely unacceptable to ever hit such an emulated range in code of any significance as it'd be a slowdown in the thousands of times, it'd make mprotect entirely pointless and effectively broken. Unless of course you add "padding" of expected page size (not applicable for guard pages though, those are just SOL), which is basically status quo except that, instead of apps doing hard-coding crashing on unexpected page size, they just become potentially 1000x slower, but of course in practice there's no difference between a crash and 1000x slower. I'd even say a crash is better - means a bug report of it can't be dismissed as just a perf issue, and it's clear that something's going wrong in the first place. The size of the mprotect region does not matter. What matters is worst-case behavior, and, regardless of the size, if a single critical byte of memory ends up in the emulated region, your program is dead to the user, causing data loss or whatever consequences from being, practically speaking, completely frozen your program has.
    - dwattttt 1y ago
      Emulating it would work quite terribly. With a 4kb page size you can write 4kb of memory as RW, then an adjacent 4kb of memory as RX, and have them interact. Trying to emulate that under 16kb pages would fault every memory operation.
  - londons_explore 1y ago
    the mprotect API could have been designed more like malloc() - ie. you don't protect a preexisting memory range - instead the API returns a new memory range with the protections you've asked for, possibly copying a load of data into it for you incase you asked for a readonly range. And that 'copy' might be zero-overhead remapping of the original pages.
    - duskwuff 1y ago
      There's some important use cases for mprotect(), like toggling W^X on JIT pages, which that wouldn't work for.
      - Jyaif 1y ago
        W^X would be supported by the API that they suggest: "possibly copying a load of data into it for you incase you asked for a readonly range."
        duskwuff 1y ago
        For JIT, you specifically need to be able to write some code to a page, then switch that page from writable to executable in place. Having the page move to a new location isn't acceptable, as the generated code will contain relative calls to other JIT generated code in other pages you've previously written to.
        Jyaif 1y ago
        Oh right, good point!
      - PhilipRoman 1y ago
        You can probably replace some of those use cases by mapping the same physical page in two places with different permissions (W^X is per-address, not per physical page).
      - IshKebab 1y ago
        If you had a way of allocating pages of at least N bytes it would work. uint8_t* mem = malloc_pages(1024); mprotect_paged(mem, 1024);
  - pjmlp 1y ago
    That isn't officially supported API, so if any app on NDK is making use of it, though luck. https://developer.android.com/ndk/guides/stable_apis https://developer.android.com/ndk/guides/stable_apis
    - duskwuff 1y ago
      I'm not sure how you get that idea - as a Linux system call, mprotect() is part of "the standard C11 library headers", and it's used internally by the dynamic linker. Android doesn't need to take any steps to support it, and there's little they could do to remove support for it.
      - pjmlp 1y ago
        I got that idea knowing ISO C standard, POSIX and not mixing themes. By not being part of the official stable APIs, Google is free to break whatever userspace application they feel like. There are seccomp and sandboxing filters in place for what is exposed to non-system apps, aka regular PlayStore apps.
- forrestthewoods 1y ago
  Could not possibly disagree more. Trying to hide details from developers causes immense and suffering.
  - gertop 1y ago
    Absolutely. Breaking apps for all users when the system's page size changes is much better than inconveniencing the handful of developers that would have to work around a more abstracted page size. Same reason why I think electron is great. Devex for 3 people is so much more important than ux for the 30M users.
    - jeffbee 1y ago
      This change doesn't break all apps. It breaks a small subset. Just like you wanted.
    - forrestthewoods 1y ago
      > why I think electron is great I take it back. It is possible for me to disagree more strongly. You don’t need to abstract away page size. Abstraction isn’t the solution to all problems. You just have to expose it. Devs shouldn’t assume page size. They simply need to be able query whether it’s 4 or 16 or 64 or however large and voila.
      - nneonneo 1y ago
        That exists via sysconf(_SC_PAGESIZE). The problem is that hard coding the page size to 4k has worked just fine for much of the software written in the last few decades, which means that the 4k size has ossified to some extent (Hyrum’s Law).
      - jyrkesh 1y ago
        > Devex for 3 people is so much more important than ux for the 30M users. I think they were being sarcastic, and you might agree more than you realize.
        1y ago
- o11c 1y ago
  The underlying problem is that there are several different things, and many of them currently don't have APIs: * the known-constant page size that must be passed so you won't get failure from `mmap(MAP_FIXED)`, `mprotect`, etc. (even if you don't call these directly, ELF relies on them and traditionally 4K was used on many platforms; raising it requires a version of ld.so with proper support). There is no macro for this. On platforms I know about, it varies from 4K to 64K. Setting it to a higher multiple of 2 is by design always safe. * the known-constant page size that unconstrained `mmap` is guaranteed to return a multiple of. There is no macro for this. This is 4K on every platform I know about, but I'm not omniscient. Setting it to a lower multiple of 2 is by design always safe. * the dynamic page size that the current kernel is actually using. You can get this using `getpagesize()` or `sysconf(_SC_PAGESIZE)` (incidentally the man page for `getpagesize(2)` is outdated by assuming page size only varies by machine, not at boot time) The macro `PAGESIZE` is traditionally provided if the upper/lower bounds are identical, and very many programs rely on it. Unfortunately, there's no way to ask the kernel to increase the alignment that unconstrained `mmap` returns, which would safely allow defining PAGESIZE to the largest possible value. Note that it is possible to get compiler errors for incompatible definitions of constants (or ranges thereof) by relying on `ld` magic (each value pulls in variable definied in a distinct object file that also defines an additional shared variable, causing multiple-definition errors), but this would need to be done by whoever provides the macros (which should be libc).
- dataflow 1y ago
  > Letting userspace know the page size was IMO a design mistake. How would you have had users handle SIMD tails?
  - pornel 1y ago
    Fixed-length SIMD was a design mistake too. But realistically, you only need to know the lower bound for the page size, so pages larger by an unknown multiple are not a problem. Or use masked loads, and don't even worry about pages.
- deleted 1y ago
  [deleted]
Wowfunhappy 1y ago
So, do 4 KB page size apps work on these 16 KB page size phones? There are plenty of legacy apps (especially games I'd imagine) that will never updated for one reason or another. I would want to know if I was buying a phone that won't support those...
- jsheard 1y ago
  That ship already sailed, most legacy Android games were killed by 32bit support being phased out both in hardware (newer SOCs are 64bit only) and software (Android 14 doesn't support 32bit apps even when running on an older SOC). They've done it before and they'll do it again.
  - mappu 1y ago
    Which is a shame, a modern SoC is fast enough to emulate 32-bit in userspace with same-or-better performance than the real 32-bit SoCs from that period.
  - iggldiggl 1y ago
    > Android 14 doesn't support 32bit apps even when running on an older SOC Huh? Tell that to my phone. Maybe Google dropped support from their own OS builds even for older phones, but I don't think AOSP as such has dropped 32-bit-compatibility yet, and other OEMs have kept 32-bit-support for those phones whose hardware still supports it. (And at least one or two Chinese OEMs have been shipping some sort of compatibility layer for phones without hardware support, though possibly only for the domestic market, not export models/OS builds.) It's true though that Google isn't too heavily invested in binary compatibility for native code, though to some extent their hand might have also been forced by the SOC manufacturers, because apparently 32-bits hardware compatibility is more expensive to provide on ARM.
  - 71bw 1y ago
    >(Android 14 doesn't support 32bit apps even when running on an older SOC). Completely untrue and can be entirely circumvented by one ADB flag (--bypass-low-target-sdk-block) or a third-party .apk installer.
malkia 1y ago
This makes it now 4x more expensive to catch memory issues by having guard pages around...
- fulafel 1y ago
  In which currency? VA space is plentiful.
  - nolist_policy 1y ago
    Only 512Gb virtual address space an Android arm64...
    - entropicdrifter 1y ago
      Sorry, are you talking about a quantity of ram in bits, or did you just forget to capitalize the b in "GB"?
      - AStonesThrow 1y ago
        “GB” would also be wrong in terms of RAM: you’re looking for “GiB”.
        shpx 1y ago
        bibibytes are not going to happen
- junon 1y ago
  Not really. Guard pages don't actually cost any physical memory.
  - malkia 1y ago
    They do... each allocation that is protecting must be aligned by 4kb, and now by 16kb (ideally padded at end) On top of that, adding 4kb (or 16kb) pages sprinkled across adds more to the TLB cache.
trebligdivad 1y ago
Why is camera start so much quicker due to the larger page size - seems a surprising (nice) benefit.
greatgib 1y ago
I'm wondering if it will not make the apps overall memory usage explode a little bit more by having a lot of objects bound to a single page with more empty space unused inside most pages for common cases.

88 points by ingve 1y ago

Letting userspace know the page size was IMO a design mistake. Imagine a world where the page size is secret to userspace. Anything that needs page size alignment will be chosen by the kernel. That in turn allows mixed page size, variable page size, heirarchical pages, etc.

Dylan16807 1y ago
It's hard to hide all the details, especially when you involve varied permissions over different chunks of memory.
duskwuff 1y ago
It's unavoidable. APIs like mprotect() operate on pages; there's no way to hide that from them.
- viraptor 1y ago
  Also various performance characteristics graphs will have steps around the multiples of page sizes. People would find out anyway. (Although maybe the runtime detection is not a bad idea)
  - londons_explore 1y ago
    Just like cache size. But when someone releases a new CPU with a larger or smaller cache, all old software continues to work. Secret page size would offer the same benefit.
    - dataflow 1y ago
      Cache line size would be a better example.
      - Tuna-Fish 1y ago
        Yup. And like with page sizes, the big problems with cache line sizes is not when people design things against a specific line size, but the (much more numerous) cases where they design something that only works well for a fixed size but they don't even know that, because they literally never thought about it, and it worked fine because every machine they used was similar. It doesn't matter what apis you were provided or what analysis you did, you don't know your software works with different hardware until you test it on it.
- londons_explore 1y ago
  you can simulate arbitrary sized mprotect() by having the kernel do the closest it can using the hardware, and then any pages crossing the boundary will be handled by page faults. The performance hit should be small as long as most mprotect regions are large (which they typically are).
  - dzaima 1y ago
    That'd fail horrifically for guard pages around a main region where the whole used region is reasonably expected to be used. Considering that it'd be literally completely entirely unacceptable to ever hit such an emulated range in code of any significance as it'd be a slowdown in the thousands of times, it'd make mprotect entirely pointless and effectively broken. Unless of course you add "padding" of expected page size (not applicable for guard pages though, those are just SOL), which is basically status quo except that, instead of apps doing hard-coding crashing on unexpected page size, they just become potentially 1000x slower, but of course in practice there's no difference between a crash and 1000x slower. I'd even say a crash is better - means a bug report of it can't be dismissed as just a perf issue, and it's clear that something's going wrong in the first place. The size of the mprotect region does not matter. What matters is worst-case behavior, and, regardless of the size, if a single critical byte of memory ends up in the emulated region, your program is dead to the user, causing data loss or whatever consequences from being, practically speaking, completely frozen your program has.
  - dwattttt 1y ago
    Emulating it would work quite terribly. With a 4kb page size you can write 4kb of memory as RW, then an adjacent 4kb of memory as RX, and have them interact. Trying to emulate that under 16kb pages would fault every memory operation.
- londons_explore 1y ago
  the mprotect API could have been designed more like malloc() - ie. you don't protect a preexisting memory range - instead the API returns a new memory range with the protections you've asked for, possibly copying a load of data into it for you incase you asked for a readonly range. And that 'copy' might be zero-overhead remapping of the original pages.
  - duskwuff 1y ago
    There's some important use cases for mprotect(), like toggling W^X on JIT pages, which that wouldn't work for.
    - Jyaif 1y ago
      W^X would be supported by the API that they suggest: "possibly copying a load of data into it for you incase you asked for a readonly range."
      - duskwuff 1y ago
        For JIT, you specifically need to be able to write some code to a page, then switch that page from writable to executable in place. Having the page move to a new location isn't acceptable, as the generated code will contain relative calls to other JIT generated code in other pages you've previously written to.
        Jyaif 1y ago
        Oh right, good point!
    - PhilipRoman 1y ago
      You can probably replace some of those use cases by mapping the same physical page in two places with different permissions (W^X is per-address, not per physical page).
    - IshKebab 1y ago
      If you had a way of allocating pages of at least N bytes it would work. uint8_t* mem = malloc_pages(1024); mprotect_paged(mem, 1024);
- pjmlp 1y ago
  That isn't officially supported API, so if any app on NDK is making use of it, though luck. https://developer.android.com/ndk/guides/stable_apis https://developer.android.com/ndk/guides/stable_apis
  - duskwuff 1y ago
    I'm not sure how you get that idea - as a Linux system call, mprotect() is part of "the standard C11 library headers", and it's used internally by the dynamic linker. Android doesn't need to take any steps to support it, and there's little they could do to remove support for it.
    - pjmlp 1y ago
      I got that idea knowing ISO C standard, POSIX and not mixing themes. By not being part of the official stable APIs, Google is free to break whatever userspace application they feel like. There are seccomp and sandboxing filters in place for what is exposed to non-system apps, aka regular PlayStore apps.
forrestthewoods 1y ago
Could not possibly disagree more. Trying to hide details from developers causes immense and suffering.
- gertop 1y ago
  Absolutely. Breaking apps for all users when the system's page size changes is much better than inconveniencing the handful of developers that would have to work around a more abstracted page size. Same reason why I think electron is great. Devex for 3 people is so much more important than ux for the 30M users.
  - jeffbee 1y ago
    This change doesn't break all apps. It breaks a small subset. Just like you wanted.
  - forrestthewoods 1y ago
    > why I think electron is great I take it back. It is possible for me to disagree more strongly. You don’t need to abstract away page size. Abstraction isn’t the solution to all problems. You just have to expose it. Devs shouldn’t assume page size. They simply need to be able query whether it’s 4 or 16 or 64 or however large and voila.
    - nneonneo 1y ago
      That exists via sysconf(_SC_PAGESIZE). The problem is that hard coding the page size to 4k has worked just fine for much of the software written in the last few decades, which means that the 4k size has ossified to some extent (Hyrum’s Law).
    - jyrkesh 1y ago
      > Devex for 3 people is so much more important than ux for the 30M users. I think they were being sarcastic, and you might agree more than you realize.
      - 1y ago
o11c 1y ago
The underlying problem is that there are several different things, and many of them currently don't have APIs: * the known-constant page size that must be passed so you won't get failure from `mmap(MAP_FIXED)`, `mprotect`, etc. (even if you don't call these directly, ELF relies on them and traditionally 4K was used on many platforms; raising it requires a version of ld.so with proper support). There is no macro for this. On platforms I know about, it varies from 4K to 64K. Setting it to a higher multiple of 2 is by design always safe. * the known-constant page size that unconstrained `mmap` is guaranteed to return a multiple of. There is no macro for this. This is 4K on every platform I know about, but I'm not omniscient. Setting it to a lower multiple of 2 is by design always safe. * the dynamic page size that the current kernel is actually using. You can get this using `getpagesize()` or `sysconf(_SC_PAGESIZE)` (incidentally the man page for `getpagesize(2)` is outdated by assuming page size only varies by machine, not at boot time) The macro `PAGESIZE` is traditionally provided if the upper/lower bounds are identical, and very many programs rely on it. Unfortunately, there's no way to ask the kernel to increase the alignment that unconstrained `mmap` returns, which would safely allow defining PAGESIZE to the largest possible value. Note that it is possible to get compiler errors for incompatible definitions of constants (or ranges thereof) by relying on `ld` magic (each value pulls in variable definied in a distinct object file that also defines an additional shared variable, causing multiple-definition errors), but this would need to be done by whoever provides the macros (which should be libc).
dataflow 1y ago
> Letting userspace know the page size was IMO a design mistake. How would you have had users handle SIMD tails?
- pornel 1y ago
  Fixed-length SIMD was a design mistake too. But realistically, you only need to know the lower bound for the page size, so pages larger by an unknown multiple are not a problem. Or use masked loads, and don't even worry about pages.
deleted 1y ago
[deleted]

Wowfunhappy 1y ago

So, do 4 KB page size apps work on these 16 KB page size phones? There are plenty of legacy apps (especially games I'd imagine) that will never updated for one reason or another. I would want to know if I was buying a phone that won't support those...

jsheard 1y ago
That ship already sailed, most legacy Android games were killed by 32bit support being phased out both in hardware (newer SOCs are 64bit only) and software (Android 14 doesn't support 32bit apps even when running on an older SOC). They've done it before and they'll do it again.
- mappu 1y ago
  Which is a shame, a modern SoC is fast enough to emulate 32-bit in userspace with same-or-better performance than the real 32-bit SoCs from that period.
- iggldiggl 1y ago
  > Android 14 doesn't support 32bit apps even when running on an older SOC Huh? Tell that to my phone. Maybe Google dropped support from their own OS builds even for older phones, but I don't think AOSP as such has dropped 32-bit-compatibility yet, and other OEMs have kept 32-bit-support for those phones whose hardware still supports it. (And at least one or two Chinese OEMs have been shipping some sort of compatibility layer for phones without hardware support, though possibly only for the domestic market, not export models/OS builds.) It's true though that Google isn't too heavily invested in binary compatibility for native code, though to some extent their hand might have also been forced by the SOC manufacturers, because apparently 32-bits hardware compatibility is more expensive to provide on ARM.
- 71bw 1y ago
  >(Android 14 doesn't support 32bit apps even when running on an older SOC). Completely untrue and can be entirely circumvented by one ADB flag (--bypass-low-target-sdk-block) or a third-party .apk installer.

malkia 1y ago

This makes it now 4x more expensive to catch memory issues by having guard pages around...

fulafel 1y ago
In which currency? VA space is plentiful.
- nolist_policy 1y ago
  Only 512Gb virtual address space an Android arm64...
  - entropicdrifter 1y ago
    Sorry, are you talking about a quantity of ram in bits, or did you just forget to capitalize the b in "GB"?
    - AStonesThrow 1y ago
      “GB” would also be wrong in terms of RAM: you’re looking for “GiB”.
      - shpx 1y ago
        bibibytes are not going to happen
junon 1y ago
Not really. Guard pages don't actually cost any physical memory.
- malkia 1y ago
  They do... each allocation that is protecting must be aligned by 4kb, and now by 16kb (ideally padded at end) On top of that, adding 4kb (or 16kb) pages sprinkled across adds more to the TLB cache.

trebligdivad 1y ago

Why is camera start so much quicker due to the larger page size - seems a surprising (nice) benefit.

greatgib 1y ago

I'm wondering if it will not make the apps overall memory usage explode a little bit more by having a lot of objects bound to a single page with more empty space unused inside most pages for common cases.