11 ms·

The Perils of Loading Native Libraries on Android

74 points by philippb 11y ago

breatheoften 11y ago
Is the problem discussed in this article something that has to be worried about still? This is hideous ... By my understanding the only way the solution discussed in the article can work is by including the native libraries for every architecture inside the apk which seems like it will make the app larger than it needs to be ... I think I'd rather just detect the issue and throw up an error telling the user that they've managed to install an apk built for the wrong architecture and need to install a different apk ...
- JupiterMoon 11y ago
  I guess you could delete the not needed libraries post installation? Then only the APK would be needlessly larger.
- gizmo686 11y ago
  If you are distributing through Google Play (or, I assume, most 3rd party app stores), then you can provide a different apk depending on the target architecture. The problem they were running into (after they worked around the installation bug) was that some users would sideload their APK, but sideload the wrong version. As far as I can tell, their ReLinker solution is only a workaround to the installation problem.
  - whoopdedo 11y ago
    If you're clever enough to sideload are you not clever enough to recognize and remedy the problem? Developers should not be enabling people who aren't able to handle that responsibility. Actually, now that I think of it, I can see a business requiring employee devices to be flashed with a set of apps but not wanting to put the effort to actually check that it'll work. In that case the developer should determine if the user is an individual or a company and in the later case sell them a contract for support.
    - mschuster91 11y ago
      > If you're clever enough to sideload are you not clever enough to recognize and remedy the problem? Hmm, I always shared apk files with friends when they simply were too big, kinda easy to do with ES File Explorer or other root-using file managers. Lucky for us, our CPU archs never proved to be a problem though. But I can very well imagine this being a problem for warez, or people in countries not served by Google Play (or who simply have no easy access to payment, e.g. because they don't have a CC), or people with devices not sold with Google Play (cheap ass chinese phones).
    - toast0 11y ago
      There are popular "mirror" sites for apks, I wouldn't expect most users of them to be able to debug a wrong arch package download. Ideally the people taking the apks to mirror wouldn't pull thin apks, but sigh
    - fulafel 11y ago
      Many people choose to not link their device to a Google account, so cannot access Google Play. Others use devices that haven't licensed Google Play. Widely used apps are frequently available officially as apk downloads (eg WhatsApp), or other more inclusive app stores (eg f-droid), others you have to get from unofficial sources.
      - mkesper 11y ago
        F-droid hides versions with wrong architecture.
  - mmebane 11y ago
    > As far as I can tell, their ReLinker solution is only a workaround to the installation problem. I guess it should be pretty simple to add a "wrong version for your device" killscreen to an app, once you realize it's a problem.
- DominikR 11y ago
  Yes, if you enable CPU ABI splits there will only be the lib for a specific architecture available in an apk. You could load the lib from a remote server after install if you need to save space but if we are talking about saving 1-3 MB I'd definitely save myself the hassle and just pack the libs for all architectures in one apk file.
voltagex_ 11y ago
I wonder if there's an Android bug report for the first part. And I wonder if Google cares.
modeless 11y ago
> users may have installed our app from other sources than the Play Store Is this a euphemism for piracy? Where are these users getting the app?
- ikeboy 11y ago
  The app https://play.google.com/store/apps/details?id=com.kii.safe https://play.google.com/store/apps/details?id=com.kii.safe is free, so no. Some users might not want google to know they're using a particular app (this one a is a security one, so probably a higher proportion of users would care). There are over 10 million users from the play store, so 100,000 is not that high for sideloaders. Plus, google might block downloads in some countries. There are a number of sites that offer apk downloads for free apps: see http://apkpure.com/ http://apkpure.com/, https://apps.evozi.com/apk-downloader/ https://apps.evozi.com/apk-downloader/, or http://apk-dl.com/ http://apk-dl.com/. See also http://www.makeuseof.com/tag/download-apk-google-play-bypass-restrictions/ http://www.makeuseof.com/tag/download-apk-google-play-bypass... for more reasons people might not want to use Google Play.
- swiley 11y ago
  Dealing with the play store can be a bit of a pain. The UI really isn't that great and the downloads are sometimes very slow. So it's honestly just a whole lot easier to side load sometimes.
- swiley 11y ago
  Dealing with the play store can be a bit of a pain. The UI really isn't that great and the downloads are sometimes very slow. So it's honestly just a whole lot easier to side load sometimes.
- detaro 11y ago
  Some devs sell APKs directly for users with devices without Play store. Samsung has their own app store. As does Amazon. And there is f-droid for open-source stuff. And I bet there are quite a few ones with regional importance.
jheriko 11y ago
surely the perils of users using shoddy app stores and installers rather than developers using native libraries. kudos to the devs for making the effort to workaround the issues though, its quite a heroic effort imo.
- ianlevesque 11y ago
  For another heroic effort read: https://www.facebook.com/notes/facebook-engineering/under-the-hood-dalvik-patch-for-facebook-for-android/10151345597798920 https://www.facebook.com/notes/facebook-engineering/under-th...
- xiphirx 11y ago
  Hi, author here. This is definitely not a case of using shoddy app stores or installers. The vast majority of the problems stemmed from installations that came from Google Play. The article mentions a tidbit about alternate installation sources as a heads up / piece of advice for people running into the problem because we were still confused as to why it was happening for a small number of people. It just turned out that they were using shoddy app stores / installing manually, and were installing the wrong version for their device.
ianlevesque 11y ago
There are two Androids: the quirky but generally usable Android described in Google's documentation and occasionally found on Nexus devices, and then the Android actually found in the wild after manufacturers, carriers, 3rd party app stores, users and general neglect have their way with it.
- davtbaum 11y ago
  But it wasn't the fragmented Android that caused this bug... The article acknowledges that the root cause was an existing bug in Package Manager.
  - ianlevesque 11y ago
    Sure, and bugs happen and Google often fixes them, but the fixes don't reliably end up in the hands of users before devices eventually die and are replaced. It's hard for me to believe this is still an issue this many years into Android's existence, but here we are.
  - JohnTHaller 11y ago
    Security vulnerabilities and bugs happen regularly in every major operating system. Windows, Mac OS X, Linux, iOS, Windows Mobile, Android, etc. Most of the time these bugs are patched in a timely fashion due to the fact that the publisher can release a bug fix directly to the end user's device. This applies to everything except most Android devices. When a major bug is discovered in Android, first, Google fixes it and published the code. This is usually done quickly. Second, the phone manufacturers take that fix and incorporate it into their own Android build process with all their extra layers (HTC Sense, Samsung TouchWiz, etc). This second step takes anywhere from a few weeks to infinity (aka it never happens). Third, the carrier takes the manufacturers build and adds their own cruft to it, maybe tests it, and then pushes it out to their customers as an over the air (OTA) update. This third step takes anywhere from a month to infinity (aka it never happens). Due to the interference of manufacturers and carriers, I would not recommend using Android on anything other than a Nexus device purchased directly from Google or a retail/online store. Even a Nexus, when purchased from a carrier, won't get updates as quickly as it should (speaking from experience with my Nexus 6 and T-Mobile).
rsp1984 11y ago
There is absolutely a ton of stuff broken with Android, no doubt about it, and I don't want to sound schoolmasterly but in this case it looks like a RTFM. From the Android NDK docs [1]: You must specify an ABI for each CPU architecture you want your app to work with ... To build machine code for two or more distinct ABIs, using spaces as delimiters. For example: APP_ABI := armeabi armeabi-v7a This setting tells the NDK to build two versions of your machine code: one for each ABI listed on this line. Whereas the post says: To reduce our APK size and ensure that our App would run on all possible devices, we have flavors of our App for the x86, Armv7 and Arm architectures. Each flavor only contains the native libraries corresponding to its respective architecture There we have it. [1] http://developer.android.com/ndk/guides/abis.html http://developer.android.com/ndk/guides/abis.html
- Aqueous 11y ago
  Does anyone have data that shows the APK size actually hurts the downloads / purchases of an application? Regardless I think this was a mistake on the developers' part and could have been anticipated. First of all, it is important to have a smaller APK for users in developing economies and/or low incomes, but today's newer devices come with so much space that going to great lengths to reduce the APK size early on is a premature optimization, especially when we're talking about removing libraries that are required for proper running of the APK on every conceivable handset configuration. You should only ever reduce the APK by removing things that absolutely are not critical to run the same application package everywhere. So: First, ensure that your app runs for everyone. Then find ways to reduce the APK size without compromising that. And yes, I'm aware that Android / Google Play has added APK splitting, allowing platform-specific APKs to be distributed from the Play store. I actually would argue on balance this was probably a bad idea which presumes an iTunes-like single distribution channel.
  - izacus 11y ago
    Yes, APK size is still critical. We still see a huge amount of people owning and running cheaper devices which have problems installing large apps. Your friends with 700EUR phones aren't really all of the market and there's still enough devices out there that will not successfully install a 30+MB APK. And that's not even talking about how wasting users storage for things he doesn't need (just because you're a bit lazy) is just... wrong.
js2 11y ago
We've seem the same problem at Yahoo: https://github.com/yahoo/ygloo-ymagine/blob/master/src/com/yahoo/mobile/client/android/ymagine/LibraryLoader.java https://github.com/yahoo/ygloo-ymagine/blob/master/src/com/y... Here's another fun Android / PackageManager behavior - Android doesn't kill an app before upgrading it (unlike on iOS). You may think this is a feature, except that after the upgrade the PackageManager is now out-of-sync with the running code. So for example, say v1 of the code is still running after an upgrade to v2, and the code calls getPackageInfo() in order to find out say, its versionCode. The running code will erroneously think it's v2.
- Natanael_L 11y ago
  Should you try to detect an initiated install and close?
ignoramous 11y ago
Fire OS fixed this bug in the Package Manager. Amazon treats both, the buyers and the developers, as its customers. Has a nice process in place to test top 10k+ apps on each upgrade and identify and try and fix issues arising due to AOSP. Google needs to start doing the same thing or something similar. I have seen a fair share of broken stuff. Something even as basic as a ListView was horrendously let loose in a middle of a refactor between JB and L.

74 points by philippb 11y ago

Is the problem discussed in this article something that has to be worried about still? This is hideous ... By my understanding the only way the solution discussed in the article can work is by including the native libraries for every architecture inside the apk which seems like it will make the app larger than it needs to be ... I think I'd rather just detect the issue and throw up an error telling the user that they've managed to install an apk built for the wrong architecture and need to install a different apk ...

JupiterMoon 11y ago
I guess you could delete the not needed libraries post installation? Then only the APK would be needlessly larger.
gizmo686 11y ago
If you are distributing through Google Play (or, I assume, most 3rd party app stores), then you can provide a different apk depending on the target architecture. The problem they were running into (after they worked around the installation bug) was that some users would sideload their APK, but sideload the wrong version. As far as I can tell, their ReLinker solution is only a workaround to the installation problem.
- whoopdedo 11y ago
  If you're clever enough to sideload are you not clever enough to recognize and remedy the problem? Developers should not be enabling people who aren't able to handle that responsibility. Actually, now that I think of it, I can see a business requiring employee devices to be flashed with a set of apps but not wanting to put the effort to actually check that it'll work. In that case the developer should determine if the user is an individual or a company and in the later case sell them a contract for support.
  - mschuster91 11y ago
    > If you're clever enough to sideload are you not clever enough to recognize and remedy the problem? Hmm, I always shared apk files with friends when they simply were too big, kinda easy to do with ES File Explorer or other root-using file managers. Lucky for us, our CPU archs never proved to be a problem though. But I can very well imagine this being a problem for warez, or people in countries not served by Google Play (or who simply have no easy access to payment, e.g. because they don't have a CC), or people with devices not sold with Google Play (cheap ass chinese phones).
  - toast0 11y ago
    There are popular "mirror" sites for apks, I wouldn't expect most users of them to be able to debug a wrong arch package download. Ideally the people taking the apks to mirror wouldn't pull thin apks, but sigh
  - fulafel 11y ago
    Many people choose to not link their device to a Google account, so cannot access Google Play. Others use devices that haven't licensed Google Play. Widely used apps are frequently available officially as apk downloads (eg WhatsApp), or other more inclusive app stores (eg f-droid), others you have to get from unofficial sources.
    - mkesper 11y ago
      F-droid hides versions with wrong architecture.
- mmebane 11y ago
  > As far as I can tell, their ReLinker solution is only a workaround to the installation problem. I guess it should be pretty simple to add a "wrong version for your device" killscreen to an app, once you realize it's a problem.
DominikR 11y ago
Yes, if you enable CPU ABI splits there will only be the lib for a specific architecture available in an apk. You could load the lib from a remote server after install if you need to save space but if we are talking about saving 1-3 MB I'd definitely save myself the hassle and just pack the libs for all architectures in one apk file.

voltagex_ 11y ago

I wonder if there's an Android bug report for the first part. And I wonder if Google cares.

modeless 11y ago

> users may have installed our app from other sources than the Play Store Is this a euphemism for piracy? Where are these users getting the app?

ikeboy 11y ago
The app https://play.google.com/store/apps/details?id=com.kii.safe https://play.google.com/store/apps/details?id=com.kii.safe is free, so no. Some users might not want google to know they're using a particular app (this one a is a security one, so probably a higher proportion of users would care). There are over 10 million users from the play store, so 100,000 is not that high for sideloaders. Plus, google might block downloads in some countries. There are a number of sites that offer apk downloads for free apps: see http://apkpure.com/ http://apkpure.com/, https://apps.evozi.com/apk-downloader/ https://apps.evozi.com/apk-downloader/, or http://apk-dl.com/ http://apk-dl.com/. See also http://www.makeuseof.com/tag/download-apk-google-play-bypass-restrictions/ http://www.makeuseof.com/tag/download-apk-google-play-bypass... for more reasons people might not want to use Google Play.
swiley 11y ago
Dealing with the play store can be a bit of a pain. The UI really isn't that great and the downloads are sometimes very slow. So it's honestly just a whole lot easier to side load sometimes.
swiley 11y ago
Dealing with the play store can be a bit of a pain. The UI really isn't that great and the downloads are sometimes very slow. So it's honestly just a whole lot easier to side load sometimes.
detaro 11y ago
Some devs sell APKs directly for users with devices without Play store. Samsung has their own app store. As does Amazon. And there is f-droid for open-source stuff. And I bet there are quite a few ones with regional importance.

jheriko 11y ago

surely the perils of users using shoddy app stores and installers rather than developers using native libraries. kudos to the devs for making the effort to workaround the issues though, its quite a heroic effort imo.

ianlevesque 11y ago
For another heroic effort read: https://www.facebook.com/notes/facebook-engineering/under-the-hood-dalvik-patch-for-facebook-for-android/10151345597798920 https://www.facebook.com/notes/facebook-engineering/under-th...
xiphirx 11y ago
Hi, author here. This is definitely not a case of using shoddy app stores or installers. The vast majority of the problems stemmed from installations that came from Google Play. The article mentions a tidbit about alternate installation sources as a heads up / piece of advice for people running into the problem because we were still confused as to why it was happening for a small number of people. It just turned out that they were using shoddy app stores / installing manually, and were installing the wrong version for their device.

ianlevesque 11y ago

There are two Androids: the quirky but generally usable Android described in Google's documentation and occasionally found on Nexus devices, and then the Android actually found in the wild after manufacturers, carriers, 3rd party app stores, users and general neglect have their way with it.

davtbaum 11y ago
But it wasn't the fragmented Android that caused this bug... The article acknowledges that the root cause was an existing bug in Package Manager.
- ianlevesque 11y ago
  Sure, and bugs happen and Google often fixes them, but the fixes don't reliably end up in the hands of users before devices eventually die and are replaced. It's hard for me to believe this is still an issue this many years into Android's existence, but here we are.
- JohnTHaller 11y ago
  Security vulnerabilities and bugs happen regularly in every major operating system. Windows, Mac OS X, Linux, iOS, Windows Mobile, Android, etc. Most of the time these bugs are patched in a timely fashion due to the fact that the publisher can release a bug fix directly to the end user's device. This applies to everything except most Android devices. When a major bug is discovered in Android, first, Google fixes it and published the code. This is usually done quickly. Second, the phone manufacturers take that fix and incorporate it into their own Android build process with all their extra layers (HTC Sense, Samsung TouchWiz, etc). This second step takes anywhere from a few weeks to infinity (aka it never happens). Third, the carrier takes the manufacturers build and adds their own cruft to it, maybe tests it, and then pushes it out to their customers as an over the air (OTA) update. This third step takes anywhere from a month to infinity (aka it never happens). Due to the interference of manufacturers and carriers, I would not recommend using Android on anything other than a Nexus device purchased directly from Google or a retail/online store. Even a Nexus, when purchased from a carrier, won't get updates as quickly as it should (speaking from experience with my Nexus 6 and T-Mobile).

rsp1984 11y ago

There is absolutely a ton of stuff broken with Android, no doubt about it, and I don't want to sound schoolmasterly but in this case it looks like a RTFM. From the Android NDK docs [1]: You must specify an ABI for each CPU architecture you want your app to work with ... To build machine code for two or more distinct ABIs, using spaces as delimiters. For example: APP_ABI := armeabi armeabi-v7a This setting tells the NDK to build two versions of your machine code: one for each ABI listed on this line. Whereas the post says: To reduce our APK size and ensure that our App would run on all possible devices, we have flavors of our App for the x86, Armv7 and Arm architectures. Each flavor only contains the native libraries corresponding to its respective architecture There we have it. [1] http://developer.android.com/ndk/guides/abis.html http://developer.android.com/ndk/guides/abis.html

Aqueous 11y ago
Does anyone have data that shows the APK size actually hurts the downloads / purchases of an application? Regardless I think this was a mistake on the developers' part and could have been anticipated. First of all, it is important to have a smaller APK for users in developing economies and/or low incomes, but today's newer devices come with so much space that going to great lengths to reduce the APK size early on is a premature optimization, especially when we're talking about removing libraries that are required for proper running of the APK on every conceivable handset configuration. You should only ever reduce the APK by removing things that absolutely are not critical to run the same application package everywhere. So: First, ensure that your app runs for everyone. Then find ways to reduce the APK size without compromising that. And yes, I'm aware that Android / Google Play has added APK splitting, allowing platform-specific APKs to be distributed from the Play store. I actually would argue on balance this was probably a bad idea which presumes an iTunes-like single distribution channel.
- izacus 11y ago
  Yes, APK size is still critical. We still see a huge amount of people owning and running cheaper devices which have problems installing large apps. Your friends with 700EUR phones aren't really all of the market and there's still enough devices out there that will not successfully install a 30+MB APK. And that's not even talking about how wasting users storage for things he doesn't need (just because you're a bit lazy) is just... wrong.

js2 11y ago

We've seem the same problem at Yahoo: https://github.com/yahoo/ygloo-ymagine/blob/master/src/com/yahoo/mobile/client/android/ymagine/LibraryLoader.java https://github.com/yahoo/ygloo-ymagine/blob/master/src/com/y... Here's another fun Android / PackageManager behavior - Android doesn't kill an app before upgrading it (unlike on iOS). You may think this is a feature, except that after the upgrade the PackageManager is now out-of-sync with the running code. So for example, say v1 of the code is still running after an upgrade to v2, and the code calls getPackageInfo() in order to find out say, its versionCode. The running code will erroneously think it's v2.

Natanael_L 11y ago
Should you try to detect an initiated install and close?

ignoramous 11y ago

Fire OS fixed this bug in the Package Manager. Amazon treats both, the buyers and the developers, as its customers. Has a nice process in place to test top 10k+ apps on each upgrade and identify and try and fix issues arising due to AOSP. Google needs to start doing the same thing or something similar. I have seen a fair share of broken stuff. Something even as basic as a ListView was horrendously let loose in a middle of a refactor between JB and L.