6 ms·

I didn't know what single/multi-dex means so i looked it up. Android application (APK) files contain executable bytecode files in the form of Dalvik Executable

by 40acres 10y ago

I didn't know what single/multi-dex means so i looked it up. Android application (APK) files contain executable bytecode files in the form of Dalvik Executable (DEX) files, which contain the compiled code used to run your app. The Dalvik Executable specification limits the total number of methods that can be referenced within a single DEX file to 65,536, including Android framework methods, library methods, and methods in your own code. Getting past this limit requires that you configure your app build process to generate more than one DEX file, known as a multidex configuration. Do Android apps regularly encompass over 65.3k methods? I assume a majority of these would be framework type functionality, but it really seems that a multi-dex product would be near impossible to maintain.

aeturnum 10y ago
Iirc, the reason you end up hitting the limit for 65k methods is that each library tends to keep independent copies of its dependencies. This simplifies dependency control, but ends up mushrooming quickly.
- unsoundInput 10y ago
  That is actually not true. Afaik multiversioning of dependencies is not actually supported by build systems without manual effort (not counting major versions in non-conflicting namespaces). Problem is really more in fat libraries and "unnecessary" (getters, setters) and synthetic (e.g. access to private methods from inner classes) methods that are usually not optimized away, especially in debug builds.
  - ashark 10y ago
    I've definitely included a library and then seen (otherlibrary).(library you included) as an option in autocomplete, fairly often. Happens a lot with utility-level things like okhttp.
    - zcoyle 10y ago
      I've seen this with okhttp as well.
      - unsoundInput 10y ago
        I can't find the okhttp case you reference, but retrofit has a bunch of extra adapter artifacts that add an external library as a transient dependency to your project. The code for the binding classes lives in retrofit.adapter.{guava|rxjava|…} [0] but the respective library still lives in its usual package. [1] If that weren't the case you could A) not manually provide a minor version via your dependencies block in your build script and B) would have interop problems between libraries and between a library and your code as the same interface copied to a different packages is not the same interface for Java. [0] https://github.com/square/retrofit/tree/master/retrofit-adapters/guava/src https://github.com/square/retrofit/tree/master/retrofit-adap... [1] https://github.com/square/retrofit/blob/master/retrofit-adapters/guava/src/main/java/retrofit2/adapter/guava/GuavaCallAdapterFactory.java#L18 https://github.com/square/retrofit/blob/master/retrofit-adap...
myroon5 10y ago
Usually most of the methods are from dependencies. Interestingly enough, some of Google's own libraries like Guava and Google Play Services are commonly what pushes apps over the limit.
- otachack 10y ago
  True, but Google Play Services can now be cherry picked so you can depend only on what you want to use rather than a blanket dependency. See table one here: https://developers.google.com/android/guides/setup https://developers.google.com/android/guides/setup
habosa 10y ago
Almost every Android app that includes the most common dependencies (Android support lib, Google Play services, etc) will end up being multi-dex. This used to be a massive pain but lately the Android Studio / gradle multidex tooling has gotten so good that I don't notice anymore.
- krschultz 10y ago
  You still incur a startup penalty with multi-dex, it takes noticeably longer than a single dex, even if you are only marginally into that 2nd dex.
bitmapbrother 10y ago
This is one of those Android design decisions that Google probably wishes they could get to do over again. The DEX header file format allocates only 2 bytes to hold the number of methods the app can reference. This was a very short sighted decision that has come back to bite them repeatedly. It would be interesting to know why they only allocated 2 bytes instead of 4 which would have given them the ability to reference over 4 billion methods. If their reasoning was for cost saving measures than it was a pretty silly decision. There has been some talk about revising the DEX header file format to correct some of the early design mistakes, but I'm not sure when they'll get around to it.
- kuschku 10y ago
  It's not that easy, and not just one field. DEX uses a method table, listing every method with an ID, and using in the rest of the file only that ID to refer to it. This ID is limited to 2 bytes. Obviously, thats an issue, but short of using varint or straightaway uint32 there's few alternatives. And those use either more CPU or memory.
  - shortstuffsushi 10y ago
    You make it sound like it is as simple as that, only a method table instead of a header field :) That said, you mention using a varint or uint32 which "use more CPU or memory" -- 16 bits vs 32 bits in applications that use megabytes seems trivial. Can you explain why that's a concern?
    - kuschku 10y ago
      It's not a concern anymore, but when Android was created, the average application was less than 20kiB. Back then, resource restrictions were a lot more problematic.
  - bitmapbrother 10y ago
    Thanks for the correction. Here's a transcript from the Android Developer podcast, courtesy of http://blog.osom.info/2014/10/multi-dex-to-rescue-from-infamous-65536.html http://blog.osom.info/2014/10/multi-dex-to-rescue-from-infam..., talking about the issue with an Android engineer: >Tor: So [regarding] the infamous 64k method [issue].. I understand that it is a dex file format limitation.. which is a short integer. Do you have plans to address this somehow, by either change the dex format or some other way? >Anwar: So we've talked about revving DEX, so that limitation doesn't exist anymore. And there's a couple of reasons we haven't done that: one, there're other things we would like to do better as well, including supporting new language features. The other reason is - it doesn't help us with devices still in the field. It's hard to go and say "by the way, we're going to upgrade your runtime.." So what do we do in order to address that? We will do the DEX byte code [change], but I think there's sort of building block that we needed first - what we're calling multi-DEX. And the idea is.. here's something people were doing - they were breaking up their files into multiple DEX files (each of which exceeds the 64k limit), the main classes in DEX file could see the classes and use them in sort of references rather than loading those classes and having limitations in how you can use them. They will go and use reflection to find the boot class path and modify it to include their secondary DEX files. So this is kind of hack, but a necessary one for them. >What we're doing in L is in runtime we will have a native support for multi-DEX. All your DEX files that you have in your app will get dexopt-ed or compiled by us and collapsed into a single .oat file which is a binary file we generate.. ..And we have a support library on top of that.. ..if you have multi-DEX it will work on older releases of Android back to Ice Cream Sandwich.. will work on Gingerbread too, but we're only validating back to ICS. So once you have that, than that free's you in the future to do the DEX byte code change, and then something that partitions it and runs on the existing [devices].. One interesting note is that he mentions that they'll hopefully have the DEX revision by M. Well, we're on N so it looks like they missed that target date. Perhaps O will finally get the long awaited DEX file format revision.
hota_mazi 10y ago
There is no practical difference between single and multidex. Recent versions of Android are even better at creating these dex files effectively so users won't even notice the difference. Most of these methods are typically contributed by dependencies, not by the app itself.
- therealasdf 10y ago
  Exactly, the only difference developers might notice is slower build times.
whatever_dude 10y ago
> Do Android apps regularly encompass over 65.3k methods? No. I've built huge applications that didn't come nowhere near that. I don't dispute some apps might run into that, but it's not a regular occurrence at all. The real problem here is that it depends on what kind of libraries you're using. If you're liberal with adding a lot of all-purpose libraries when you only need a feature or two, you can run into that limit fast. For example, look at the comparison table ("Replacing existing libraries") here: http://jeroenmols.com/blog/2016/05/06/methodcount/ http://jeroenmols.com/blog/2016/05/06/methodcount/ If you just need to load an image, do you want a library with 12k methods (!) or 800?
- ronocod 10y ago
  It depends, Picasso is smallest but doesn't support GIFs. Glid is larger and does support GIFs. Fresco is larger again but has more advanced memory management and better transition handling.

by 40acres 10y ago

aeturnum 10y ago

Iirc, the reason you end up hitting the limit for 65k methods is that each library tends to keep independent copies of its dependencies. This simplifies dependency control, but ends up mushrooming quickly.

unsoundInput 10y ago
That is actually not true. Afaik multiversioning of dependencies is not actually supported by build systems without manual effort (not counting major versions in non-conflicting namespaces). Problem is really more in fat libraries and "unnecessary" (getters, setters) and synthetic (e.g. access to private methods from inner classes) methods that are usually not optimized away, especially in debug builds.
- ashark 10y ago
  I've definitely included a library and then seen (otherlibrary).(library you included) as an option in autocomplete, fairly often. Happens a lot with utility-level things like okhttp.
  - zcoyle 10y ago
    I've seen this with okhttp as well.
    - unsoundInput 10y ago
      I can't find the okhttp case you reference, but retrofit has a bunch of extra adapter artifacts that add an external library as a transient dependency to your project. The code for the binding classes lives in retrofit.adapter.{guava|rxjava|…} [0] but the respective library still lives in its usual package. [1] If that weren't the case you could A) not manually provide a minor version via your dependencies block in your build script and B) would have interop problems between libraries and between a library and your code as the same interface copied to a different packages is not the same interface for Java. [0] https://github.com/square/retrofit/tree/master/retrofit-adapters/guava/src https://github.com/square/retrofit/tree/master/retrofit-adap... [1] https://github.com/square/retrofit/blob/master/retrofit-adapters/guava/src/main/java/retrofit2/adapter/guava/GuavaCallAdapterFactory.java#L18 https://github.com/square/retrofit/blob/master/retrofit-adap...

myroon5 10y ago

Usually most of the methods are from dependencies. Interestingly enough, some of Google's own libraries like Guava and Google Play Services are commonly what pushes apps over the limit.

otachack 10y ago
True, but Google Play Services can now be cherry picked so you can depend only on what you want to use rather than a blanket dependency. See table one here: https://developers.google.com/android/guides/setup https://developers.google.com/android/guides/setup

habosa 10y ago

Almost every Android app that includes the most common dependencies (Android support lib, Google Play services, etc) will end up being multi-dex. This used to be a massive pain but lately the Android Studio / gradle multidex tooling has gotten so good that I don't notice anymore.

krschultz 10y ago
You still incur a startup penalty with multi-dex, it takes noticeably longer than a single dex, even if you are only marginally into that 2nd dex.

bitmapbrother 10y ago

This is one of those Android design decisions that Google probably wishes they could get to do over again. The DEX header file format allocates only 2 bytes to hold the number of methods the app can reference. This was a very short sighted decision that has come back to bite them repeatedly. It would be interesting to know why they only allocated 2 bytes instead of 4 which would have given them the ability to reference over 4 billion methods. If their reasoning was for cost saving measures than it was a pretty silly decision. There has been some talk about revising the DEX header file format to correct some of the early design mistakes, but I'm not sure when they'll get around to it.

kuschku 10y ago
It's not that easy, and not just one field. DEX uses a method table, listing every method with an ID, and using in the rest of the file only that ID to refer to it. This ID is limited to 2 bytes. Obviously, thats an issue, but short of using varint or straightaway uint32 there's few alternatives. And those use either more CPU or memory.
- shortstuffsushi 10y ago
  You make it sound like it is as simple as that, only a method table instead of a header field :) That said, you mention using a varint or uint32 which "use more CPU or memory" -- 16 bits vs 32 bits in applications that use megabytes seems trivial. Can you explain why that's a concern?
  - kuschku 10y ago
    It's not a concern anymore, but when Android was created, the average application was less than 20kiB. Back then, resource restrictions were a lot more problematic.
- bitmapbrother 10y ago
  Thanks for the correction. Here's a transcript from the Android Developer podcast, courtesy of http://blog.osom.info/2014/10/multi-dex-to-rescue-from-infamous-65536.html http://blog.osom.info/2014/10/multi-dex-to-rescue-from-infam..., talking about the issue with an Android engineer: >Tor: So [regarding] the infamous 64k method [issue].. I understand that it is a dex file format limitation.. which is a short integer. Do you have plans to address this somehow, by either change the dex format or some other way? >Anwar: So we've talked about revving DEX, so that limitation doesn't exist anymore. And there's a couple of reasons we haven't done that: one, there're other things we would like to do better as well, including supporting new language features. The other reason is - it doesn't help us with devices still in the field. It's hard to go and say "by the way, we're going to upgrade your runtime.." So what do we do in order to address that? We will do the DEX byte code [change], but I think there's sort of building block that we needed first - what we're calling multi-DEX. And the idea is.. here's something people were doing - they were breaking up their files into multiple DEX files (each of which exceeds the 64k limit), the main classes in DEX file could see the classes and use them in sort of references rather than loading those classes and having limitations in how you can use them. They will go and use reflection to find the boot class path and modify it to include their secondary DEX files. So this is kind of hack, but a necessary one for them. >What we're doing in L is in runtime we will have a native support for multi-DEX. All your DEX files that you have in your app will get dexopt-ed or compiled by us and collapsed into a single .oat file which is a binary file we generate.. ..And we have a support library on top of that.. ..if you have multi-DEX it will work on older releases of Android back to Ice Cream Sandwich.. will work on Gingerbread too, but we're only validating back to ICS. So once you have that, than that free's you in the future to do the DEX byte code change, and then something that partitions it and runs on the existing [devices].. One interesting note is that he mentions that they'll hopefully have the DEX revision by M. Well, we're on N so it looks like they missed that target date. Perhaps O will finally get the long awaited DEX file format revision.

hota_mazi 10y ago

There is no practical difference between single and multidex. Recent versions of Android are even better at creating these dex files effectively so users won't even notice the difference. Most of these methods are typically contributed by dependencies, not by the app itself.

therealasdf 10y ago
Exactly, the only difference developers might notice is slower build times.

whatever_dude 10y ago

> Do Android apps regularly encompass over 65.3k methods? No. I've built huge applications that didn't come nowhere near that. I don't dispute some apps might run into that, but it's not a regular occurrence at all. The real problem here is that it depends on what kind of libraries you're using. If you're liberal with adding a lot of all-purpose libraries when you only need a feature or two, you can run into that limit fast. For example, look at the comparison table ("Replacing existing libraries") here: http://jeroenmols.com/blog/2016/05/06/methodcount/ http://jeroenmols.com/blog/2016/05/06/methodcount/ If you just need to load an image, do you want a library with 12k methods (!) or 800?

ronocod 10y ago
It depends, Picasso is smallest but doesn't support GIFs. Glid is larger and does support GIFs. Fresco is larger again but has more advanced memory management and better transition handling.