5 ms·

This is one of those Android design decisions that Google probably wishes they could get to do over again. The DEX header file format allocates only 2 bytes to

This is one of those Android design decisions that Google probably wishes they could get to do over again. The DEX header file format allocates only 2 bytes to hold the number of methods the app can reference. This was a very short sighted decision that has come back to bite them repeatedly. It would be interesting to know why they only allocated 2 bytes instead of 4 which would have given them the ability to reference over 4 billion methods. If their reasoning was for cost saving measures than it was a pretty silly decision. There has been some talk about revising the DEX header file format to correct some of the early design mistakes, but I'm not sure when they'll get around to it.

kuschku 10y ago
It's not that easy, and not just one field. DEX uses a method table, listing every method with an ID, and using in the rest of the file only that ID to refer to it. This ID is limited to 2 bytes. Obviously, thats an issue, but short of using varint or straightaway uint32 there's few alternatives. And those use either more CPU or memory.
- shortstuffsushi 10y ago
  You make it sound like it is as simple as that, only a method table instead of a header field :) That said, you mention using a varint or uint32 which "use more CPU or memory" -- 16 bits vs 32 bits in applications that use megabytes seems trivial. Can you explain why that's a concern?
  - kuschku 10y ago
    It's not a concern anymore, but when Android was created, the average application was less than 20kiB. Back then, resource restrictions were a lot more problematic.
- bitmapbrother 10y ago
  Thanks for the correction. Here's a transcript from the Android Developer podcast, courtesy of http://blog.osom.info/2014/10/multi-dex-to-rescue-from-infamous-65536.html http://blog.osom.info/2014/10/multi-dex-to-rescue-from-infam..., talking about the issue with an Android engineer: >Tor: So [regarding] the infamous 64k method [issue].. I understand that it is a dex file format limitation.. which is a short integer. Do you have plans to address this somehow, by either change the dex format or some other way? >Anwar: So we've talked about revving DEX, so that limitation doesn't exist anymore. And there's a couple of reasons we haven't done that: one, there're other things we would like to do better as well, including supporting new language features. The other reason is - it doesn't help us with devices still in the field. It's hard to go and say "by the way, we're going to upgrade your runtime.." So what do we do in order to address that? We will do the DEX byte code [change], but I think there's sort of building block that we needed first - what we're calling multi-DEX. And the idea is.. here's something people were doing - they were breaking up their files into multiple DEX files (each of which exceeds the 64k limit), the main classes in DEX file could see the classes and use them in sort of references rather than loading those classes and having limitations in how you can use them. They will go and use reflection to find the boot class path and modify it to include their secondary DEX files. So this is kind of hack, but a necessary one for them. >What we're doing in L is in runtime we will have a native support for multi-DEX. All your DEX files that you have in your app will get dexopt-ed or compiled by us and collapsed into a single .oat file which is a binary file we generate.. ..And we have a support library on top of that.. ..if you have multi-DEX it will work on older releases of Android back to Ice Cream Sandwich.. will work on Gingerbread too, but we're only validating back to ICS. So once you have that, than that free's you in the future to do the DEX byte code change, and then something that partitions it and runs on the existing [devices].. One interesting note is that he mentions that they'll hopefully have the DEX revision by M. Well, we're on N so it looks like they missed that target date. Perhaps O will finally get the long awaited DEX file format revision.
  - pjmlp 10y ago
    Available on 0.x% of devices, thanks to them not forcing OEMs to update as part of Play Store contract.
    - bitmapbrother 10y ago
      Yeah, why they don't enforce this is baffling. I can give them a break for letting their OEM's take their time with OS updates, but not requiring their OEM's to issue security patches is negligent.
      - izacus 10y ago
        Microsoft enforced that and OEMs decided they'd rather build Android phones than Windows Phone ones. The rest is history.
        pjmlp 10y ago
        I think having an OS available for free was more relevant.
        MichaelGG 10y ago
        Yeah, it's not like Samsung can go make iOS phones. And I highly doubt any of these hardware OEMs could possibly pull off a software project that doesn't suck, let alone a whole OS and dev ecosystem.
        izacus 10y ago
        Samsung already releases phones running Tizen to have a side bet against Android. The quality of software is not very important when your only competitor builds phones that are significantly more expensive.
        pjmlp 10y ago
- kartickv 10y ago
  So why can't Google create a new version of the dex format, update Android Studio to generate it, and update the store to accept and distribute it to new devices? Seems easy enough for Google to be able to do.
  - pjmlp 10y ago
    Because of their clever idea of not forcing OEMs to update devices, almost no one will get that.
    - kartickv 10y ago
      New phones will get it. Surely you're not arguing that Google shouldn't make any improvements to core Android because of the update situation?
      - pjmlp 10y ago
        No, I am arguing that Google doesn't have the guts to improve the situation. Most people around the world are on pre-paid, not contracts, and they only update their devices when the current one dies. Even now, the majority of the devices being sold on the 300 € border are Android 5 or 6, with no planned upgrade to 7.
    - lmm 10y ago
      The best time to ship support for an updated format was five years ago. The second-best time is today.
hota_mazi 10y ago
> This is one of those Android design decisions that Google probably wishes they could get to do over again. Doubtful. This decision is what allowed Android to perform reasonably even in the early days circa 2008. Without such a restriction, it's possible that Android couldn't have competed with iOS on performance and die as a consequence. It's easy to make fun of technical decisions made ten years ago, try to put things in perspective. Besides, even today, this 65k limit is barely a minor annoyance that only a very tiny percentage of Android apps ever hit (and there are easy workarounds).
- wasyl 10y ago
  > this 65k limit is barely a minor annoyance that only a very tiny percentage of Android apps ever hit This is not true, you can easily hit this limit because of a dependency that you _need_ that pulls e.g. Gauva, of when you use 2 or 3 libraries from Play Services, which are huge > (and there are easy workarounds) Easy maybe, but the issue is still more than _a minor annoyance_ imo
  - crystalPalace 10y ago
    I've hit the limit when pulling in all Play services. The key is to only pull in what you will be using. Many tutorials(including one from a third party analytics API) are a bit lazy and just have you import Play in its entirety.

kuschku 10y ago

It's not that easy, and not just one field. DEX uses a method table, listing every method with an ID, and using in the rest of the file only that ID to refer to it. This ID is limited to 2 bytes. Obviously, thats an issue, but short of using varint or straightaway uint32 there's few alternatives. And those use either more CPU or memory.

shortstuffsushi 10y ago
You make it sound like it is as simple as that, only a method table instead of a header field :) That said, you mention using a varint or uint32 which "use more CPU or memory" -- 16 bits vs 32 bits in applications that use megabytes seems trivial. Can you explain why that's a concern?
- kuschku 10y ago
  It's not a concern anymore, but when Android was created, the average application was less than 20kiB. Back then, resource restrictions were a lot more problematic.
bitmapbrother 10y ago
Thanks for the correction. Here's a transcript from the Android Developer podcast, courtesy of http://blog.osom.info/2014/10/multi-dex-to-rescue-from-infamous-65536.html http://blog.osom.info/2014/10/multi-dex-to-rescue-from-infam..., talking about the issue with an Android engineer: >Tor: So [regarding] the infamous 64k method [issue].. I understand that it is a dex file format limitation.. which is a short integer. Do you have plans to address this somehow, by either change the dex format or some other way? >Anwar: So we've talked about revving DEX, so that limitation doesn't exist anymore. And there's a couple of reasons we haven't done that: one, there're other things we would like to do better as well, including supporting new language features. The other reason is - it doesn't help us with devices still in the field. It's hard to go and say "by the way, we're going to upgrade your runtime.." So what do we do in order to address that? We will do the DEX byte code [change], but I think there's sort of building block that we needed first - what we're calling multi-DEX. And the idea is.. here's something people were doing - they were breaking up their files into multiple DEX files (each of which exceeds the 64k limit), the main classes in DEX file could see the classes and use them in sort of references rather than loading those classes and having limitations in how you can use them. They will go and use reflection to find the boot class path and modify it to include their secondary DEX files. So this is kind of hack, but a necessary one for them. >What we're doing in L is in runtime we will have a native support for multi-DEX. All your DEX files that you have in your app will get dexopt-ed or compiled by us and collapsed into a single .oat file which is a binary file we generate.. ..And we have a support library on top of that.. ..if you have multi-DEX it will work on older releases of Android back to Ice Cream Sandwich.. will work on Gingerbread too, but we're only validating back to ICS. So once you have that, than that free's you in the future to do the DEX byte code change, and then something that partitions it and runs on the existing [devices].. One interesting note is that he mentions that they'll hopefully have the DEX revision by M. Well, we're on N so it looks like they missed that target date. Perhaps O will finally get the long awaited DEX file format revision.
- pjmlp 10y ago
  Available on 0.x% of devices, thanks to them not forcing OEMs to update as part of Play Store contract.
  - bitmapbrother 10y ago
    Yeah, why they don't enforce this is baffling. I can give them a break for letting their OEM's take their time with OS updates, but not requiring their OEM's to issue security patches is negligent.
    - izacus 10y ago
      Microsoft enforced that and OEMs decided they'd rather build Android phones than Windows Phone ones. The rest is history.
      - pjmlp 10y ago
        I think having an OS available for free was more relevant.
        MichaelGG 10y ago
        Yeah, it's not like Samsung can go make iOS phones. And I highly doubt any of these hardware OEMs could possibly pull off a software project that doesn't suck, let alone a whole OS and dev ecosystem.
        izacus 10y ago
        Samsung already releases phones running Tizen to have a side bet against Android. The quality of software is not very important when your only competitor builds phones that are significantly more expensive.
        pjmlp 10y ago
kartickv 10y ago
So why can't Google create a new version of the dex format, update Android Studio to generate it, and update the store to accept and distribute it to new devices? Seems easy enough for Google to be able to do.
- pjmlp 10y ago
  Because of their clever idea of not forcing OEMs to update devices, almost no one will get that.
  - kartickv 10y ago
    New phones will get it. Surely you're not arguing that Google shouldn't make any improvements to core Android because of the update situation?
    - pjmlp 10y ago
      No, I am arguing that Google doesn't have the guts to improve the situation. Most people around the world are on pre-paid, not contracts, and they only update their devices when the current one dies. Even now, the majority of the devices being sold on the 300 € border are Android 5 or 6, with no planned upgrade to 7.
  - lmm 10y ago
    The best time to ship support for an updated format was five years ago. The second-best time is today.

hota_mazi 10y ago

> This is one of those Android design decisions that Google probably wishes they could get to do over again. Doubtful. This decision is what allowed Android to perform reasonably even in the early days circa 2008. Without such a restriction, it's possible that Android couldn't have competed with iOS on performance and die as a consequence. It's easy to make fun of technical decisions made ten years ago, try to put things in perspective. Besides, even today, this 65k limit is barely a minor annoyance that only a very tiny percentage of Android apps ever hit (and there are easy workarounds).

wasyl 10y ago
> this 65k limit is barely a minor annoyance that only a very tiny percentage of Android apps ever hit This is not true, you can easily hit this limit because of a dependency that you _need_ that pulls e.g. Gauva, of when you use 2 or 3 libraries from Play Services, which are huge > (and there are easy workarounds) Easy maybe, but the issue is still more than _a minor annoyance_ imo
- crystalPalace 10y ago
  I've hit the limit when pulling in all Play services. The key is to only pull in what you will be using. Many tutorials(including one from a third party analytics API) are a bit lazy and just have you import Play in its entirety.