6 ms·

Facebook Does It Again. Cheating Dalvik

90 points by redDragon 12y ago

stickydink 12y ago
We have this problem. We have a large, mostly-Java game for Android. It won't run on gingerbread, because we just have too much code. While the official solution looks quite brief, retrofitting it is a nightmare. Unless Facebook got around to rewriting everything again, I can understand why they'd do it their way.
- higherpurpose 12y ago
  If it's too much of a hassle to make your app work on Gingerbread, just don't bother with it anymore. I bet Google themselves will end all support for Gingerbread next year when version M comes out (as they should). It just doesn't make sense to support it longer than that, and by then the market share of Gingerbread should be under 15-20 percent, too. Gingerbread is the XP of Android - an old popular version, right before major architectural changes. Android L seems to be the next major architectural change, but I don't think KitKat will be the new Gingerbread, because Google is now requiring OEMs to only launch devices that are 2 versions and/or 9 months old, which will soon mean (going by the new release cycle) just 1 version behind. So the transition for new versions should happen a lot smoother in the future.
  - ZoFreX 12y ago
    > Google is now requiring OEMs to only launch devices that are 2 versions and/or 9 months old Are they requiring anything from carriers or phone shops, though? Because everywhere in my town is still selling Gingerbread devices to anyone that doesn't know better.
  - gabemart 12y ago
    > by then the market share of Gingerbread should be under 15-20 percent, too According to the Android dashboard [1], among users of Google Play Gingerbread has a market share of 13.5% [1] https://developer.android.com/about/dashboards/index.html https://developer.android.com/about/dashboards/index.html
    - ryandvm 12y ago
      I'd like to see what percentage of app sales come from Gingerbread. My guess is that people that are willing to buy apps are not* using a 3 year old phone. (*edited)
      - judk 12y ago
        People are not willing, right? Gingerbread phones are old and underpowered and can't handle the demands of flashy new apps. I should buy one.
      - drzaiusapelord 12y ago
        A gingerbread phone can by of any age. That's why google is moving towards a new contract for its gapps. Going forward you can only release the current Android or go as far as 2 versions back. So we're on 4.4 now, thus making 4.2 the oldest version for a phone released today (assuming the new contracts are in place). A few manufacturers went a bit overboard mass producing GB-only phones lately because of how cheap their components are. I'm not sure what this means for the budget phone market now that they need to be 4.2 or higher on release date. From what I've seen, the new budget phone is the low-end Nokia Windows Phone with carriers like Cricket literally giving them away with rebates or charging next to nothing for them like $50, and that's on a no contract plan! Getting Android out of the GB-era budget ghetto is a good thing. If the Moto E can handle 4.4 then so can everyone else. Supposedly 4.4 uses less ram than previous versions of the 4.x line. The head of google claimed that it can run on 512mb devices. They really, really want to kill GB. I imagine there will always be a android budget phone out there, just not no name junk running a near 4 year old OS. The real question is how many more GB phones are there in the supply chain that will be sold this year and next? Millions? How long will these things be in play? It really does look like there is a real abandoned version of Android that can't survive fragmentation now. Many apps simply will not work. Shame really because I'm sure no one explained to the budget phone buyers that they were buying an ancient OS that can't be upgraded on that system with so little ram.
- tormeh 12y ago
  I don't think anyone with a phone running 2.x can run games. I have an HTC One or whatever (4 years old; got it from a friend) and I could never ever imagine playing a game on it. Even the home screen lags!
  - CalRobert 12y ago
    I just played through Swordigo on a Galaxy S (the first one!) and it ran very smoothly. I am, it may be worth noting, using Cyanogenmod 11 nightlies.
  - redDragon 12y ago
    Oh you will be surprised: Not every game requires higher Android version, they are mostly satisfied with c++, http://imgur.com/iX4dcL5 http://imgur.com/iX4dcL5 Card Games: Gingerbread (14%)
  - dagw 12y ago
    I've got an old cheap 4 year old Acer phone running 2.3 as a backup phone and it plays lots of games fine. Sure not games using a lot of 3d graphics or games which require very precise timing, but it plays lots of other games.
  - speeder 12y ago
    I own a Xperia Play, that don't support anything after 2.3, and I have lots of gaming here, including shooters, platformers and racing
  - pjmlp 12y ago
    If you code in C++, which most games do, the Android version is not that important, given the little exposure of Android APIs to the NDK. Everything that matters, besides the existing graphics, sensors and audio APIs, can be built with standard C++ libraries.
    - tormeh 12y ago
      The problem lies with the hardware, I think. Truly, there's never a smooth moment with my phone. Sometimes it hangs for a minute. Random shutdowns and similar bugs in software as well.
- redDragon 12y ago
  There is a way: https://github.com/creativepsyco/secondary-dex-gradle https://github.com/creativepsyco/secondary-dex-gradle The new gradle build system really simplifies a lot of things in terms of the build cycle
  - zBard 12y ago
    When did it come out ? I suspect it wasn't adopted widely a year ago.
- gabemart 12y ago
  I'm in the end stages of developing my first Android app. Throughout development I used the Android compatibility libraries to get it working on Gingerbread. However, now that I'm almost finished, I realize that many Gingerbread devices have a hard limit on the number of simultaneous active MediaPlayer objects that means my app will not (and cannot) function properly. (All devices tested on ICS++ seem to work fine). It works on some Gingerbread devices (and, annoyingly, all emulated Gingerbread devices I've tried) but not all. So now I'm in the awkward position of either changing the minimum API level of my app to exclude all Gingerbread devices (even though it works on some) or annoying some Gingerbread users by getting them to download my (large) apk only to give them a "sorry, your device is not supported" message when they first open it. I know nothing about iOS development, but I'm envious of the narrow range of hardware and software in that ecosystem.
  - runeks 12y ago
    IMHO, Google (Android) seems to be at fault here, as they lack a proper way to determine whether an app is compatible with a particular device, without false negatives. Instead of getting angry at developers (eg. calling it "cheating" and a "horrible hack"), who want to maximize the compatibility of their app, the Android team should add a standard way for apps to define what they need to be compatible with a device, without getting false negatives.
  - bane 12y ago
    I wouldn't be too worried. Gingerbread devices are something like <13% of all Android devices (out of something like 800 million). I'm going to go out on a limb here and assume that people still running 2.3 are probably not people who will be buying apps anyways.
- CmonDev 12y ago
  Consider switching to Xamarin + MonoGame. You will have to pay, but features might be worth it (better yet familiar language more mainstream in gamedev, decent cross-platform support). Alternatively try using Unity3d if you don't mind scripting instead of programming.
jhgg 12y ago
I'm a bit confused by this article, the author mentions: While the official standard way to to fix this problem is described in the offical Android Blog. Seems like Facebook missed it. What a waste. However, in the post linked by facebook, they explain why they couldn't use it (vaguely) After a bit of panic, we realized that we could work around this problem by breaking our app into multiple dex files, using the technique described here (http://android-developers.blogspot.com/2011/07/custom-class-loading-in-dalvik.html), which focuses on using secondary dex files for extension modules, not core parts of the app.
- redDragon 12y ago
  Yes i sort of forgot about it, the facebook article was old, i read it a long time ago, i rediscovered the issue when running FB on one of the Gingerbread devices.
lambada 12y ago
I don't see how facebook cheated Dalvik 'again'? Seems like it's still just the one 'patch'/'hack'. Am I missing something?
- skrebbel 12y ago
  What they do "again" is pull some nice engineering feat that's also a bit of a hack, and therefore fun to read about. I assume the author is impressed, and likens it to some of Facebook's similar bend-the-world-to-my-turd undertakings such as patching git to be able to deal with Facebook's single gigantic (!) source code repository.
  - lambada 12y ago
    Oh, I see. I assumed the 'it' in the post title referred to the 'Cheating Dalvik' that followed. Not that it referred to a more generic concept of bad-practise.
tinco 12y ago
There's a weird and arbitrary limitation in Dalvik, and the Facebook guys needed to work around it. The author makes it sound like it's some horrible sin. How about shaming the Google developers who thought 65k methods was enough for anyone? Wouldn't be good proper either, because they probably had a good reason for it. Software development isn't always stacking neat abstractions on top of each other. Sometimes you have real hardware and real legacy and you just got to deal with it, one way or another.
- ZoFreX 12y ago
  The bug they hit isn't the 65k limit, but the smaller LinearAlloc buffer size on older versions of the OS.
  - redDragon 12y ago
    Yep i think the blog post referred the stuff as being LinearAlloc == dex method count, I have made it clear now.
- Millennium 12y ago
  2^16 counts as "weird and arbitrary"?
  - rwallace 12y ago
    Since we stopped using 16-bit CPUs, yes, it does.
    - warfangle 12y ago
      Since Dalvik Executables run on a virtual machine, and the Dalvik virtual machine's instruction set is 16-bit, it makes perfect sense.
- drzaiusapelord 12y ago
  I imagine the devs and management don't care much as they're going to migrate to ART soon, which I imagine does not have Dalvik's circa 2003 design decisions. Google hasn't been investing in Dalvik because its days are numbered. edit: why is this being downvoted? Do we really think ART will have these limitations? Android has been hot to get off Dalvik for many reasons and they're finally doing it in L. This is very good news and makes Andy Rubin-era Android decisions, which made sense for a 2003 phone, replaced with more modern thinking. edit2: Apparantly, I'm wrong. Looks like the 16 bit limitation in still there in the current version of ART. Maybe this will change in the final production version.
praseodym 12y ago
Note that the Facebook blog post about this hack is from 4 March 2013, i.e. more than a year ago.
- redDragon 12y ago
  Yes and still FB relies on this patch to run its app, and sometimes it does fail as happened on my device.
__Joker 12y ago
I understand, Dalvik uses 2 bytes to select a method found in the list at the start of the dex file and then to execute it. Is there any particular reason it is limited to 2 bytes ?
- ntlve 12y ago
  This is just me guessing but one of the initial goals of Dalvik was to use as little space as possible (due to memory constraints of hardware at that time). Perhaps they thought saving a a byte or two on method identifiers was a good idea. This would not be a problem if it weren't for the fact that dx also squashes all classes into one single class, meaning that instead of being 65K methods per class you now get 65K per apk.
  - judk 12y ago
    Ah, mobile. Recapitulating all the horribly constrained hacks of the early PC era, in handheld size containers.
smanuel 12y ago
Zuckerberg 2012: Our biggest mistake was betting too much on HTML5 Zuckerberg 2014: Our biggest mistake was betting too much on Dalvik
- ProAm 12y ago
  Sounds like his gambling problem is the issue.
rurban 12y ago
Maybe it would have helped missing out in the mandatory spying features, and providing only the required customer features. Those features really take a lot of memory: * “Record audio with the microphone … at any time without your confirmation” * Take videos and photos using the camera * Access the phone’s call log * Read data about contacts stored on the phone, “including the frequency with which you’ve called, emailed or communicated in other ways with specific individuals”
- kalleboo 12y ago
  > * “Record audio with the microphone … at any time without your confirmation” There's no way in Android to record audio with the microphone at only certain times with your confirmation. Facebook needs microphone access for video and voice calling. > * Take videos and photos using the camera The Facebook app lets you take photos and videos to post on your feed. How else would they be able to do that without camera access? I can't speak for the other two (I use iOS myself) but Androids permissions are generally way too broad (stuff like to be able to pause music when the user is making a call you have to ask for access to their phone number and the phone numbers of everyone calling them) and users have to accept the kitchen sink. The "Facebook spying" stuff is an Android bug, not Facebook.
  - danielweber 12y ago
    The Facebook app lets you take photos and videos to post on your feed. How else would they be able to do that without camera access? Isn't the point of the Android app model of breaking everything up into Activities that you can call another Activity which does the video/photo recording, and then you just deal with the result? (That's not a rhetorical question; I may be misunderstanding the intent of the design, or how people use the design in reality, or both.)
    - aguyhere123 12y ago
      Yes, you're right that activities are broken up like that. Permissions are defined in the application manifest, and apply to all activities within the application. Facebook would have to break their application into a calling app and newsfeed app to split the permissions apart. It would still request permission to record and take videos for both, though, if they implemented video recording on the newsfeed app.
bhouston 12y ago
I understand blackberry messenger also has the 65k method hack in its android app.
robmccoll 12y ago
On the 65K methods restriction, how many lines of code per method do these projects have on average? That just seems like a lot of code. Is this a result of the Java acessor-and-mutator-methods-for-everything approach? For reference, Tomcat and Maven are both 1M lines of code (would be ~15 lines of code per method @ 65K methods). Lucene is <500K.
- jug6ernaut 12y ago
  I used to think this was a non issue. 65k methods is a lot right? Well I ran into the 65k method limit with my one developer game I'm working on. Can't remember if it while i was using gradle or maven(recent switched) but it would print out all the methods for each package. Method counts add up quick when you are importing libraries. For example google guice by it self(if memory servers me) had ~6k methods. Thats one library. Starting adding different Apache libraries and its possible. Granted the majority of these get stripped out by Proguard, but having to run proguard for every build is a major pain. I have since refactored and am now well under the limit, but this issue is a lot more prevalent then i thought.
spo81rty 12y ago
I wonder if this 65k limit is part of the reason they made messenger its own app?
- jug6ernaut 12y ago
  I'm sure it was a consideration. Also from a team/project structure standpoint it probably a lot easier.
- dragonwriter 12y ago
  They made Messenger its own app and made the separate app required for messaging on iOS first, and later on Android. So it really doesn't make sense if any significant part of the motivation was working around an Android platform issue.
coldcode 12y ago
Google early on made decisions about code size that made sense at the time, but now seem to be a serious limitation (at least in the interim). FB's decision on getting around this architectural limitation isn't all that bad. What I don't understand is how their solution can affect other applications. As an iOS dev screwing up my app is one thing but being able to screw up other apps seems like an even worse architectural problem.
- dubcanada 12y ago
  I'm confused on how it can screw up another application? Aren't all the applications sandboxed?
  - danielweber 12y ago
    Given the state of Operating Systems 20 years ago, it would be very ironic if today the Apple OS doesn't let one app crash another, but the non-Apple OS does.
  - kevb 12y ago
    This hack can't screw up another application, the author is wrong about that. It's changing the local process only and has no influence on any other apps (which are not just sandboxed into their own processes but also their own user ids)
yincrash 12y ago
This article seems to conflate the limit on number of methods vs the amount of memory required for one of the dexopt steps.
j_s 12y ago
Below are some choice quotes from the last time this came up almost 1.5 years ago; interesting how the peanut gallery has mellowed over time. https://news.ycombinator.com/item?id=5321634 https://news.ycombinator.com/item?id=5321634 > I can't believe an app requires 8M RAM just for method names! - https://news.ycombinator.com/item?id=5323153 https://news.ycombinator.com/item?id=5323153 > They bloated the app so badly, they would have had to monkey patch the OS to let it run at all -- https://news.ycombinator.com/item?id=5323930 https://news.ycombinator.com/item?id=5323930 > Basically these engineers decided to be really clever, painted themselves into a corner, smashed a hole in the wall so as to escape from said corner, and then bragged about how great they are. -- https://news.ycombinator.com/item?id=5322286 https://news.ycombinator.com/item?id=5322286 Also, the bug that was filed was also referenced: Dexopt fails with "LinearAlloc exceeded" for deep interface hierarchies https://code.google.com/p/android/issues/detail?id=22586 https://code.google.com/p/android/issues/detail?id=22586
- krschultz 12y ago
  I don't think we've mellowed, just that we have become more sympathetic to Facebook's problem. I think the reality is that the 65k method limit problem has turned into something Android Developers put squarely at Google's feet. At the time I thought, what the hell is Facebook doing? Now, I've run into the 65k limit myself on an app written by 4 people over 18 months. Sure it's not a throwaway weekend project, but that is a pretty standard app. Google either needs to blow away the 65k method limit, or figure out a way to make Google Play Services a lot lighter. As it stands right now, if you want just push notifications you are giving up ~20k methods of your 65k limit for that alone. Just look at this post from Jake Wharton (leading Android open source developer): http://jakewharton.com/play-services-is-a-monolith/ http://jakewharton.com/play-services-is-a-monolith/
  - emil10001 12y ago
    Slimming down Play Services just punts on the problem for a while. My view is that Google needs to do two things with this: 1) Split up the Play Services client libraries. 2) Figure out some solution for developers hitting the 65K limit. I'm sort of assuming that the solution would come in the form of a framework or tooling that we'd have to implement. Such a solution should allow us to build fully working apps with multiple dex files, with some decent documentation. It should also work fine for debug builds without proguard, and also without bumping build times up beyond a two minutes. What's more, we should be able to split off pieces that are defined in the manifest into secondary dex files, and fire intents at them. That, to me, seems like a reasonable response to this problem by Google. The company I work for has been hitting the limit for the last few months, and some of our other dependencies are becoming more and more expensive with newer releases (things that our users actually like.) So far, we've been able to build with a stripped down Play Services jar, but I'm not terribly happy about that approach. Regardless, I'm not going to blame Facebook for a problem that Google caused. If Facebook can find a solution, and tell the community how they did it, I'm all ears.
d0vs 12y ago
Mirror?

90 points by redDragon 12y ago

We have this problem. We have a large, mostly-Java game for Android. It won't run on gingerbread, because we just have too much code. While the official solution looks quite brief, retrofitting it is a nightmare. Unless Facebook got around to rewriting everything again, I can understand why they'd do it their way.

higherpurpose 12y ago
If it's too much of a hassle to make your app work on Gingerbread, just don't bother with it anymore. I bet Google themselves will end all support for Gingerbread next year when version M comes out (as they should). It just doesn't make sense to support it longer than that, and by then the market share of Gingerbread should be under 15-20 percent, too. Gingerbread is the XP of Android - an old popular version, right before major architectural changes. Android L seems to be the next major architectural change, but I don't think KitKat will be the new Gingerbread, because Google is now requiring OEMs to only launch devices that are 2 versions and/or 9 months old, which will soon mean (going by the new release cycle) just 1 version behind. So the transition for new versions should happen a lot smoother in the future.
- ZoFreX 12y ago
  > Google is now requiring OEMs to only launch devices that are 2 versions and/or 9 months old Are they requiring anything from carriers or phone shops, though? Because everywhere in my town is still selling Gingerbread devices to anyone that doesn't know better.
- gabemart 12y ago
  > by then the market share of Gingerbread should be under 15-20 percent, too According to the Android dashboard [1], among users of Google Play Gingerbread has a market share of 13.5% [1] https://developer.android.com/about/dashboards/index.html https://developer.android.com/about/dashboards/index.html
  - ryandvm 12y ago
    I'd like to see what percentage of app sales come from Gingerbread. My guess is that people that are willing to buy apps are not* using a 3 year old phone. (*edited)
    - judk 12y ago
      People are not willing, right? Gingerbread phones are old and underpowered and can't handle the demands of flashy new apps. I should buy one.
    - drzaiusapelord 12y ago
      A gingerbread phone can by of any age. That's why google is moving towards a new contract for its gapps. Going forward you can only release the current Android or go as far as 2 versions back. So we're on 4.4 now, thus making 4.2 the oldest version for a phone released today (assuming the new contracts are in place). A few manufacturers went a bit overboard mass producing GB-only phones lately because of how cheap their components are. I'm not sure what this means for the budget phone market now that they need to be 4.2 or higher on release date. From what I've seen, the new budget phone is the low-end Nokia Windows Phone with carriers like Cricket literally giving them away with rebates or charging next to nothing for them like $50, and that's on a no contract plan! Getting Android out of the GB-era budget ghetto is a good thing. If the Moto E can handle 4.4 then so can everyone else. Supposedly 4.4 uses less ram than previous versions of the 4.x line. The head of google claimed that it can run on 512mb devices. They really, really want to kill GB. I imagine there will always be a android budget phone out there, just not no name junk running a near 4 year old OS. The real question is how many more GB phones are there in the supply chain that will be sold this year and next? Millions? How long will these things be in play? It really does look like there is a real abandoned version of Android that can't survive fragmentation now. Many apps simply will not work. Shame really because I'm sure no one explained to the budget phone buyers that they were buying an ancient OS that can't be upgraded on that system with so little ram.
tormeh 12y ago
I don't think anyone with a phone running 2.x can run games. I have an HTC One or whatever (4 years old; got it from a friend) and I could never ever imagine playing a game on it. Even the home screen lags!
- CalRobert 12y ago
  I just played through Swordigo on a Galaxy S (the first one!) and it ran very smoothly. I am, it may be worth noting, using Cyanogenmod 11 nightlies.
- redDragon 12y ago
  Oh you will be surprised: Not every game requires higher Android version, they are mostly satisfied with c++, http://imgur.com/iX4dcL5 http://imgur.com/iX4dcL5 Card Games: Gingerbread (14%)
- dagw 12y ago
  I've got an old cheap 4 year old Acer phone running 2.3 as a backup phone and it plays lots of games fine. Sure not games using a lot of 3d graphics or games which require very precise timing, but it plays lots of other games.
- speeder 12y ago
  I own a Xperia Play, that don't support anything after 2.3, and I have lots of gaming here, including shooters, platformers and racing
- pjmlp 12y ago
  If you code in C++, which most games do, the Android version is not that important, given the little exposure of Android APIs to the NDK. Everything that matters, besides the existing graphics, sensors and audio APIs, can be built with standard C++ libraries.
  - tormeh 12y ago
    The problem lies with the hardware, I think. Truly, there's never a smooth moment with my phone. Sometimes it hangs for a minute. Random shutdowns and similar bugs in software as well.
redDragon 12y ago
There is a way: https://github.com/creativepsyco/secondary-dex-gradle https://github.com/creativepsyco/secondary-dex-gradle The new gradle build system really simplifies a lot of things in terms of the build cycle
- zBard 12y ago
  When did it come out ? I suspect it wasn't adopted widely a year ago.
gabemart 12y ago
I'm in the end stages of developing my first Android app. Throughout development I used the Android compatibility libraries to get it working on Gingerbread. However, now that I'm almost finished, I realize that many Gingerbread devices have a hard limit on the number of simultaneous active MediaPlayer objects that means my app will not (and cannot) function properly. (All devices tested on ICS++ seem to work fine). It works on some Gingerbread devices (and, annoyingly, all emulated Gingerbread devices I've tried) but not all. So now I'm in the awkward position of either changing the minimum API level of my app to exclude all Gingerbread devices (even though it works on some) or annoying some Gingerbread users by getting them to download my (large) apk only to give them a "sorry, your device is not supported" message when they first open it. I know nothing about iOS development, but I'm envious of the narrow range of hardware and software in that ecosystem.
- runeks 12y ago
  IMHO, Google (Android) seems to be at fault here, as they lack a proper way to determine whether an app is compatible with a particular device, without false negatives. Instead of getting angry at developers (eg. calling it "cheating" and a "horrible hack"), who want to maximize the compatibility of their app, the Android team should add a standard way for apps to define what they need to be compatible with a device, without getting false negatives.
- bane 12y ago
  I wouldn't be too worried. Gingerbread devices are something like <13% of all Android devices (out of something like 800 million). I'm going to go out on a limb here and assume that people still running 2.3 are probably not people who will be buying apps anyways.
CmonDev 12y ago
Consider switching to Xamarin + MonoGame. You will have to pay, but features might be worth it (better yet familiar language more mainstream in gamedev, decent cross-platform support). Alternatively try using Unity3d if you don't mind scripting instead of programming.

jhgg 12y ago

I'm a bit confused by this article, the author mentions: While the official standard way to to fix this problem is described in the offical Android Blog. Seems like Facebook missed it. What a waste. However, in the post linked by facebook, they explain why they couldn't use it (vaguely) After a bit of panic, we realized that we could work around this problem by breaking our app into multiple dex files, using the technique described here (http://android-developers.blogspot.com/2011/07/custom-class-loading-in-dalvik.html), which focuses on using secondary dex files for extension modules, not core parts of the app.

redDragon 12y ago
Yes i sort of forgot about it, the facebook article was old, i read it a long time ago, i rediscovered the issue when running FB on one of the Gingerbread devices.

lambada 12y ago

I don't see how facebook cheated Dalvik 'again'? Seems like it's still just the one 'patch'/'hack'. Am I missing something?

skrebbel 12y ago
What they do "again" is pull some nice engineering feat that's also a bit of a hack, and therefore fun to read about. I assume the author is impressed, and likens it to some of Facebook's similar bend-the-world-to-my-turd undertakings such as patching git to be able to deal with Facebook's single gigantic (!) source code repository.
- lambada 12y ago
  Oh, I see. I assumed the 'it' in the post title referred to the 'Cheating Dalvik' that followed. Not that it referred to a more generic concept of bad-practise.

tinco 12y ago

There's a weird and arbitrary limitation in Dalvik, and the Facebook guys needed to work around it. The author makes it sound like it's some horrible sin. How about shaming the Google developers who thought 65k methods was enough for anyone? Wouldn't be good proper either, because they probably had a good reason for it. Software development isn't always stacking neat abstractions on top of each other. Sometimes you have real hardware and real legacy and you just got to deal with it, one way or another.

ZoFreX 12y ago
The bug they hit isn't the 65k limit, but the smaller LinearAlloc buffer size on older versions of the OS.
- redDragon 12y ago
  Yep i think the blog post referred the stuff as being LinearAlloc == dex method count, I have made it clear now.
Millennium 12y ago
2^16 counts as "weird and arbitrary"?
- rwallace 12y ago
  Since we stopped using 16-bit CPUs, yes, it does.
  - warfangle 12y ago
    Since Dalvik Executables run on a virtual machine, and the Dalvik virtual machine's instruction set is 16-bit, it makes perfect sense.
drzaiusapelord 12y ago
I imagine the devs and management don't care much as they're going to migrate to ART soon, which I imagine does not have Dalvik's circa 2003 design decisions. Google hasn't been investing in Dalvik because its days are numbered. edit: why is this being downvoted? Do we really think ART will have these limitations? Android has been hot to get off Dalvik for many reasons and they're finally doing it in L. This is very good news and makes Andy Rubin-era Android decisions, which made sense for a 2003 phone, replaced with more modern thinking. edit2: Apparantly, I'm wrong. Looks like the 16 bit limitation in still there in the current version of ART. Maybe this will change in the final production version.

praseodym 12y ago

Note that the Facebook blog post about this hack is from 4 March 2013, i.e. more than a year ago.

redDragon 12y ago
Yes and still FB relies on this patch to run its app, and sometimes it does fail as happened on my device.

__Joker 12y ago

I understand, Dalvik uses 2 bytes to select a method found in the list at the start of the dex file and then to execute it. Is there any particular reason it is limited to 2 bytes ?

ntlve 12y ago
This is just me guessing but one of the initial goals of Dalvik was to use as little space as possible (due to memory constraints of hardware at that time). Perhaps they thought saving a a byte or two on method identifiers was a good idea. This would not be a problem if it weren't for the fact that dx also squashes all classes into one single class, meaning that instead of being 65K methods per class you now get 65K per apk.
- judk 12y ago
  Ah, mobile. Recapitulating all the horribly constrained hacks of the early PC era, in handheld size containers.

smanuel 12y ago

Zuckerberg 2012: Our biggest mistake was betting too much on HTML5 Zuckerberg 2014: Our biggest mistake was betting too much on Dalvik

ProAm 12y ago
Sounds like his gambling problem is the issue.

rurban 12y ago

Maybe it would have helped missing out in the mandatory spying features, and providing only the required customer features. Those features really take a lot of memory: * “Record audio with the microphone … at any time without your confirmation” * Take videos and photos using the camera * Access the phone’s call log * Read data about contacts stored on the phone, “including the frequency with which you’ve called, emailed or communicated in other ways with specific individuals”

kalleboo 12y ago
> * “Record audio with the microphone … at any time without your confirmation” There's no way in Android to record audio with the microphone at only certain times with your confirmation. Facebook needs microphone access for video and voice calling. > * Take videos and photos using the camera The Facebook app lets you take photos and videos to post on your feed. How else would they be able to do that without camera access? I can't speak for the other two (I use iOS myself) but Androids permissions are generally way too broad (stuff like to be able to pause music when the user is making a call you have to ask for access to their phone number and the phone numbers of everyone calling them) and users have to accept the kitchen sink. The "Facebook spying" stuff is an Android bug, not Facebook.
- danielweber 12y ago
  The Facebook app lets you take photos and videos to post on your feed. How else would they be able to do that without camera access? Isn't the point of the Android app model of breaking everything up into Activities that you can call another Activity which does the video/photo recording, and then you just deal with the result? (That's not a rhetorical question; I may be misunderstanding the intent of the design, or how people use the design in reality, or both.)
  - aguyhere123 12y ago
    Yes, you're right that activities are broken up like that. Permissions are defined in the application manifest, and apply to all activities within the application. Facebook would have to break their application into a calling app and newsfeed app to split the permissions apart. It would still request permission to record and take videos for both, though, if they implemented video recording on the newsfeed app.

bhouston 12y ago

I understand blackberry messenger also has the 65k method hack in its android app.

robmccoll 12y ago

On the 65K methods restriction, how many lines of code per method do these projects have on average? That just seems like a lot of code. Is this a result of the Java acessor-and-mutator-methods-for-everything approach? For reference, Tomcat and Maven are both 1M lines of code (would be ~15 lines of code per method @ 65K methods). Lucene is <500K.

jug6ernaut 12y ago
I used to think this was a non issue. 65k methods is a lot right? Well I ran into the 65k method limit with my one developer game I'm working on. Can't remember if it while i was using gradle or maven(recent switched) but it would print out all the methods for each package. Method counts add up quick when you are importing libraries. For example google guice by it self(if memory servers me) had ~6k methods. Thats one library. Starting adding different Apache libraries and its possible. Granted the majority of these get stripped out by Proguard, but having to run proguard for every build is a major pain. I have since refactored and am now well under the limit, but this issue is a lot more prevalent then i thought.

spo81rty 12y ago

I wonder if this 65k limit is part of the reason they made messenger its own app?

jug6ernaut 12y ago
I'm sure it was a consideration. Also from a team/project structure standpoint it probably a lot easier.
dragonwriter 12y ago
They made Messenger its own app and made the separate app required for messaging on iOS first, and later on Android. So it really doesn't make sense if any significant part of the motivation was working around an Android platform issue.

coldcode 12y ago

Google early on made decisions about code size that made sense at the time, but now seem to be a serious limitation (at least in the interim). FB's decision on getting around this architectural limitation isn't all that bad. What I don't understand is how their solution can affect other applications. As an iOS dev screwing up my app is one thing but being able to screw up other apps seems like an even worse architectural problem.

dubcanada 12y ago
I'm confused on how it can screw up another application? Aren't all the applications sandboxed?
- danielweber 12y ago
  Given the state of Operating Systems 20 years ago, it would be very ironic if today the Apple OS doesn't let one app crash another, but the non-Apple OS does.
- kevb 12y ago
  This hack can't screw up another application, the author is wrong about that. It's changing the local process only and has no influence on any other apps (which are not just sandboxed into their own processes but also their own user ids)

yincrash 12y ago

This article seems to conflate the limit on number of methods vs the amount of memory required for one of the dexopt steps.

j_s 12y ago

Below are some choice quotes from the last time this came up almost 1.5 years ago; interesting how the peanut gallery has mellowed over time. https://news.ycombinator.com/item?id=5321634 https://news.ycombinator.com/item?id=5321634 > I can't believe an app requires 8M RAM just for method names! - https://news.ycombinator.com/item?id=5323153 https://news.ycombinator.com/item?id=5323153 > They bloated the app so badly, they would have had to monkey patch the OS to let it run at all -- https://news.ycombinator.com/item?id=5323930 https://news.ycombinator.com/item?id=5323930 > Basically these engineers decided to be really clever, painted themselves into a corner, smashed a hole in the wall so as to escape from said corner, and then bragged about how great they are. -- https://news.ycombinator.com/item?id=5322286 https://news.ycombinator.com/item?id=5322286 Also, the bug that was filed was also referenced: Dexopt fails with "LinearAlloc exceeded" for deep interface hierarchies https://code.google.com/p/android/issues/detail?id=22586 https://code.google.com/p/android/issues/detail?id=22586

krschultz 12y ago
I don't think we've mellowed, just that we have become more sympathetic to Facebook's problem. I think the reality is that the 65k method limit problem has turned into something Android Developers put squarely at Google's feet. At the time I thought, what the hell is Facebook doing? Now, I've run into the 65k limit myself on an app written by 4 people over 18 months. Sure it's not a throwaway weekend project, but that is a pretty standard app. Google either needs to blow away the 65k method limit, or figure out a way to make Google Play Services a lot lighter. As it stands right now, if you want just push notifications you are giving up ~20k methods of your 65k limit for that alone. Just look at this post from Jake Wharton (leading Android open source developer): http://jakewharton.com/play-services-is-a-monolith/ http://jakewharton.com/play-services-is-a-monolith/
- emil10001 12y ago
  Slimming down Play Services just punts on the problem for a while. My view is that Google needs to do two things with this: 1) Split up the Play Services client libraries. 2) Figure out some solution for developers hitting the 65K limit. I'm sort of assuming that the solution would come in the form of a framework or tooling that we'd have to implement. Such a solution should allow us to build fully working apps with multiple dex files, with some decent documentation. It should also work fine for debug builds without proguard, and also without bumping build times up beyond a two minutes. What's more, we should be able to split off pieces that are defined in the manifest into secondary dex files, and fire intents at them. That, to me, seems like a reasonable response to this problem by Google. The company I work for has been hitting the limit for the last few months, and some of our other dependencies are becoming more and more expensive with newer releases (things that our users actually like.) So far, we've been able to build with a stripped down Play Services jar, but I'm not terribly happy about that approach. Regardless, I'm not going to blame Facebook for a problem that Google caused. If Facebook can find a solution, and tell the community how they did it, I'm all ears.

d0vs 12y ago

Mirror?