25 ms·

Dalvik patch for Facebook for Android

220 points by ctice 14y ago

deleted 14y ago
[deleted]
- saidajigumi 14y ago
  If you read TFA, they bent over backwards to attempt app-only methods... but ultimately hit a hard wall on the LinearAlloc buffer size in Android 2.x. So, off to extraordinary measures land they went.
  - marhumph 14y ago
    No, they went off down a path, and then found that their approach didn't work on one of their target devices. If they had known about this hard wall in one of their target devices, before making and implementing their code design choices, would they have taken the same approach? What they did is neat, why they did, given their journey, is understandable, how they frame what they did and how they did it is masturbatory.
cek 14y ago
It's interesting to see the parallels between Android and Windows in this regard. Here's a story from my past, copied from a blog post I wrote last year: My first job at Microsoft was providing developer support for the early Windows SDKs. To do my job well, I spent hours studying the Windows SDK documentation, the Windows source code, and writing sample applications. I then spent hours poring over customers’ (such as Lotus, WordPerfect, and Computer Associates) code helping them figure out what was not working. This gave me a deep appreciation for API design early in my career. I saw clear examples of elegant APIs as well as horrific monstrosities. I once created a sample for selecting a font in GDI that I called “ReallyCreateFont” because both of the built-in Windows APIs (CreateFont() and CreateFontEx()) were basically broken. The CreateFont case taught me first hand the pain resulting from exposing an API publicly, having customers use it in ways not expected, and then having to do unnatural acts to avoid breaking backwards compatibility. I was supporting WordPerfect’s development of WordPerfect for Windows. This was sometime in early 1991 if I remember correctly. I had the WordPerfect engineer on the phone. He had given me access to the WordPerfect source code, but I couldn’t give him access to the Windows code. So we were each stepping through WordPerfect in the debugger, him in Utah and me in Bellevue. I could see what his code was doing and what Windows was doing, but he could only see disassembly for Windows. The problem he was seeing had to do with font rendering. In the debugger we stepped into CreateFontEx(), which calls into CreateFont(), which calls into some internal GDI function, which calls into the device driver’s Escape() function (I can’t believe I actually remember all this crap like it was yesterday). Somewhere in this call stack I came across a block of code in Windows with comments that read something like // This is a hack to deal with Adobe’s Type Manager. // ATM *injects code* into GDI replacing our code with their own. // In order to retain backwards compatability we detect // ATM’s attempt to do this, let it do it, and then trick it // into thinking it actually hacked us this way. I am not making this up (although that comment is paraphrased from my memory). It turns out that the way WordPerfect was using CreateFontEx() was reasonable, but pushing the envelope of what anyone had done before, effectively exposing a bug caused not by the original API design or implementation, but something the API provider had to do to ensure app compatibility because of something another 3rd party had done! Holy-shit! Let me get this straight: * A 3rd party app (Adobe Type Manager) used CreateFontEx() in a way the API designer failed to anticipate. * The API designer needed to retain backwards compatibility so it had to resort to putting 3rd party app specific code in the API implementation. * Another 3rd party comes along and the complexity of the ‘fix’ caused another bug to surface. Welcome to the world of a true virtuous platform...
- ajtaylor 14y ago
  As soon as you mentioned "Adobe Type Manager" I remembered reading about this case some time ago. But knowing it was you that discovered this makes it all the better. If only we could create perfect APIs the first time around...
  - Arnt 14y ago
    I've created more than a few and IMO there's a reasonable approximation of perfect, namely an API which is possible to explain completely in reasonable prose. If the documentation has to go off and spend much verbiage on side cases, something's wrong. There's an imbalance in the API that's reflected in the documentation. If the first few sentences of the documentation read like committeework, or sound like the explanation of the explanation, then something's wrong. The API is not going to be compatible with its users. If the documentation talks more about specific use-cases instead of classes of behaviour, then something's wrong. IF the documentation tells you to do this, then that, then that, then something's wrong. It should talk about useful, describable atoms, not things that make sense only as a sequence. There are a few more. All this doesn't help to improve an API before release, at least not directly. But reading the documentation makes it easy to detect API brokenness while it can still be unfucked.
- blinkingled 14y ago
  Came here to opine on the exact same topic! I was going to dig up some Raymond Chen blog entries but your CreateFont example illustrates it equally well. I guess hacks are inevitable when you are dealing with that kind of scale/complexity.
  - prostoalex 14y ago
    It's less of a scale issue and more of an ease of update issue. Back in the days someone who bought a copy of Windows would take it home, and if it broke some existing software they used, they were going to blame the OS and return it to the store (they surely are not returning WordPerfect or Adobe product that they bought eons ago). Having always-connected clients and ability to force-update the code changes the game. I wonder if the backwards compatibility culture is still prevalent in Microsoft Windows division, provided they could always push a fix via Windows Update, or tell Adobe, etc. to fix their stuff.
MichaelGG 14y ago
Certainly if there's a lot of small methods, they could have an optimization pass that inlines some of them and bring their overall method count down. I wouldn't be surprised if that was a small performance gain, too.
- jerf 14y ago
  I thought of that, then the pragmatic engineer in me observed that they only discovered this after cracking their app into so many small methods. Given the time choice of 1. uncracking their app into small methods (which presumably wasn't atomic, so now that's all mixed in with other development, so it's not like it can just be reverted) or 2. hacking the system to handle more methods, I am forced to admit that choice 2 is two or possibly three orders of magnitude faster, in man-hours. (Probably two or three of their best engineers for a couple of weeks, vs. "the entire team", from dev to QA, tasked for months, on work that results in no new features). Not better in all possible ways, but it's hard to justify "do it right" against that sort of cost/benefit ratio. Then I was going to say "I hope they move towards a more permanent solution" when again my inner pragmatist popped up and said "Yes, they did; unsupport Android 2 in the future." So... yeah.. nasty, nasty hack, probably an enormous win almost all the way around. So the wheel turns.
  - magic_haze 14y ago
    I think GP meant something like a build process that inlines the bytecode, not the source code itself. I don't see why this is unreasonable, especially compared to what they actually did.
    - jerf 14y ago
      For as nasty as the hack is, it still boils down to twiddling with a few numbers. Also, inlining is probably easier said than done. It's not enough to inline the methods, you need to make them entirely disappear, and ISTR reading within the last few days somebody else commenting that Java still has to keep the metadata about the methods around (which is the problematic part, not the methods themselves, if I'm reading this right) because reflection may demand them. You'd need something more sophisticated to do it at the source level, and with an imperative language with unrestricted side effects, while that is certainly possible, it's also very much easier said than done. That's not a weekend hack either.
thefreeman 14y ago
Should we be worried that they are able to override things in the OS like this? I assume there is no built in "permission" prompt for this type of access.
- julianz 14y ago
  It's not in the OS, it's in the individual VM that Facebook is running in. Every Android app gets it's own VM - if they mess it up badly enough then it'll just crash.
marhumph 14y ago
The first paragraph is just rampant self praise. Basically these engineers decided to be really clever, painted themselves into a corner, smashed a hole in the wall so as to escape from said corner, and then bragged about how great they are.
- mikeash 14y ago
  Hitting an arbitrary (and undocumented?) platform limit on the number of methods your app is allowed to have is hardly "painting themselves into a corner".
  - myko 14y ago
    I'm not sure if it is listed in the official documentation but it was talked about by the Android team on their blog: http://android-developers.blogspot.com/2011/07/custom-class-loading-in-dalvik.html http://android-developers.blogspot.com/2011/07/custom-class-...
    - mikeash 14y ago
      This appears to discuss the per-dex method limit but makes no mention of the per-process limit.
- zeckalpha 14y ago
  Welcome to Facebook engineering.
- bhaak 14y ago
  As a programmer if you feel clever, most of the time you're doing something wrong. There might be 1% of code that should or can be clever but I prefer my code to be robust. They clearly say that "this idea seemed completely insane" so it's not like they took this road lightly. They searched for a solution and became rather desperate. I wonder if they asked Google for help. Having said that I'm not sure I would be frank about such a crazy hack for a slow behemoth app that is only considered "fast" because the previous version was insultingly slow.
barbs 14y ago
I really love reading recounts like this. Seeing other developers try, fail, and try again to come up with creative solutions for difficult problems, and then finally arriving at a solution, is so satisfying. It really demonstrates how creativity is manifested in programming, and is truly inspiring. A favourite of mine is Jari Komppa's article on his process of porting Death Rally from DOS to Windows: http://sol.gfxile.net/dr_gdm.html http://sol.gfxile.net/dr_gdm.html. Does anyone have any other articles along the same lines? I'd love to read them!
- anyfoo 14y ago
  Thanks, that was indeed a great read. You might also enjoy this, which is fairly recent: http://fabiensanglard.net/duke3d/index.php http://fabiensanglard.net/duke3d/index.php And the article links to other promising articles at the top, though I haven't checked them out. The general question remains: how would we go about finding similar stuff? Oh, one more thing, if you don't know yet, you will find great gems in Raymond Chen's blog, http://blogs.msdn.com/b/oldnewthing/ http://blogs.msdn.com/b/oldnewthing/ Not everything is archeological (a lot is still interesting though), but the articles often link to a whole bunch of other articles, which in turn do the same. I suggest, for example, http://blogs.msdn.com/b/oldnewthing/archive/2012/06/29/10325295.aspx http://blogs.msdn.com/b/oldnewthing/archive/2012/06/29/10325... . I deliberately chose a later article in the series, since it also links to previous ones.
imrehg 14y ago
"We were certainly glad and grateful that Android is open source—otherwise, this change wouldn’t have been possible." In return, maybe you could open-source some of your app, for example?
- nuclear_eclipse 14y ago
  We do already open source a wide variety of projects, for those who aren't already aware: https://github.com/facebook https://github.com/facebook
  - navpatel 14y ago
    nuclear_eclipse: can you point out the repo where this fix was applied?
  - ansible 14y ago
    We use flashcache on one of our build servers. It has been good so far. Thanks!
ajtaylor 14y ago
I can't tell if I should be excited for the developers to have been so clever, or horrified with all the hacks they had to incorporate to make it work. Either way, the new FB app is definitely an improvement over the older version (lag central!). Is it reasonable for the OS to make such a low-level component configurable so this type of convolutions aren't required?
- aredington 14y ago
  You should be angry that they have managed to persuade you that theirs was the only way out. The odds that properly modularizing their source tree such that they didn't provoke the limitations of the VM they were running on was impossible are vanishingly small, and the odds that they correctly hacked the VM are also vanishingly small. Facebook's Android Engineers chose the ego stroking solution which involved clever hacks. This only reinforces my being happy to have uninstalled the FB android app 2 years ago.
navpatel 14y ago
This issue has been a colossal pain in the ass! I need this! Are they planning on releasing this code to replace the buffer?
- klewelling 14y ago
  I am working on a product that may solve this problem for you. The product is designed for a different purpose but it should work for you as well. The only caveat would be the need to break your app into multiple pieces that run in separate processes. Give me a shout at kenneth@inappsquared.com if you are interested.
jmillikin 14y ago
According to http://techcrunch.com/2013/03/04/facebook-google-dalvik/ http://techcrunch.com/2013/03/04/facebook-google-dalvik/ and http://venturebeat.com/2013/03/04/google-facebook-android/ http://venturebeat.com/2013/03/04/google-facebook-android/ , the specific limit being hit was a cap of three million methods per instance of the Dalvik VM. Wisdom of a three-million-method app aside, I wonder why they didn't fix this in the compilation toolchain rather than by poking around in native memory areas during app startup. Facebook wrote a PHP->C++ compiler; changing the Dalvik compiler to more aggressively inline small methods seems trivial in comparison.
- pbiggar 14y ago
  They mention in the article that they tried. I have a small background in static analysis of Java, and between reflection, the class loader, and the huge amount of polymorphism that you find in java, it is very very difficult to inline basically anything statically.
- zenocon 14y ago
  I really can't fathom how, Facebook (the mobile / client app) could be made up of 3 million methods. That's an enormous amount of code -- none of it adds up.
  - fidotron 14y ago
    It's probably Thrift. It generates absolutely incredible amounts of code for even relatively small pieces of data. Quite unsuitable for the purpose of mobile dev.
    - Slizyboy 14y ago
      Thrift isn't used on the mobile apps, it is meant (and used for) communication between different services in their backend.
- georgemcbay 14y ago
  Three million methods? Having both programmed for Android and having used the Facebook App for Android, I find it very difficult to believe that app has anywhere near 3 million methods, even if each one is basically a one-liner. If the number is correct, that really is some kind of crazytown code.
pbiggar 14y ago
This seems like it will paint the Android core developers into a corner, supporting this particular hack for years to come, like the way Microsoft had to for Windows. If I were android core developers, this would not be a good day.
- pserwylo 14y ago
  I think the limitation is only on older versions, so they would probably only do the hack if it was pre Gingerbread. They mention that ICS works as expected, so I would be more than a bit mortified if they did this hack for new working versions too.
- kijiki 14y ago
  This hack is specific to ancient versions of Android, and is not used on current or future versions. So there is no need to deal with it in any codebase that isn't completely dead.
  - objclxt 14y ago
    > "specific to ancient versions of Android" Ancient but still shipping. I can go into a store and pick up a Samsung Galaxy Ace running 2.3 today. A significant minority of 2.3 based phones are still being sold, especially for budget/Pay As You Go users. This hack will need to remain in the codebase for a significant time.
- kalleboo 14y ago
  In addition to what others said about this only applying to already dead versions of Android, this is one of the upsides of distributing apps online. Nobody is installing an old boxed copy of Facebook 2.0 - google can just ask facebook to remove the hack when they've fixed it, and instantly everyone who installs the app it has the fix, and for anyone else the update is a tap away. This wasn't possible with the hacks to keep old Windows/DOS apps running.
chuinard 14y ago
"choose between cutting significant features from the app or only shipping our new version to the newest Android phones (ICS and up)" The ICS move wouldn't have been a bad option. It would've helped a lot of users convert to ICS+ devices, which Google probably would've loved. The APIs from 4+ are much more robust than pre-ICS, and it probably would've saved more headaches while building the new version in addition to the one already described in the post.
- bvdbijl 14y ago
  Yeah except they souls shut out lots of users
- tspiteri 14y ago
  The ICS move wouldn't have been a bad option. Maybe in a couple of years, but not yet. Pre-ICS Android still has more than 50% of Android's share. http://developer.android.com/about/dashboards/index.html http://developer.android.com/about/dashboards/index.html
- chucknthem 14y ago
  Android 2.2 and 2.3 devices are still being released on low end phones, especially in developing countries, so the number of pre-ICS users are actually growing, not shrinking.
  - psionski 14y ago
    Bought one a year ago (2.3.3, I think). It's a Wildfire S, very small and lightweight, has a web browser. Can make calls and probably can send short messages. I was using Windows Mobile 2003 before that and thought I was finally on the "bleeding edge" of mobile technology, only to find out a month later that Android 2.3 is considered "legacy" and they are talking about not supporting it? What, are we supposed to buy new phones every five years now?
    - dhugiaskmak 14y ago
      are we supposed to buy new phones every five years now? No. Every two.
gingerjoos 14y ago
>>> It seemed like we would have to choose between cutting significant features from the app or only shipping our new version to the newest Android phones (ICS and up). Neither seemed acceptable. We needed a better solution. This. "There has to be a better solution", a quality which defines hackers - is it not? It's that pushing the boundaries for the sake of getting things done that drive progress, isn't it? It's tempting to take the easy route out. In this case they could have justified supporting devices ICS up, Gingerbread and down does not have a lot of market share I assume. But they chose to persist. It's that persistence that we need to cultivate to become successful hackers.
- mbetter 14y ago
  Problem: You can't develop your film outside at night because the moon is too bright. Solution: Destroy the moon.
no_more_death 14y ago
Uninstalled. I checked, and Facebook took up more memory than any other application I have installed, even more than TSF Shell. (By the way, a MUCH more "feature-rich," useful, and snappy app than facebook!) Facebook isn't doing anything comparable to what TSF Shell does, not for me anyway. I bet this is why my phone recently started taking 30 seconds to make a phonecall (!). I can't believe an app requires 8M RAM just for method names! That's 3% of my entire usable memory (what's left after Motorola Blur takes its share) -- just for the method names for a single app. Surely I must be misunderstanding what goes into that buffer. It's unimaginable to me that they really need such a vast amount of space. It's unimaginable that they really expect their app to work at all on an old phone without choking it to death. If I grasp this correctly, printing just the method names would create a book double the size of the King James Version of the Bible. Maybe you are going to argue having so many methods is good programming practice, not bloat. But don't argue that you "support" a system when just the method names total around 3% of usable memory for that system. And there are a lot of devices with considerably less memory than what I've got. Why don't you release a stripped down, "lite" version of the app that works on an Android 2 phone even if it has more than 5 applications installed? Some phones don't even allow uninstalling facebook. They are simply stuck with the bloat (I mean, good programming practices :-).
- kristianp 14y ago
  Yes, I'm in the same boat. On my Android 2.2 phone (HTC Legend) it is hard to fit some current apps and facebook is a system app so I can't uninstall it. However I can uninstall the updates, so my facebook is the original factory version. I won't be trying out this latest version. It would be interesting to know what it is doing to be that complex. I'm guessing lots of third party libraries for ads, logging, etc?
- markokocic 14y ago
  The only reason to root my old HTC Desire that I still use happily was to be able to uninstall Facebook, Weather, Twitter and a few more apps I didn't need. Needless to say, the phone suddenly felt much snappier and my battery life doubled.
- inoop 14y ago
cbsmith 14y ago
Honestly, even on my far more powerful Nexus 4, running modern Jelly Bean, the Facebook app is worse than the Web app. I think their problems are way bigger than they appreciate.
exabrial 14y ago
...too bad they reinvented the wheel with Dalvik. It'd be awesome to be running this stuff under openjdk or hotspot
vidoc 14y ago
Is this me or Facebook tends to self-celebrate itself in most of its tech blog posts? I remember them posting a similar piece a few years back on their memcached scaling hacks - the article had a very similar tone, it was equally modest (e.g: not), and was actually nothing but software engineering imposture aimed at impressing a non-tech audience.
buster 14y ago
From my experience the FB app is one of the worst apps on my phone. The messaging notifications are crap, the reloading of my feed is crap (aka doesn't load notifications or in an unpredeictable manner). I have much more complex apps on my phone which show no such behaviour, yet it somehow is a Dalvik problem? Mhhh... How many Twitter apps and FB apps are there that are snappy and behave well ( = much better then the FB app!)..
kumarm 14y ago
Unfortunately TechCrunch reported incorrectly and no one at Facebook cared enough to correct them (OR thought its cool to boost 3M Methods in an app). The limitation in older Dalvik VM's is 64K Methods and not 1 Million methods which is realistic count that a large app would hit (even decent sized app with high number of third party libraries). This limitation is known for a while and Google Engineers talked about it multiple times.
JulianMorrison 14y ago
No, that is not cool, that is an admission that 1. They bloated the app so badly, they would have had to monkey patch the OS to let it run at all. 2. They actually did have the stonkingly huge arrogance to monkey patch the OS. What they should have done is shattered it down into a constellation of related apps and background processes. This app is FB news, that app is FB messaging, and so on. Bonus: people could pick and choose the parts of the system they cared for. But that last is what they wanted to avoid, I think. Facebook à la carte means that they can't foist unwanted "services" on you. And so they hacked.
jcfrei 14y ago
it's great for employees at facebook that the company allows their engineers such tinkering (to an absolute extreme in this point). but I think fb engineers should work on bigger problems than hacking the dalvik vm - like stopping the erosion of their younger user base.
yeureka 14y ago
I don't get it. FB is so proud of "hiring the best engineers" and then they have to resort to this kind of hacks to run their app?

220 points by ctice 14y ago

deleted 14y ago

[deleted]

saidajigumi 14y ago
If you read TFA, they bent over backwards to attempt app-only methods... but ultimately hit a hard wall on the LinearAlloc buffer size in Android 2.x. So, off to extraordinary measures land they went.
- marhumph 14y ago
  No, they went off down a path, and then found that their approach didn't work on one of their target devices. If they had known about this hard wall in one of their target devices, before making and implementing their code design choices, would they have taken the same approach? What they did is neat, why they did, given their journey, is understandable, how they frame what they did and how they did it is masturbatory.

cek 14y ago

It's interesting to see the parallels between Android and Windows in this regard. Here's a story from my past, copied from a blog post I wrote last year: My first job at Microsoft was providing developer support for the early Windows SDKs. To do my job well, I spent hours studying the Windows SDK documentation, the Windows source code, and writing sample applications. I then spent hours poring over customers’ (such as Lotus, WordPerfect, and Computer Associates) code helping them figure out what was not working. This gave me a deep appreciation for API design early in my career. I saw clear examples of elegant APIs as well as horrific monstrosities. I once created a sample for selecting a font in GDI that I called “ReallyCreateFont” because both of the built-in Windows APIs (CreateFont() and CreateFontEx()) were basically broken. The CreateFont case taught me first hand the pain resulting from exposing an API publicly, having customers use it in ways not expected, and then having to do unnatural acts to avoid breaking backwards compatibility. I was supporting WordPerfect’s development of WordPerfect for Windows. This was sometime in early 1991 if I remember correctly. I had the WordPerfect engineer on the phone. He had given me access to the WordPerfect source code, but I couldn’t give him access to the Windows code. So we were each stepping through WordPerfect in the debugger, him in Utah and me in Bellevue. I could see what his code was doing and what Windows was doing, but he could only see disassembly for Windows. The problem he was seeing had to do with font rendering. In the debugger we stepped into CreateFontEx(), which calls into CreateFont(), which calls into some internal GDI function, which calls into the device driver’s Escape() function (I can’t believe I actually remember all this crap like it was yesterday). Somewhere in this call stack I came across a block of code in Windows with comments that read something like // This is a hack to deal with Adobe’s Type Manager. // ATM *injects code* into GDI replacing our code with their own. // In order to retain backwards compatability we detect // ATM’s attempt to do this, let it do it, and then trick it // into thinking it actually hacked us this way. I am not making this up (although that comment is paraphrased from my memory). It turns out that the way WordPerfect was using CreateFontEx() was reasonable, but pushing the envelope of what anyone had done before, effectively exposing a bug caused not by the original API design or implementation, but something the API provider had to do to ensure app compatibility because of something another 3rd party had done! Holy-shit! Let me get this straight: * A 3rd party app (Adobe Type Manager) used CreateFontEx() in a way the API designer failed to anticipate. * The API designer needed to retain backwards compatibility so it had to resort to putting 3rd party app specific code in the API implementation. * Another 3rd party comes along and the complexity of the ‘fix’ caused another bug to surface. Welcome to the world of a true virtuous platform...

ajtaylor 14y ago
As soon as you mentioned "Adobe Type Manager" I remembered reading about this case some time ago. But knowing it was you that discovered this makes it all the better. If only we could create perfect APIs the first time around...
- Arnt 14y ago
  I've created more than a few and IMO there's a reasonable approximation of perfect, namely an API which is possible to explain completely in reasonable prose. If the documentation has to go off and spend much verbiage on side cases, something's wrong. There's an imbalance in the API that's reflected in the documentation. If the first few sentences of the documentation read like committeework, or sound like the explanation of the explanation, then something's wrong. The API is not going to be compatible with its users. If the documentation talks more about specific use-cases instead of classes of behaviour, then something's wrong. IF the documentation tells you to do this, then that, then that, then something's wrong. It should talk about useful, describable atoms, not things that make sense only as a sequence. There are a few more. All this doesn't help to improve an API before release, at least not directly. But reading the documentation makes it easy to detect API brokenness while it can still be unfucked.
blinkingled 14y ago
Came here to opine on the exact same topic! I was going to dig up some Raymond Chen blog entries but your CreateFont example illustrates it equally well. I guess hacks are inevitable when you are dealing with that kind of scale/complexity.
- prostoalex 14y ago
  It's less of a scale issue and more of an ease of update issue. Back in the days someone who bought a copy of Windows would take it home, and if it broke some existing software they used, they were going to blame the OS and return it to the store (they surely are not returning WordPerfect or Adobe product that they bought eons ago). Having always-connected clients and ability to force-update the code changes the game. I wonder if the backwards compatibility culture is still prevalent in Microsoft Windows division, provided they could always push a fix via Windows Update, or tell Adobe, etc. to fix their stuff.

MichaelGG 14y ago

Certainly if there's a lot of small methods, they could have an optimization pass that inlines some of them and bring their overall method count down. I wouldn't be surprised if that was a small performance gain, too.

jerf 14y ago
I thought of that, then the pragmatic engineer in me observed that they only discovered this after cracking their app into so many small methods. Given the time choice of 1. uncracking their app into small methods (which presumably wasn't atomic, so now that's all mixed in with other development, so it's not like it can just be reverted) or 2. hacking the system to handle more methods, I am forced to admit that choice 2 is two or possibly three orders of magnitude faster, in man-hours. (Probably two or three of their best engineers for a couple of weeks, vs. "the entire team", from dev to QA, tasked for months, on work that results in no new features). Not better in all possible ways, but it's hard to justify "do it right" against that sort of cost/benefit ratio. Then I was going to say "I hope they move towards a more permanent solution" when again my inner pragmatist popped up and said "Yes, they did; unsupport Android 2 in the future." So... yeah.. nasty, nasty hack, probably an enormous win almost all the way around. So the wheel turns.
- magic_haze 14y ago
  I think GP meant something like a build process that inlines the bytecode, not the source code itself. I don't see why this is unreasonable, especially compared to what they actually did.
  - jerf 14y ago
    For as nasty as the hack is, it still boils down to twiddling with a few numbers. Also, inlining is probably easier said than done. It's not enough to inline the methods, you need to make them entirely disappear, and ISTR reading within the last few days somebody else commenting that Java still has to keep the metadata about the methods around (which is the problematic part, not the methods themselves, if I'm reading this right) because reflection may demand them. You'd need something more sophisticated to do it at the source level, and with an imperative language with unrestricted side effects, while that is certainly possible, it's also very much easier said than done. That's not a weekend hack either.

thefreeman 14y ago

Should we be worried that they are able to override things in the OS like this? I assume there is no built in "permission" prompt for this type of access.

julianz 14y ago
It's not in the OS, it's in the individual VM that Facebook is running in. Every Android app gets it's own VM - if they mess it up badly enough then it'll just crash.

marhumph 14y ago

The first paragraph is just rampant self praise. Basically these engineers decided to be really clever, painted themselves into a corner, smashed a hole in the wall so as to escape from said corner, and then bragged about how great they are.

mikeash 14y ago
Hitting an arbitrary (and undocumented?) platform limit on the number of methods your app is allowed to have is hardly "painting themselves into a corner".
- myko 14y ago
  I'm not sure if it is listed in the official documentation but it was talked about by the Android team on their blog: http://android-developers.blogspot.com/2011/07/custom-class-loading-in-dalvik.html http://android-developers.blogspot.com/2011/07/custom-class-...
  - mikeash 14y ago
    This appears to discuss the per-dex method limit but makes no mention of the per-process limit.
zeckalpha 14y ago
Welcome to Facebook engineering.
bhaak 14y ago
As a programmer if you feel clever, most of the time you're doing something wrong. There might be 1% of code that should or can be clever but I prefer my code to be robust. They clearly say that "this idea seemed completely insane" so it's not like they took this road lightly. They searched for a solution and became rather desperate. I wonder if they asked Google for help. Having said that I'm not sure I would be frank about such a crazy hack for a slow behemoth app that is only considered "fast" because the previous version was insultingly slow.

barbs 14y ago

I really love reading recounts like this. Seeing other developers try, fail, and try again to come up with creative solutions for difficult problems, and then finally arriving at a solution, is so satisfying. It really demonstrates how creativity is manifested in programming, and is truly inspiring. A favourite of mine is Jari Komppa's article on his process of porting Death Rally from DOS to Windows: http://sol.gfxile.net/dr_gdm.html http://sol.gfxile.net/dr_gdm.html. Does anyone have any other articles along the same lines? I'd love to read them!

anyfoo 14y ago
Thanks, that was indeed a great read. You might also enjoy this, which is fairly recent: http://fabiensanglard.net/duke3d/index.php http://fabiensanglard.net/duke3d/index.php And the article links to other promising articles at the top, though I haven't checked them out. The general question remains: how would we go about finding similar stuff? Oh, one more thing, if you don't know yet, you will find great gems in Raymond Chen's blog, http://blogs.msdn.com/b/oldnewthing/ http://blogs.msdn.com/b/oldnewthing/ Not everything is archeological (a lot is still interesting though), but the articles often link to a whole bunch of other articles, which in turn do the same. I suggest, for example, http://blogs.msdn.com/b/oldnewthing/archive/2012/06/29/10325295.aspx http://blogs.msdn.com/b/oldnewthing/archive/2012/06/29/10325... . I deliberately chose a later article in the series, since it also links to previous ones.

imrehg 14y ago

"We were certainly glad and grateful that Android is open source—otherwise, this change wouldn’t have been possible." In return, maybe you could open-source some of your app, for example?

nuclear_eclipse 14y ago
We do already open source a wide variety of projects, for those who aren't already aware: https://github.com/facebook https://github.com/facebook
- navpatel 14y ago
  nuclear_eclipse: can you point out the repo where this fix was applied?
- ansible 14y ago
  We use flashcache on one of our build servers. It has been good so far. Thanks!

ajtaylor 14y ago

I can't tell if I should be excited for the developers to have been so clever, or horrified with all the hacks they had to incorporate to make it work. Either way, the new FB app is definitely an improvement over the older version (lag central!). Is it reasonable for the OS to make such a low-level component configurable so this type of convolutions aren't required?

aredington 14y ago
You should be angry that they have managed to persuade you that theirs was the only way out. The odds that properly modularizing their source tree such that they didn't provoke the limitations of the VM they were running on was impossible are vanishingly small, and the odds that they correctly hacked the VM are also vanishingly small. Facebook's Android Engineers chose the ego stroking solution which involved clever hacks. This only reinforces my being happy to have uninstalled the FB android app 2 years ago.

navpatel 14y ago

This issue has been a colossal pain in the ass! I need this! Are they planning on releasing this code to replace the buffer?

klewelling 14y ago
I am working on a product that may solve this problem for you. The product is designed for a different purpose but it should work for you as well. The only caveat would be the need to break your app into multiple pieces that run in separate processes. Give me a shout at kenneth@inappsquared.com if you are interested.

jmillikin 14y ago

According to http://techcrunch.com/2013/03/04/facebook-google-dalvik/ http://techcrunch.com/2013/03/04/facebook-google-dalvik/ and http://venturebeat.com/2013/03/04/google-facebook-android/ http://venturebeat.com/2013/03/04/google-facebook-android/ , the specific limit being hit was a cap of three million methods per instance of the Dalvik VM. Wisdom of a three-million-method app aside, I wonder why they didn't fix this in the compilation toolchain rather than by poking around in native memory areas during app startup. Facebook wrote a PHP->C++ compiler; changing the Dalvik compiler to more aggressively inline small methods seems trivial in comparison.

pbiggar 14y ago
They mention in the article that they tried. I have a small background in static analysis of Java, and between reflection, the class loader, and the huge amount of polymorphism that you find in java, it is very very difficult to inline basically anything statically.
zenocon 14y ago
I really can't fathom how, Facebook (the mobile / client app) could be made up of 3 million methods. That's an enormous amount of code -- none of it adds up.
- fidotron 14y ago
  It's probably Thrift. It generates absolutely incredible amounts of code for even relatively small pieces of data. Quite unsuitable for the purpose of mobile dev.
  - Slizyboy 14y ago
    Thrift isn't used on the mobile apps, it is meant (and used for) communication between different services in their backend.
georgemcbay 14y ago
Three million methods? Having both programmed for Android and having used the Facebook App for Android, I find it very difficult to believe that app has anywhere near 3 million methods, even if each one is basically a one-liner. If the number is correct, that really is some kind of crazytown code.

pbiggar 14y ago

This seems like it will paint the Android core developers into a corner, supporting this particular hack for years to come, like the way Microsoft had to for Windows. If I were android core developers, this would not be a good day.

pserwylo 14y ago
I think the limitation is only on older versions, so they would probably only do the hack if it was pre Gingerbread. They mention that ICS works as expected, so I would be more than a bit mortified if they did this hack for new working versions too.
kijiki 14y ago
This hack is specific to ancient versions of Android, and is not used on current or future versions. So there is no need to deal with it in any codebase that isn't completely dead.
- objclxt 14y ago
  > "specific to ancient versions of Android" Ancient but still shipping. I can go into a store and pick up a Samsung Galaxy Ace running 2.3 today. A significant minority of 2.3 based phones are still being sold, especially for budget/Pay As You Go users. This hack will need to remain in the codebase for a significant time.
kalleboo 14y ago
In addition to what others said about this only applying to already dead versions of Android, this is one of the upsides of distributing apps online. Nobody is installing an old boxed copy of Facebook 2.0 - google can just ask facebook to remove the hack when they've fixed it, and instantly everyone who installs the app it has the fix, and for anyone else the update is a tap away. This wasn't possible with the hacks to keep old Windows/DOS apps running.

chuinard 14y ago

"choose between cutting significant features from the app or only shipping our new version to the newest Android phones (ICS and up)" The ICS move wouldn't have been a bad option. It would've helped a lot of users convert to ICS+ devices, which Google probably would've loved. The APIs from 4+ are much more robust than pre-ICS, and it probably would've saved more headaches while building the new version in addition to the one already described in the post.

bvdbijl 14y ago
Yeah except they souls shut out lots of users
tspiteri 14y ago
The ICS move wouldn't have been a bad option. Maybe in a couple of years, but not yet. Pre-ICS Android still has more than 50% of Android's share. http://developer.android.com/about/dashboards/index.html http://developer.android.com/about/dashboards/index.html
chucknthem 14y ago
Android 2.2 and 2.3 devices are still being released on low end phones, especially in developing countries, so the number of pre-ICS users are actually growing, not shrinking.
- psionski 14y ago
  Bought one a year ago (2.3.3, I think). It's a Wildfire S, very small and lightweight, has a web browser. Can make calls and probably can send short messages. I was using Windows Mobile 2003 before that and thought I was finally on the "bleeding edge" of mobile technology, only to find out a month later that Android 2.3 is considered "legacy" and they are talking about not supporting it? What, are we supposed to buy new phones every five years now?
  - dhugiaskmak 14y ago
    are we supposed to buy new phones every five years now? No. Every two.

gingerjoos 14y ago

>>> It seemed like we would have to choose between cutting significant features from the app or only shipping our new version to the newest Android phones (ICS and up). Neither seemed acceptable. We needed a better solution. This. "There has to be a better solution", a quality which defines hackers - is it not? It's that pushing the boundaries for the sake of getting things done that drive progress, isn't it? It's tempting to take the easy route out. In this case they could have justified supporting devices ICS up, Gingerbread and down does not have a lot of market share I assume. But they chose to persist. It's that persistence that we need to cultivate to become successful hackers.

mbetter 14y ago
Problem: You can't develop your film outside at night because the moon is too bright. Solution: Destroy the moon.

no_more_death 14y ago

Uninstalled. I checked, and Facebook took up more memory than any other application I have installed, even more than TSF Shell. (By the way, a MUCH more "feature-rich," useful, and snappy app than facebook!) Facebook isn't doing anything comparable to what TSF Shell does, not for me anyway. I bet this is why my phone recently started taking 30 seconds to make a phonecall (!). I can't believe an app requires 8M RAM just for method names! That's 3% of my entire usable memory (what's left after Motorola Blur takes its share) -- just for the method names for a single app. Surely I must be misunderstanding what goes into that buffer. It's unimaginable to me that they really need such a vast amount of space. It's unimaginable that they really expect their app to work at all on an old phone without choking it to death. If I grasp this correctly, printing just the method names would create a book double the size of the King James Version of the Bible. Maybe you are going to argue having so many methods is good programming practice, not bloat. But don't argue that you "support" a system when just the method names total around 3% of usable memory for that system. And there are a lot of devices with considerably less memory than what I've got. Why don't you release a stripped down, "lite" version of the app that works on an Android 2 phone even if it has more than 5 applications installed? Some phones don't even allow uninstalling facebook. They are simply stuck with the bloat (I mean, good programming practices :-).

kristianp 14y ago
Yes, I'm in the same boat. On my Android 2.2 phone (HTC Legend) it is hard to fit some current apps and facebook is a system app so I can't uninstall it. However I can uninstall the updates, so my facebook is the original factory version. I won't be trying out this latest version. It would be interesting to know what it is doing to be that complex. I'm guessing lots of third party libraries for ads, logging, etc?
markokocic 14y ago
The only reason to root my old HTC Desire that I still use happily was to be able to uninstall Facebook, Weather, Twitter and a few more apps I didn't need. Needless to say, the phone suddenly felt much snappier and my battery life doubled.
inoop 14y ago

cbsmith 14y ago

Honestly, even on my far more powerful Nexus 4, running modern Jelly Bean, the Facebook app is worse than the Web app. I think their problems are way bigger than they appreciate.

exabrial 14y ago

...too bad they reinvented the wheel with Dalvik. It'd be awesome to be running this stuff under openjdk or hotspot

vidoc 14y ago

Is this me or Facebook tends to self-celebrate itself in most of its tech blog posts? I remember them posting a similar piece a few years back on their memcached scaling hacks - the article had a very similar tone, it was equally modest (e.g: not), and was actually nothing but software engineering imposture aimed at impressing a non-tech audience.

buster 14y ago

From my experience the FB app is one of the worst apps on my phone. The messaging notifications are crap, the reloading of my feed is crap (aka doesn't load notifications or in an unpredeictable manner). I have much more complex apps on my phone which show no such behaviour, yet it somehow is a Dalvik problem? Mhhh... How many Twitter apps and FB apps are there that are snappy and behave well ( = much better then the FB app!)..

kumarm 14y ago

Unfortunately TechCrunch reported incorrectly and no one at Facebook cared enough to correct them (OR thought its cool to boost 3M Methods in an app). The limitation in older Dalvik VM's is 64K Methods and not 1 Million methods which is realistic count that a large app would hit (even decent sized app with high number of third party libraries). This limitation is known for a while and Google Engineers talked about it multiple times.

JulianMorrison 14y ago

No, that is not cool, that is an admission that 1. They bloated the app so badly, they would have had to monkey patch the OS to let it run at all. 2. They actually did have the stonkingly huge arrogance to monkey patch the OS. What they should have done is shattered it down into a constellation of related apps and background processes. This app is FB news, that app is FB messaging, and so on. Bonus: people could pick and choose the parts of the system they cared for. But that last is what they wanted to avoid, I think. Facebook à la carte means that they can't foist unwanted "services" on you. And so they hacked.

jcfrei 14y ago

it's great for employees at facebook that the company allows their engineers such tinkering (to an absolute extreme in this point). but I think fb engineers should work on bigger problems than hacking the dalvik vm - like stopping the erosion of their younger user base.

yeureka 14y ago

I don't get it. FB is so proud of "hiring the best engineers" and then they have to resort to this kind of hacks to run their app?