10 ms·

iOS at Facebook [pdf]

160 points by GarethX 11y ago

spydum 11y ago
So, FB app is bloated because they choose to be sloppy is what I read. Wonder how sustainable this is over the long term?
- phn 11y ago
  Sloppy?! Those guys have needs the majority of us will never need to deal with. Did you read the presentation? If anything they're the opposite of sloppy.
  - Touche 11y ago
    The parent is not referring to those needs (which are legitimate). The presentation talks a lot about the hacking culture; how there is a lot of duplicate code, a lot of dead code, a lot of "temporary" code. We call these things technical debt. But it sounds like they just never fix them. Refactors never happen.
    - chupy 11y ago
      I sincerely doubt the fact that there are companies which thoroughly fix "technical debt".
    - phn 11y ago
      They don't call it debt. It is simply their way of keeping their agility with a huge number of developers. If the flip side is junk that mostly impacts install space, I don't see it as a bad trade-off. What alternative way would you suggest to keep your mega team productive and shipping features?
      - xd1936 11y ago
        Hiring a team whose sole job description is garbage cleanup?
      - smrtinsert 11y ago
        Productivity != Total Output.
      - Touche 11y ago
        You can't avoid fixing tech debt. Eventually it impedes your ability to be productive. It happens, to some degree, to every software project. Someone else joked about "we wrote the app from the ground up" but that really does happen. You're not "shipping features" if you're rewriting the app.
      - s73v3r 11y ago
        1. Having a core team who's job it is to maintain the app and clean up the debt. 2. I seriously have to question that they absolutely need that pace. I don't buy it for a second that any of this is necessary or better than the alternative.
    - ch8230 11y ago
      Eventually you just hear "New version of our app rewritten from the ground up" when it all becomes one big pile of technical debt.
  - rimantas 11y ago
    If the were not sloppy they wouldn't have to deal with those needs too.
    - L-four 11y ago
      Because they wouldn't exist.
  - jonknee 11y ago
    Their needs don't explain all the redundant and unused code. It's slop, plain and simple.
    - phn 11y ago
      It's not. It's just decoupling. Redundant code exists because different groups of people needed the same thing in different contexts (with probably slightly different requirements), and co-ordinating to keep this stuff centralised is hard, and most of all time-consuming. This has two clear advantages. You don't need to fill a form every time you change the smallest of functions and when you mess up, you don't impact everyone else. Honestly, I don't get the negativity. EDIT: I don't mention dead code because that can be, if really needed, removed by a tool so I think the point is moot.
      - jonknee 11y ago
        > I don't mention dead code because that can be, if really needed, removed by a tool so I think the point is moot. It can easily be cleaned up in an automated fashion and they don't do it. That's exactly what everyone means by slop. If you could pick up the clothes on your floor but don't, it's not because you're a hacker. Update: not only do they not clean it up, they ship this shit to a billion people. Talk about lazy.
  - s73v3r 11y ago
    Their needs are because they're sloppy. Having an overarching Core team who's job is to maintain the app and guide the architecture would cut down on what they need immensely.
- applecore 11y ago
  This strategy was hugely successful for Microsoft since the early days of Windows NT 3.1: “What Intel giveth, Microsoft taketh away.” The performance advancements on the hardware side of the platform are almost entirely consumed by the ever-increasing complexity of the most popular app on the software side.
  - mwcampbell 11y ago
    That may be successful for the software companies. But pity the user who's stuck with old hardware and wants to know why used computers don't last as long as used cars. Maybe when Moore's Law finally becomes invalid...
    - buffoon 11y ago
      Yep. 2005 G5 powermac. Totally usable machine. Can get a dual 2.5GHz boz with 4Gb of RAM for equivalent $40 in the UK. But no, it's stuck on OSX 10.5...
      - Redoubts 11y ago
        Sounds like a job for Debian!
        buffoon 11y ago
        Debian was a complete dick on my old ppc g4. It sort of put me off. The was going to try FreeBSD but I have to persuade the wife to let me buy a g5 yet as they're large, noisy and entirely unnecessary. I just want one to play with and go ooh and aah and stuff :) I did the same with old SPARC kit a number of years ago too.
        pja 11y ago
        PPC Debian runs just fine on my old Mac Mini. I could replace the thing, but it just keeps chugging on.
- paulojreis 11y ago
  It's not sustainable. But since software developers have been dealt a good hand (demand-wise) in these decades, we get away with it. And call it "hacker culture", so it sounds hip and even correct.
  - kevinchen 11y ago
    Actually, I'd counter that it is sustainable -- for Facebook. They have so much user data and so much ad revenue that they can afford to throw engineers and money at the problem. Big companies can work hard and make things work, smaller shops have to work efficiently out of necessity.
    - davnicwil 11y ago
      Yes indeed. From a slightly different perspective, assuming every good dev tries to be as efficient as they can be, even assuming they are, but with varying degrees of success due to various factors, a big company can hire lots of good people and only needs to get lucky and hit the mark occasionally. A small company on the other hand has to get lucky pretty much every single time.
- Apocryphon 11y ago
  Product-wise, it's likely that in the long term they're going to break up the FB app into a federation of related apps. They already did that with Messenger.
randomsearch 11y ago
This is truly a superb read. Don't just read the comments!
m4tthumphrey 11y ago
This was an awesome read. Thanks Simon!
shyber 11y ago
FYI when Sun owned the FB HQ building the address was 1 Network Drive :)
joeblau 11y ago
It seems like with such a strong engineering team, Facebook should have really built their own phone. It's impressive that they re-wrote a lot of the major components the iOS development process and optimized them for their scale. I think a Facebook style development platform on it's own hardware would be an interesting contrast to Google's and Apple's solutions. One challenge with learning this type of development stack is that you're almost not even learning traditional iOS development so your skills aren't truly transferrable.
- artmageddon 11y ago
  > It seems like with such a strong engineering team, Facebook should have really built their own phone. They actually tried not too long ago... [0] [0] http://www.cnet.com/news/heres-why-the-facebook-phone-flopped/ http://www.cnet.com/news/heres-why-the-facebook-phone-floppe...
- canistr 11y ago
  I'd say Facebook's emphasis on developer tools shows they'd be more committed to go down the route of building custom hardware for their devs like mouses, keyboards, laptops, workstations, etc. As opposed to building hardware for end-users.
JamesBaxter 11y ago
Very interesting, even as a fan of native apps I can see the thought process behind doing it web based when you have to build so many tools for each platform.
- miguelrochefort 11y ago
  Except it's mostly not web based.
  - Synaesthesia 11y ago
    Yes interestly enough there was a period when the front-end for mobile was html5 based but they since ditched that for greater (native) performance, in fact as this article reveals, a number of implementations!
  - JamesBaxter 11y ago
    As Synaesthesia notes, it was. I meant with processes and tools like the post describes building once sound like a good idea...
micampe 11y ago
This didn't really tell me why Facebook needs 400 people to work on the iOS app and 75 commits per hour.
- miguelrochefort 11y ago
  Surely most of these commits are either automated, or similar in nature (i.e., adding some static resources to some file).
  - jonknee 11y ago
    > Here are the stats for the week starting 24 August this year.In that week: 429 unique contributors 2823 total commits Right below it: > Doesn’t include commits to bump version numbers, add translated strings etc - actual number is over 4,000 Any way you slice it that is a chaotic pace of change.
- onewaystreet 11y ago
  Facebook doesn't have a dedicated iOS team, rather each product team (feed, photos, search, messages, etc.) commits code to the app. This allows updates to be made faster and more consistent since there aren't multiple product teams for each product for each platform, but it also leads to having a large number of contributors, commits and duplicate code.
- joeblau 11y ago
  What's crazy is that WhatApp has 4 Android engineers and 1 billion downloads.
  - themartorana 11y ago
    I would suggest that WhatsApp does a whole lot less.
    - joeblau 11y ago
      I agree, and I think that focus is actually a strength of the product.
rckrd 11y ago
Some sidenotes: I'm surprised to see so many custom libraries for Paper, an app which I had forgotten about. And another reminder that a lot of companies at scale are choosing mercurial over git.
bsdpython 11y ago
tl;dr Software engineering at extreme edge cases of performance and scalability can look very different than your typical use cases. Don't do this at home (until you have to). Interesting none-the-less.
- mnem 11y ago
  > extreme edge cases of performance and scalability That gave me a chortle :)
cableshaft 11y ago
This was pretty interesting. It would have never occurred to me that XCode, Git, etc. couldn't handle as much as you throw at it, but it kind of makes sense how they explain it here. Still, I don't think I would like to actually work in that environment where the codebase changed so much so often and had so much dead and redundant code, so I guess I'm glad my interview with them didn't go terribly well (got asked questions by non tech-type person who was looking for keywords to obscure concepts as answers and didn't understand my conceptual but keyword-less responses).
- manmal 11y ago
  I've worked with clients that had all their code in one repo at Github, and it took me 5 tries to even pull it down initially. You need not operate at Facebook scale to overwhelm git :)
- ch8230 11y ago
  What keywords do you think the interviewer was looking for? Was it along the lines of "disruptive", "game-changing", "innovative", etc.?
  - cableshaft 11y ago
    I don't remember what all of them were, but I specifically remember one of them had the words "Toll Booth" in it, like "Toll Booth Blah", where blah is the key word I can't remember. It's some networking/communication concept, I think. The phone interview person had a fairly difficult accent too so I don't think I even heard the whole answer. I've only ever encountered the term once online since. I just tried to Google for it again and I can't find the write combo of terms to find it ("toll booth networking? no.... toll booth computer science? no.....") So I tried to answer that question with what happens on a concept level, and the person on the line said "No. The answer is 'Toll Booth Blah'. 'Toll Booh Blah'" and moved on to the next question. Like she was Alex Trebek and there was only one way to answer the question. Meanwhile Google thought I was good enough to fly over for an in-person interview at around the same time, so I just wrote Facebook off as being terrible at hiring and moved on. To date it's still the worst phone interview experience I've had, and I've had some awful ones.
buffoon 11y ago
"Done is better than perfect" "Move fast and break things" I'm glad they're not holding any critical data like our personal information, what we do, what we think and where we are then! If this set of slides embodies 'hacker culture' which it claims to, I am not a hacker, neither are my colleagues and I'm proud of that. The whole thing stinks of arrogance hiding incompetence. I'm not putting anything near facebook, ever.
- asdfqwe1 11y ago
  To be fair, most of these specific signs have been removed now/changed (I was an intern there fall 2013). They actually talked about how that used to be a slogan, but breaking stuff isn't really acceptable anymore because facebook is so big. They still wanted us to move really fast though...
  - iLoch 11y ago
    Facebook's new slogan is "Move fast with stable infrastructure."
  - zeeshanm 11y ago
    From what I have heard they are now expecting their employees to still move fast, break stuff, but be ready to be penalized for it if you do. So, really, they are encouraging their employees to move at a steady pace with occasional hack nights when there is an upcoming release in the near-term. Facebook is growing old.
    - gorena 11y ago
      Does Facebook pay overtime for "hack nights"?
      - buffoon 11y ago
        High profile companies seem to have a privilege complex I.e. you're lucky to be working for them.
      - 11y ago
vinceyuan 11y ago
This post is very interesting. News reported Facebook would create a simplified Android app for low end phone owners in Africa. I think They should do it for iOS too.
- untog 11y ago
  They already have, it's called Facebook Lite. You can't get it from the Play Store in the US, but I download an APK. It's nice. I only really downloaded it so that I could get notifications, and now m.facebook.com can send them from Chrome, so I'm very happy to totally app-less.
rabino 11y ago
The unnecessary bashing of PHP, the lame justification for a sloppy architecture, the immense ego... I don't think hack culture should be used to justify asshole-ery.
emanuelev 11y ago
Can't even imagine the amount of crap in their code-base. I bet that amount of commits is necessary to continuously correct the shitload of mistakes coming from their "Hacking" culture.
- onewaystreet 11y ago
  They deal with this with automated code review tools like their Infer static analysis tool (http://fbinfer.com http://fbinfer.com) as well as abstracting away the Objective-C with React Native.
  - emanuelev 11y ago
    Yeah that might work for programming mistakes, but not sure it can help with bad design choices.
bjano 11y ago
So the main purpose of most of the 18000 classes is to make it possible for the app to scale up to 18000 classes? :)
- zacblazic 11y ago
  Partly! The tools they mentioned (IDE, Mercurial) don't impact the app itself though. Which makes them irrelevant and doesn't factor into why their codebase is massive.
  - 0x4a42 11y ago
    They are relevant if you take them for what they are: examples that explains, in a general way, why FB has large codebase. Because when something doesn't fits their needs they will write their own version. So I guess with iOS they wrote/rewrote some part of the system functionnalities to better fit their needs.
sabujp 11y ago
what is fb doing for iOS builds and compiles?
- aaronbrethorst 11y ago
  https://github.com/facebook/xctool https://github.com/facebook/xctool
  - rimantas 11y ago
    which is just a wrapper around Apple's xcodebuild.
    - aaronbrethorst 11y ago
      What's your point?
randomsearch 11y ago
It's interesting to note that this presentation strongly contradicts the advice of "The Pragmatic Programmer" and "Clean Code", which are highly respected by many. Does this mean that those books are wrong, or does it mean that Facebook will eventually come to regret their philosophy? Or maybe writing this kind of code is somehow fundamentally different, so the usual wisdom and rules of thumb don't apply. I really don't know the answer, but find it fascinating and can't wait to see what Facebook is doing in 10 years time.
- alexpersian 11y ago
  I think what it means is that as it stands now, there is no definite good/bad way to make successful software. There are just opinions.
  - buffoon 11y ago
    There are definite bad ways. I've watched a few companies go down the shitter using known bad ways.
    - Shish2k 11y ago
      I've watched a few companies go down the shitter using known good ways; and seen companies with mind-blowingly horrific software succeed based on marketing and contracts. There are a lot of anecdotes, but I have yet to see any data suggesting a correlation.
  - icefox 11y ago
    It also depends on your definition of successful. To be successful does software need to (check off what you think would matter): * Make money directly * Make money in-directly * Be used by at least 1 user (not including dev + dev friends and family) * Be alive and working in 6 months * Be alive and working in 2 years * Be alive and working in 10 years * Simply ship so dev gets a positive yearly review * Be bug free * Only crash N% of the time (who cares just re-run it and don't do the bad thing) * Quality / features don't matter as long as marketing is talking about it and people play with it so it looks successful. * Simply exist, but be subsidized by some one or something so devs keep hacking on it even if no one uses it. * Still be useful when there are zero devs working on it.
xenadu02 11y ago
It's all a bunch of bored engineers looking for ways to be clever but without anyone to take ownership of the codebase. There is absolutely no reason for the Facebook app to be so large and have so many classes. The app itself doesn't do that much.
- gorena 11y ago
  Yeah, the difference between Facebook and Instagram or Twitter is incremental. They basically do the same stuff, Facebook does a little bit more (although, they entirely removed direct messages). Why is the Facebook app so huge?
  - Narretz 11y ago
    I wouldn't be surprised if all the messages code is still in the app.
    - edgyswingset 11y ago
      I've got an alpha of the Facebook for Android app and it's got all of my conversations loaded, including text from the last message. It'll even light up with a red indicator when I get a new message. All the actions start an Intent which launches my Messenger app, though. So yeah, it's probably all dead code that's still there.
- brazzledazzle 11y ago
  >It's all a bunch of bored engineers looking for ways to be clever but without anyone to take ownership of the codebase. You couldn't think of a more charitable interpretation? I'm not one to care about rules but the hn guidelines on commenting are important for making sure the community doesn't eat itself alive like reddit did/does. >The app itself doesn't do that much. Right, but in the presentation he made it clear that it was less about what the app does and more about the number of engineers making changes to it. Given the size of their teams all making their own changes to the iOS codebase, doesn't it make sense to modularize it so they aren't tripping over each other breaking the application constantly? They don't have a dedicated iOS team and having one would bottleneck the other teams.
bsaul 11y ago
So i guess maybe Apple didn't code their ios and ios apps using xcode, xcode-git, autolayout and coredata after all. Because their scale seems pretty big too.
colinbartlett 11y ago
I loved the anecdote starting on page 41 where he accidentally "memorializes" his Facebook profile thus triggering an email to his wife about his death.
chuckcode 11y ago
Would be interested to see some comparison with the Android app which seems to have most of the same functionality at 1/4 the size with 25MB [1] compared to the 100MB of iOS. Seems like either iOS team at FB has significant cruft laying around or iOS doesn't scale as well as Android and FB is hacking around that. Possibly a bit of both but think it would be an interesting comparison in addition to the war stories about hacking their entire tool chain. [1] https://gigaom.com/2015/01/26/facebooks-new-android-app-goes-light-on-the-data/ https://gigaom.com/2015/01/26/facebooks-new-android-app-goes...
- dmishe 11y ago
  iOS apps are shipped as fat binaries, so for one, part of that 100mb app is duplicated slices of binaries for armv7, arm64 and possibly, but not likely, armv7s. Android apps enjoy bytecode that is getting JIT-ed on install.
  - dgallagher 11y ago
    Apple has introduced App Thinning in iOS 9 apps which should cut down on their size: https://developer.apple.com/library/watchos/documentation/IDEs/Conceptual/AppDistributionGuide/AppThinning/AppThinning.html https://developer.apple.com/library/watchos/documentation/ID...
    - azsromej 11y ago
      The feature was recently delayed, but it should help once it's ready
  - notNow 11y ago
    Is there any technical justification for this or it's purely economical/financial?
    - dmishe 11y ago
      For what, fat binaries? That's pretty much what Mach-O format had from inception under NeXTSTEP, and survived until now. As it was said in the sibling comment, iOS 9 is introducing a way to cut exactly the slice you need for a particular device.
brodd 11y ago
Am I reading this right, there are interns and engineers running PHP in a debugger on production data? How else would he be able to memorialize his own account and trigger the email?
- randomsearch 11y ago
  That is a good point, how did no-one else pick up on that???! It reminds me of engineers where I used to work, who would run through a £0.01 transaction on their credit card to check the changes to the charging system were still working...
- thrusong 11y ago
  I've seen talks with engineers that say this is this case. I think the one in particular was from @Scale. They've had the occasional issue with an intern dropping a table in production, but they said they haven't really run into issues with granting access to production data.
gorkemyurt 11y ago
(Another post about React Native) Facebook engineers keep telling everyone that React Native is not used in the main Facebook app. I always wondered what is stopping them from start using it. Perhaps app size is a massive concern? Maybe all this tooling around their app is not really compatible with React Native? Its interesting that they have a ton of legacy Obj-c code (considering that companies started to build apps not very long time ago, and they already re-wrote their app several times) I wonder if any other company is a situation like this well other than Apple.
- zghst 11y ago
  It would probably be a huge pain to migrate all 18,000 classes, tests, tools, etc., over to a React Native app. Also JS has less fine grained tooling for performance profiling (try stepping through JS bytecode for performance analysis, it's a PITA). One day it might get there, but not today. If it works, why change it?
  - gorkemyurt 11y ago
    I am not talking about migrating the whole app but how about for new features? I noticed that they redesigned their friends request page, why not build that with react native?
    - plorkyeran 11y ago
      The engineers who did that redesign didn't want to use react native or felt it wasn't a good choice. Just because one person at Facebook built a tool doesn't mean that everyone else at Facebook is suddenly required to use that tool where ever possible.

160 points by GarethX 11y ago

So, FB app is bloated because they choose to be sloppy is what I read. Wonder how sustainable this is over the long term?

phn 11y ago
Sloppy?! Those guys have needs the majority of us will never need to deal with. Did you read the presentation? If anything they're the opposite of sloppy.
- Touche 11y ago
  The parent is not referring to those needs (which are legitimate). The presentation talks a lot about the hacking culture; how there is a lot of duplicate code, a lot of dead code, a lot of "temporary" code. We call these things technical debt. But it sounds like they just never fix them. Refactors never happen.
  - chupy 11y ago
    I sincerely doubt the fact that there are companies which thoroughly fix "technical debt".
  - phn 11y ago
    They don't call it debt. It is simply their way of keeping their agility with a huge number of developers. If the flip side is junk that mostly impacts install space, I don't see it as a bad trade-off. What alternative way would you suggest to keep your mega team productive and shipping features?
    - xd1936 11y ago
      Hiring a team whose sole job description is garbage cleanup?
    - smrtinsert 11y ago
      Productivity != Total Output.
    - Touche 11y ago
      You can't avoid fixing tech debt. Eventually it impedes your ability to be productive. It happens, to some degree, to every software project. Someone else joked about "we wrote the app from the ground up" but that really does happen. You're not "shipping features" if you're rewriting the app.
    - s73v3r 11y ago
      1. Having a core team who's job it is to maintain the app and clean up the debt. 2. I seriously have to question that they absolutely need that pace. I don't buy it for a second that any of this is necessary or better than the alternative.
  - ch8230 11y ago
    Eventually you just hear "New version of our app rewritten from the ground up" when it all becomes one big pile of technical debt.
- rimantas 11y ago
  If the were not sloppy they wouldn't have to deal with those needs too.
  - L-four 11y ago
    Because they wouldn't exist.
- jonknee 11y ago
  Their needs don't explain all the redundant and unused code. It's slop, plain and simple.
  - phn 11y ago
    It's not. It's just decoupling. Redundant code exists because different groups of people needed the same thing in different contexts (with probably slightly different requirements), and co-ordinating to keep this stuff centralised is hard, and most of all time-consuming. This has two clear advantages. You don't need to fill a form every time you change the smallest of functions and when you mess up, you don't impact everyone else. Honestly, I don't get the negativity. EDIT: I don't mention dead code because that can be, if really needed, removed by a tool so I think the point is moot.
    - jonknee 11y ago
      > I don't mention dead code because that can be, if really needed, removed by a tool so I think the point is moot. It can easily be cleaned up in an automated fashion and they don't do it. That's exactly what everyone means by slop. If you could pick up the clothes on your floor but don't, it's not because you're a hacker. Update: not only do they not clean it up, they ship this shit to a billion people. Talk about lazy.
- s73v3r 11y ago
  Their needs are because they're sloppy. Having an overarching Core team who's job is to maintain the app and guide the architecture would cut down on what they need immensely.
applecore 11y ago
This strategy was hugely successful for Microsoft since the early days of Windows NT 3.1: “What Intel giveth, Microsoft taketh away.” The performance advancements on the hardware side of the platform are almost entirely consumed by the ever-increasing complexity of the most popular app on the software side.
- mwcampbell 11y ago
  That may be successful for the software companies. But pity the user who's stuck with old hardware and wants to know why used computers don't last as long as used cars. Maybe when Moore's Law finally becomes invalid...
  - buffoon 11y ago
    Yep. 2005 G5 powermac. Totally usable machine. Can get a dual 2.5GHz boz with 4Gb of RAM for equivalent $40 in the UK. But no, it's stuck on OSX 10.5...
    - Redoubts 11y ago
      Sounds like a job for Debian!
      - buffoon 11y ago
        Debian was a complete dick on my old ppc g4. It sort of put me off. The was going to try FreeBSD but I have to persuade the wife to let me buy a g5 yet as they're large, noisy and entirely unnecessary. I just want one to play with and go ooh and aah and stuff :) I did the same with old SPARC kit a number of years ago too.
        pja 11y ago
        PPC Debian runs just fine on my old Mac Mini. I could replace the thing, but it just keeps chugging on.
paulojreis 11y ago
It's not sustainable. But since software developers have been dealt a good hand (demand-wise) in these decades, we get away with it. And call it "hacker culture", so it sounds hip and even correct.
- kevinchen 11y ago
  Actually, I'd counter that it is sustainable -- for Facebook. They have so much user data and so much ad revenue that they can afford to throw engineers and money at the problem. Big companies can work hard and make things work, smaller shops have to work efficiently out of necessity.
  - davnicwil 11y ago
    Yes indeed. From a slightly different perspective, assuming every good dev tries to be as efficient as they can be, even assuming they are, but with varying degrees of success due to various factors, a big company can hire lots of good people and only needs to get lucky and hit the mark occasionally. A small company on the other hand has to get lucky pretty much every single time.
Apocryphon 11y ago
Product-wise, it's likely that in the long term they're going to break up the FB app into a federation of related apps. They already did that with Messenger.

randomsearch 11y ago

This is truly a superb read. Don't just read the comments!

m4tthumphrey 11y ago

This was an awesome read. Thanks Simon!

shyber 11y ago

FYI when Sun owned the FB HQ building the address was 1 Network Drive :)

joeblau 11y ago

It seems like with such a strong engineering team, Facebook should have really built their own phone. It's impressive that they re-wrote a lot of the major components the iOS development process and optimized them for their scale. I think a Facebook style development platform on it's own hardware would be an interesting contrast to Google's and Apple's solutions. One challenge with learning this type of development stack is that you're almost not even learning traditional iOS development so your skills aren't truly transferrable.

artmageddon 11y ago
> It seems like with such a strong engineering team, Facebook should have really built their own phone. They actually tried not too long ago... [0] [0] http://www.cnet.com/news/heres-why-the-facebook-phone-flopped/ http://www.cnet.com/news/heres-why-the-facebook-phone-floppe...
canistr 11y ago
I'd say Facebook's emphasis on developer tools shows they'd be more committed to go down the route of building custom hardware for their devs like mouses, keyboards, laptops, workstations, etc. As opposed to building hardware for end-users.

JamesBaxter 11y ago

Very interesting, even as a fan of native apps I can see the thought process behind doing it web based when you have to build so many tools for each platform.

miguelrochefort 11y ago
Except it's mostly not web based.
- Synaesthesia 11y ago
  Yes interestly enough there was a period when the front-end for mobile was html5 based but they since ditched that for greater (native) performance, in fact as this article reveals, a number of implementations!
- JamesBaxter 11y ago
  As Synaesthesia notes, it was. I meant with processes and tools like the post describes building once sound like a good idea...

micampe 11y ago

This didn't really tell me why Facebook needs 400 people to work on the iOS app and 75 commits per hour.

miguelrochefort 11y ago
Surely most of these commits are either automated, or similar in nature (i.e., adding some static resources to some file).
- jonknee 11y ago
  > Here are the stats for the week starting 24 August this year.In that week: 429 unique contributors 2823 total commits Right below it: > Doesn’t include commits to bump version numbers, add translated strings etc - actual number is over 4,000 Any way you slice it that is a chaotic pace of change.
onewaystreet 11y ago
Facebook doesn't have a dedicated iOS team, rather each product team (feed, photos, search, messages, etc.) commits code to the app. This allows updates to be made faster and more consistent since there aren't multiple product teams for each product for each platform, but it also leads to having a large number of contributors, commits and duplicate code.
joeblau 11y ago
What's crazy is that WhatApp has 4 Android engineers and 1 billion downloads.
- themartorana 11y ago
  I would suggest that WhatsApp does a whole lot less.
  - joeblau 11y ago
    I agree, and I think that focus is actually a strength of the product.

rckrd 11y ago

Some sidenotes: I'm surprised to see so many custom libraries for Paper, an app which I had forgotten about. And another reminder that a lot of companies at scale are choosing mercurial over git.

bsdpython 11y ago

tl;dr Software engineering at extreme edge cases of performance and scalability can look very different than your typical use cases. Don't do this at home (until you have to). Interesting none-the-less.

mnem 11y ago
> extreme edge cases of performance and scalability That gave me a chortle :)

cableshaft 11y ago

This was pretty interesting. It would have never occurred to me that XCode, Git, etc. couldn't handle as much as you throw at it, but it kind of makes sense how they explain it here. Still, I don't think I would like to actually work in that environment where the codebase changed so much so often and had so much dead and redundant code, so I guess I'm glad my interview with them didn't go terribly well (got asked questions by non tech-type person who was looking for keywords to obscure concepts as answers and didn't understand my conceptual but keyword-less responses).

manmal 11y ago
I've worked with clients that had all their code in one repo at Github, and it took me 5 tries to even pull it down initially. You need not operate at Facebook scale to overwhelm git :)
ch8230 11y ago
What keywords do you think the interviewer was looking for? Was it along the lines of "disruptive", "game-changing", "innovative", etc.?
- cableshaft 11y ago
  I don't remember what all of them were, but I specifically remember one of them had the words "Toll Booth" in it, like "Toll Booth Blah", where blah is the key word I can't remember. It's some networking/communication concept, I think. The phone interview person had a fairly difficult accent too so I don't think I even heard the whole answer. I've only ever encountered the term once online since. I just tried to Google for it again and I can't find the write combo of terms to find it ("toll booth networking? no.... toll booth computer science? no.....") So I tried to answer that question with what happens on a concept level, and the person on the line said "No. The answer is 'Toll Booth Blah'. 'Toll Booh Blah'" and moved on to the next question. Like she was Alex Trebek and there was only one way to answer the question. Meanwhile Google thought I was good enough to fly over for an in-person interview at around the same time, so I just wrote Facebook off as being terrible at hiring and moved on. To date it's still the worst phone interview experience I've had, and I've had some awful ones.

buffoon 11y ago

"Done is better than perfect" "Move fast and break things" I'm glad they're not holding any critical data like our personal information, what we do, what we think and where we are then! If this set of slides embodies 'hacker culture' which it claims to, I am not a hacker, neither are my colleagues and I'm proud of that. The whole thing stinks of arrogance hiding incompetence. I'm not putting anything near facebook, ever.

asdfqwe1 11y ago
To be fair, most of these specific signs have been removed now/changed (I was an intern there fall 2013). They actually talked about how that used to be a slogan, but breaking stuff isn't really acceptable anymore because facebook is so big. They still wanted us to move really fast though...
- iLoch 11y ago
  Facebook's new slogan is "Move fast with stable infrastructure."
- zeeshanm 11y ago
  From what I have heard they are now expecting their employees to still move fast, break stuff, but be ready to be penalized for it if you do. So, really, they are encouraging their employees to move at a steady pace with occasional hack nights when there is an upcoming release in the near-term. Facebook is growing old.
  - gorena 11y ago
    Does Facebook pay overtime for "hack nights"?
    - buffoon 11y ago
      High profile companies seem to have a privilege complex I.e. you're lucky to be working for them.
    - 11y ago

vinceyuan 11y ago

This post is very interesting. News reported Facebook would create a simplified Android app for low end phone owners in Africa. I think They should do it for iOS too.

untog 11y ago
They already have, it's called Facebook Lite. You can't get it from the Play Store in the US, but I download an APK. It's nice. I only really downloaded it so that I could get notifications, and now m.facebook.com can send them from Chrome, so I'm very happy to totally app-less.

rabino 11y ago

The unnecessary bashing of PHP, the lame justification for a sloppy architecture, the immense ego... I don't think hack culture should be used to justify asshole-ery.

emanuelev 11y ago

Can't even imagine the amount of crap in their code-base. I bet that amount of commits is necessary to continuously correct the shitload of mistakes coming from their "Hacking" culture.

onewaystreet 11y ago
They deal with this with automated code review tools like their Infer static analysis tool (http://fbinfer.com http://fbinfer.com) as well as abstracting away the Objective-C with React Native.
- emanuelev 11y ago
  Yeah that might work for programming mistakes, but not sure it can help with bad design choices.

bjano 11y ago

So the main purpose of most of the 18000 classes is to make it possible for the app to scale up to 18000 classes? :)

zacblazic 11y ago
Partly! The tools they mentioned (IDE, Mercurial) don't impact the app itself though. Which makes them irrelevant and doesn't factor into why their codebase is massive.
- 0x4a42 11y ago
  They are relevant if you take them for what they are: examples that explains, in a general way, why FB has large codebase. Because when something doesn't fits their needs they will write their own version. So I guess with iOS they wrote/rewrote some part of the system functionnalities to better fit their needs.

sabujp 11y ago

what is fb doing for iOS builds and compiles?

aaronbrethorst 11y ago
https://github.com/facebook/xctool https://github.com/facebook/xctool
- rimantas 11y ago
  which is just a wrapper around Apple's xcodebuild.
  - aaronbrethorst 11y ago
    What's your point?

randomsearch 11y ago

It's interesting to note that this presentation strongly contradicts the advice of "The Pragmatic Programmer" and "Clean Code", which are highly respected by many. Does this mean that those books are wrong, or does it mean that Facebook will eventually come to regret their philosophy? Or maybe writing this kind of code is somehow fundamentally different, so the usual wisdom and rules of thumb don't apply. I really don't know the answer, but find it fascinating and can't wait to see what Facebook is doing in 10 years time.

alexpersian 11y ago
I think what it means is that as it stands now, there is no definite good/bad way to make successful software. There are just opinions.
- buffoon 11y ago
  There are definite bad ways. I've watched a few companies go down the shitter using known bad ways.
  - Shish2k 11y ago
    I've watched a few companies go down the shitter using known good ways; and seen companies with mind-blowingly horrific software succeed based on marketing and contracts. There are a lot of anecdotes, but I have yet to see any data suggesting a correlation.
- icefox 11y ago
  It also depends on your definition of successful. To be successful does software need to (check off what you think would matter): * Make money directly * Make money in-directly * Be used by at least 1 user (not including dev + dev friends and family) * Be alive and working in 6 months * Be alive and working in 2 years * Be alive and working in 10 years * Simply ship so dev gets a positive yearly review * Be bug free * Only crash N% of the time (who cares just re-run it and don't do the bad thing) * Quality / features don't matter as long as marketing is talking about it and people play with it so it looks successful. * Simply exist, but be subsidized by some one or something so devs keep hacking on it even if no one uses it. * Still be useful when there are zero devs working on it.

xenadu02 11y ago

It's all a bunch of bored engineers looking for ways to be clever but without anyone to take ownership of the codebase. There is absolutely no reason for the Facebook app to be so large and have so many classes. The app itself doesn't do that much.

gorena 11y ago
Yeah, the difference between Facebook and Instagram or Twitter is incremental. They basically do the same stuff, Facebook does a little bit more (although, they entirely removed direct messages). Why is the Facebook app so huge?
- Narretz 11y ago
  I wouldn't be surprised if all the messages code is still in the app.
  - edgyswingset 11y ago
    I've got an alpha of the Facebook for Android app and it's got all of my conversations loaded, including text from the last message. It'll even light up with a red indicator when I get a new message. All the actions start an Intent which launches my Messenger app, though. So yeah, it's probably all dead code that's still there.
brazzledazzle 11y ago
>It's all a bunch of bored engineers looking for ways to be clever but without anyone to take ownership of the codebase. You couldn't think of a more charitable interpretation? I'm not one to care about rules but the hn guidelines on commenting are important for making sure the community doesn't eat itself alive like reddit did/does. >The app itself doesn't do that much. Right, but in the presentation he made it clear that it was less about what the app does and more about the number of engineers making changes to it. Given the size of their teams all making their own changes to the iOS codebase, doesn't it make sense to modularize it so they aren't tripping over each other breaking the application constantly? They don't have a dedicated iOS team and having one would bottleneck the other teams.

bsaul 11y ago

So i guess maybe Apple didn't code their ios and ios apps using xcode, xcode-git, autolayout and coredata after all. Because their scale seems pretty big too.

colinbartlett 11y ago

I loved the anecdote starting on page 41 where he accidentally "memorializes" his Facebook profile thus triggering an email to his wife about his death.

chuckcode 11y ago

Would be interested to see some comparison with the Android app which seems to have most of the same functionality at 1/4 the size with 25MB [1] compared to the 100MB of iOS. Seems like either iOS team at FB has significant cruft laying around or iOS doesn't scale as well as Android and FB is hacking around that. Possibly a bit of both but think it would be an interesting comparison in addition to the war stories about hacking their entire tool chain. [1] https://gigaom.com/2015/01/26/facebooks-new-android-app-goes-light-on-the-data/ https://gigaom.com/2015/01/26/facebooks-new-android-app-goes...

dmishe 11y ago
iOS apps are shipped as fat binaries, so for one, part of that 100mb app is duplicated slices of binaries for armv7, arm64 and possibly, but not likely, armv7s. Android apps enjoy bytecode that is getting JIT-ed on install.
- dgallagher 11y ago
  Apple has introduced App Thinning in iOS 9 apps which should cut down on their size: https://developer.apple.com/library/watchos/documentation/IDEs/Conceptual/AppDistributionGuide/AppThinning/AppThinning.html https://developer.apple.com/library/watchos/documentation/ID...
  - azsromej 11y ago
    The feature was recently delayed, but it should help once it's ready
- notNow 11y ago
  Is there any technical justification for this or it's purely economical/financial?
  - dmishe 11y ago
    For what, fat binaries? That's pretty much what Mach-O format had from inception under NeXTSTEP, and survived until now. As it was said in the sibling comment, iOS 9 is introducing a way to cut exactly the slice you need for a particular device.

brodd 11y ago

Am I reading this right, there are interns and engineers running PHP in a debugger on production data? How else would he be able to memorialize his own account and trigger the email?

randomsearch 11y ago
That is a good point, how did no-one else pick up on that???! It reminds me of engineers where I used to work, who would run through a £0.01 transaction on their credit card to check the changes to the charging system were still working...
thrusong 11y ago
I've seen talks with engineers that say this is this case. I think the one in particular was from @Scale. They've had the occasional issue with an intern dropping a table in production, but they said they haven't really run into issues with granting access to production data.

gorkemyurt 11y ago

(Another post about React Native) Facebook engineers keep telling everyone that React Native is not used in the main Facebook app. I always wondered what is stopping them from start using it. Perhaps app size is a massive concern? Maybe all this tooling around their app is not really compatible with React Native? Its interesting that they have a ton of legacy Obj-c code (considering that companies started to build apps not very long time ago, and they already re-wrote their app several times) I wonder if any other company is a situation like this well other than Apple.

zghst 11y ago
It would probably be a huge pain to migrate all 18,000 classes, tests, tools, etc., over to a React Native app. Also JS has less fine grained tooling for performance profiling (try stepping through JS bytecode for performance analysis, it's a PITA). One day it might get there, but not today. If it works, why change it?
- gorkemyurt 11y ago
  I am not talking about migrating the whole app but how about for new features? I noticed that they redesigned their friends request page, why not build that with react native?
  - plorkyeran 11y ago
    The engineers who did that redesign didn't want to use react native or felt it wasn't a good choice. Just because one person at Facebook built a tool doesn't mean that everyone else at Facebook is suddenly required to use that tool where ever possible.