17 ms·

Facebook’s code quality problem (2015)

502 points by setra 10y ago

neals 10y ago
[2015]
BinaryIdiot 10y ago
As pointed out this is from 2015 but I'd love to hear some updated. Has any Facebook employee created newer presentations that discussed, say, their 18,000 iOS classes and the fact that in a single week 429 people contributed to the same iOS app? I'd also love to hear an update to the "Our site works when the engineers go on holiday" claim.
- arjie 10y ago
  I'd bet that the 18000 classes is just Thrift-generated stuff or whatever.
  - panic 10y ago
    You don't have to bet, you can see them here: http://quellish.tumblr.com/post/126712999812/how-on-earth-the-facebook-ios-application-is-so http://quellish.tumblr.com/post/126712999812/how-on-earth-th...
    - rimliu 10y ago
      Even with a list my imagination is still not powerful enough to understand how an app like FB needs 18000 classes :(
      - mmjaa 10y ago
        It needs them .. to keep all those programmers busy.
    - Sacho 10y ago
      what's up with the ZR* and ZZ* classes?
- icebraining 10y ago
  I'd also love to hear an update to the "Our site works when the engineers go on holiday" claim. Isn't this an inevitable outcome once your systems are actually reliable? I've been reading the Google SRE book and they make the same point: every new feature (read: change) is risky. Hence they delay deployments once their "error budget" starts running low. Seems to me that the only systems that don't work more reliably when the engineers are out are the ones which require hands on deck just to keep them running.
dep_b 10y ago
code quality can't handle our scale
userbinator 10y ago
It is ironic that a codebase with so many classes is likely to occur precisely because of dogmatic adherence to those "best practices" which were originally intended to improve code quality --- aggressive refactoring/modularisation being a likely culprit. What I think they really need is not more design, not more architecture, not more of anything but KISS and YAGNI.
- afarrell 10y ago
  The thing that having an overall architecture buys you is conceptual clarity. That lets someone much more easily understand what is going on. You need to have a minimal set of ideas that are well thought out and explained. Just saying "YAGNI" and "KISS" doesn't actually make the resulting code simpler. Why? You do have some things to build and refusing to lay out a plan for what that is going to be doesn't stop that. It just means that the complexity doesn't have a clear narrative
- sidlls 10y ago
  It seems to me your point about KISS and the author's observations are compatible. Facebook is successful for reasons that have almost nothing to do with their code, however, and as long as that's true there isn't a reason for them to change how they develop their software. In that sense, code quality doesn't actually matter (at least, until it does).
sushobhan 10y ago
I don't think all the things discussed are still relevant as it's a fairly old post. Also, "best practices" are not always best depending on the scalability of the problem itself.
_pmf_ 10y ago
If a company hails React as a sane way to develop software, you know where it stands.
- peletiah 10y ago
  Just trolling or could you argue that?
- qaq 10y ago
  OK so what should have they used instead?
sz4kerto 10y ago
It's interesting to note though that they have code quality problems but that doesn't prevent them from being massively successful.
- bsaul 10y ago
  It's a very hard to swallow pill for engineers, but many (if not most) startup or IT company success or failure is very independant of technological matters. Once you get to "average" quality, then business and product decisions become far more important than anything else.
  - Gracana 10y ago
    And it only needs to be average, or approaching average, in your niche. There's a lot of truly awful enterprise software out there, and it gets sold on promises of features and support, not quality. I wish software engineering had the barriers and requirements of other engineering fields. I deal with industrial control systems, and we have the national electric code, UL, CE, NFPA 79, etc. With all that you can still make a machine that doesn't work well, but at least it's going to conform to certain conventions to make installation and maintenance easier, it will have a robust safety system, and it will be well-documented. Most fields of software engineering have no such minimum standards.
    - agentultra 10y ago
      > I wish software engineering had the barriers and requirements of other engineering fields. It does where I live you're required to be licensed if what you're doing passes these 3 criteria: 1. any act of planning, designing, composing, evaluating, advising, reporting, directing or supervising (or the managing of any such act) 2. that requires the application of engineering principles 3. concerns the safeguarding of life, health, property, economic interests, the public welfare or the environment, or the managing of any such act. This includes software development. It is illegal to use, "engineer," in your professional title unless your accredited. I think this is a good thing. Better software that works towards the public interest requires liability and professional accreditation. You can't take the, "move fast, break stuff" approach when your software is being used to monitor and control water filtration plants or food safety processes in a manufacturing facility. That works for the Facebook's of the world but they are by no means the be all and end all of professional software development.
rumcajz 10y ago
I believe "our site works when the engineers go on holiday" thing is not fair. Of course that application is less stable when it's actively modified. There's no way to make it more reliable on weekdays than on weekends except for stopping the development altogether or maybe deploying on weekends.
- panic 10y ago
  What other engineering discipline would say "there's no way to improve our reliability except stopping work altogether"? There's always a way to improve reliability. Arguably, with formal verification, you could ensure large parts of your system are perfectly reliable given simple assumptions. The problem isn't that it's impossible -- it's that it's more expensive than just hiring one more engineer to keep papering over the problems.
  - icebraining 10y ago
    What other engineering discipline would say "there's no way to improve our reliability except stopping work altogether"? All of them. Are your roads more reliable when they're constantly being changed or when they are just being maintained? Is NASA achieving its reliability by constantly changing the designs of their ships, or by reusing the same design over and over? Arguably, with formal verification, you could ensure large parts of your system are perfectly reliable given simple assumptions. Yes. But a fixed formally verified system will still be more reliable than a formally verified system being constantly changed. What was said wasn't that FB couldn't be more reliable. It's that they are already so reliable that only new changes introduce problems. Sure you can still work on minimizing those problems, but that's a different point.
    - panic 10y ago
      I think I misread the OP. You're right that there's no straightforward way to make Facebook more reliable on weekdays than on weekends.
- dsr_ 10y ago
  No way at all? FB has a lot of servers and a lot of users. They have opportunities for quality practices that few other organizations get. Here's a simple example: Split the user population into 365* groups. Assign them to days of the year. Test new code only the group whose day has come up. Follow them until they stop having problems. Now you can deploy that to a month's worth of groups. All good? Deploy to everyone. Yes, that means that you can't have more than 365 changes in simultaneous development. Tough. *Yes, yes, leap years. Take a day off from deploying.
aamederen 10y ago
I believe, from time to time, you just need to stop developing new stuff for a period of time and dedicate the team to improve the overall quality, pay technical depts, hunt bugs, remove legacy code and refactor still existing ones.
- muse900 10y ago
  I agree with you. This is one of the main issues small-medium scale companies I've worked for have, they just have a dev team that has to keep on creating stuff constantly. Although I've never worked on a company with the size of FB, and I believe the same logic wouldn't apply to them. They have to keep creating stuff, as fb still has a lot of potential on becoming way more profitable.
- icebraining 10y ago
  Sounds essentially like GC. And like in GCs, it seems preferable to have pause-free processes than stop-the-world.
- martinknafve 10y ago
  Why not plan tasks like that continously and make it part of your normal iterations?
- Cthulhu_ 10y ago
  Even so, with an organization the size of Facebook, it'd still cause a lot of churn. I don't know if there's any hierarchy there, like, lead developers and architects that can align architecture and maintain quality and consistency.
- maxlybbert 10y ago
  I worked at a place where non-technical management didn't like hearing things had to be refactored occasionally. So the technical managers became very good at including refactoring of old code in estimates for new (related) features. This generally worked, except when a lot of refactoring was needed. Occasionally, we would learn after the fact that one of our designs, programs, etc. was fundamentally flawed in some way, but we couldn't fix it with a series of moderate changes. Those cases really need a dedicated refactoring effort.
peletiah 10y ago
Makes me wonder how their decision for an open-plan office is related with this...
amelius 10y ago
It also seems that improvements in code quality caused by React happened not because they were driven by management, but because a few "heroic" engineers thought it was an interesting side-project. At least, that's my impression from the publicly available talks they gave.
_pmf_ 10y ago
> The article moves on, without wondering whether releases regularly breaking your app are a normal part of the software engineering process. To be fair, that's absolutely their call to make. Nothing of value is ever lost if your platform does not provide any non-ephemeral value.
paganel 10y ago
I read Richard Gabriel's "Worse Is Better" essay in my first month hired as a professional programmer, that was almost 12 years ago. Since that time the lessons of that essay have only increased in significance for me, with this FB story as additional proof. Their back-end code might be bollocks, and I can certainly believe that judging by how sluggish their FB app feels on my phone, but the fact is that they've conquered the Internet (together with Google and a couple of other companies). It's a fact that I personally hate, but they're still winners in the end.
- aaron-lebo 10y ago
  But it's not better, it is worse as clearly put forward by this article. ;) We shouldn't attribute Facebook's success to poor quality code, unless we really believe that 429 people working that that slow sluggish app is productive. Do you? Maybe Facebook would be in a better situation than they are if they had a firm foundation? What happens when a competitor comes along and knocks on their door? That mess of code becomes an anchor. They succeeded not because "worse is better" but because ...they succeeded, maybe because they had the greatest social network in the world to expand within, or because of some other reason we don't know. But they didn't succeed because of worse is better, they succeeded in spite of it.
  - paganel 10y ago
    > Do you? It's written between the article's lines, so to speak, but it seems like the bad code quality was generated in part by FB moving at a very fast pace. I think they wouldn't have been as successful as they are had they chosen a slower path with more "correct" code and architecture. > What happens when a competitor comes along and knocks on their door? They successfully managed to "eat" and integrate both Instagram and WahtsApp, both worthy contenders at the time, and they're now on their way to stealing Snap's lunch, so they've proved to be pretty resilient on this front. Not to forget how they successfully kept Google+ in check.
  - jheriko 10y ago
    how would that success have been different if they had used different engineering practices? a big part of their success was delivering results... if they hadn't quickly crapped everything out of the door, how would facebook look today?
segmondy 10y ago
So? I've become very cynical since the past year or so because there is a lot of noise/articles on the net, all knowing what's best for you or wrong with what you are doing. Yet without context or true authority to talk about it. My question these days when I read such article, "What has the author done that is equivalent?" There is rarely anything to be found. Facebook might have a "code quality problem." But until you have worked at an organization that is big like Facebook please hold forth your tongues or fingers in this case. Startup principles don't apply nor academia. Facebook's "code problem" is what really does happen in the real world with such humongous business enterprises. Lot's of moving pieces in code, in people, in ideas and the amalgamation of these results in what most programmers will see as "code quality problem" yet the market sees as billions of dollars.
- aaron-lebo 10y ago
  Isn't that an appeal to authority? Because someone hasn't done X they don't have the right to criticize X? Facebook's "code problem" is what really does happen in the real world with such humongous business enterprises. But does it have to be this way?
  - evgen 10y ago
    Until someone provided s counter-example the assumption is that yes, this is just what happens. We have entire industries of competing methodologies, some almost cult-like in their slavish devotion to "the one true process", and none have birthed any great successes. Why is that?
    - deleted 10y ago
      [deleted]
    - dustingetz 10y ago
      History is littered with counter-examples and great successes. Each generation of company is more efficient or solves more complicated problems than those before it. Facebook itself is an industry leader in open source innovation, which is why their product is so smooth. But not all companies specialize in hard tech, and all companies get old.
jheriko 10y ago
one look at hhvm tells me they have poor quality development practices. :P i get the impression that they use the struct keyword to avoid having to type public everywhere for instance...
watwut 10y ago
Reality check is that most companies, including most startups, attempting to create software of such size end up tangled in their own mess way sooner and fail. Keeping it maintainable at such scale is not as easy as keeping it maintainable when there are four of you.
planetjones 10y ago
Well when there is a codebase of the scale of Facebook's, with so many activate contributors of course there will be a code quality problem. I see so much stuff at Google breaking or becoming inconsistent too (away from the core search features). Even if they didn't move fast and break things, they would still have code quality issues. It's just the nature of a huge codebase like they have. And let's face facts - the development process needs to be suitable to the type of problem being solved. This is not a life and death situation and users would value new features, more than extreme reliability and consistency.
fishnchips 10y ago
I spent a few months at Facebook and left because of that. Not necessarily because of poor code quality but because of all the issues it was causing, and a culture of turning a blind eye to (or worse still, being proud of!) those issues. It's obviously true that quality of their code does not affect their market position, but it sure affects who they hire and who they retain.
- joeletizia 10y ago
  That may be true. But in the end, does it matter? Until we see someone taking market share away, I'd argue it doesn't. Facebook still retains top tier people regardless of any perceived quality problems. Sorry you left. I just started here, and I do agree that what I feel about code quality isn't generally shared here. But as I get older, I start realizing that in the end, as long as things work relatively well at our scale and in our problem domain (we're not medical device software, we can fail) alls well.
  - feedjoelpie 10y ago
    I did some contract work for FB around when this article was written. I don't think that nobody cares about clean code. It's more like... results will vary widely based on who reviews you. And the governing philosophy is more focused on the automated checks and balances that try to keep you from doing harm to the overall system. While human code reviews were pretty lax, the machines would slap you around for all sorts of things, none of which would force you to write beautiful code. Personally, I like for humans to be pickier than they seemed to be at Facebook, but it really opened my eyes to the value of what I think of as a "keep your garbage in your own trash can" approach to quick-and-dirty coding. As long as you didn't spill your trash into the hallway, the machines were satisfied, and your reviewer didn't find your work to be _insane_, your work shipped. [EDIT: And I find that this can often be a wonderfully pragmatic way to work if you have the tools to keep you from doing harm.]
  - Sevii 10y ago
    It matters because profits are lower. 400 devs on an iOS app is not free.
ptero 10y ago
Code quality is important, but for an organization the size of the FB good systems engineering is much more important than good software engineering. If the overall system is organized sanely, failures of subcomponents (which happen due to software bugs, hardware, humans, etc.) can be isolated, firewalled and fixed in reasonable time without bringing the system down.
chiefalchemist 10y ago
Move fast...and just leave things broken. Users be damned.
chiefalchemist 10y ago
But seriously (i.e., ref to my previous comment). All the tools. All the talent. All the leadership and management. All the money. And still even the mighty FB struggles. Point being: This shit might not be rocket scientist hard, but it ain't easy either. And when you don't have the war chest of the likes of FB you're in for an ongoing and never ending (quality) battle.
raygelogic 10y ago
this strikes me the same way as the story of the guys who figured out where to add armor to bombers during WW2. the places on the plane that had bullet holes after a mission clearly were operational enough to return to base. the places that never had any damage were probably critical, so that's where armor was added. if a company can make billions with a poor-quality codebase, clearly quality isn't a bottom-line concern. what is a concern is shipping the damn product.
dawidloubser 10y ago
I would like to argue that, in most of these cases, it's not "code quality" that is at fault, but "design quality" (before coding) - which is often absent entirely. The problem with design, in software, is not that most people forget to do it. It's that they never learn to do it. It always comes back to bite you. I don't want to start a discussion on design, and how most people mess it up because of lack of skill or experience therein. But hacker culture seems to be allergic to design, and hacker culture seems to be what everybody strives for these days.
- lefty2 10y ago
  I think you hit the nail on the head. My motto for developing code is "make it simple as possible, but no simpler". Whoever, created a design that used 18,000 classes did not follow that rule.
typetypetype 10y ago
What's the point of an app? Generating high quality code or generating revenue? Until you hit intersections where revenue can be increased by changing code, the priority will always be on new or changed features.
baggachipz 10y ago
> The “Hack” and “Move fast and break things” culture must make it very hard for developers to focus on quality. Well, yeah, that's what the "break things" part means. The problem is when people/companies try to have it both ways. "Move fast and have high quality" isn't possible.
thedevil 10y ago
It's hard to argue with the business sense of pushing out features quickly at the cost of code quality. But I've learned that that there are two kinds of development teams: (A) the teams that are "moving fast and breaking things" while "creating business value" and (B) the teams that are "only maintaining legacy code", "not getting anything done", "costing a lot of money" and "complaining too much" about code quality that team A wrote. As an engineer, I've learned that it's less work and more rewarding to be on the (A) teams than on the (B) teams.
- nickpeterson 10y ago
  The difference is that Team A can leave and a project is delayed. If Team B all left the company would fold.
  - ceronman 10y ago
    Both are necessary. If Team A leaves, and no more business value is produced, and the devs only care about beautiful code, the company will fail too. Both kinds of teams are needed. Or even better, teams should not be divided like this, everyone should do their shares of both breaking things, and creating value and cleaning the mess and scaling the things.
  - scaryclam 10y ago
    Funnily I've actually seen a Team B leave. There was much panic and thinking it would all go horribly wrong, and indeed it was madness for the first week. Then the people who took over started to stabilise some of the worst parts of the system and it all sort of fell into place after that. Fast forward a couple of months, the project was pretty much hands off and we all realised that keeping a whole team on that project had pretty much been a giant waste of money. They did nothing but create fires for themselves to fix. The Team B's exodus was actually a really great thing for the company. I suppose the thing to learn here is, that you need redundancy in your teams. If we hadn't had other devs to step in and take over, it would have been a bit different. Still, I'm not convinced that a handful of contractors couldn't have been brought in to the same result, just more expensive.
mybrid 10y ago
It is worth noting that when something goes viral we still don't know why. Facebook represents the ultimate in viral success. Given we do not understand why things go viral then it stands to reason that the underlying software methodology is not causation. This means that the fickle finger of fate could easily be visited upon Facebook the same as with MySpace and Facebook will be in no control. This is independent of the any software methodology they use. Claims made without evidence can be dismissed without evidence.
strken 10y ago
Can someone explain the example with the ramdisk? It's not clear to me how a ramdisk is relevant to the problems put forward in the paper, i.e. disk is too slow, they rely on lazy page allocation, and they have insufficient ram to hold two copies of everything simultaneously.
alex- 10y ago
> "when Facebook employees are not actively making changes . . . the site experiences higher levels of reliability." This seems like a great thing to me. i.e. The system is stable and the error budget is being used to facilitate change.
dwils098 10y ago
links to slides are all dead... in parent articles and reference articles
bobbyi_settv 10y ago
> The second exhibit is from Facebook Research... wouldn’t you just write your disk files to a ramdisk? Surely they noticed this too I'm not sure the author understand what a Research team is and what it does. Trying out a new solution to an existing problem is sort of their job. I'm also not sure how a research team publishing a paper discussing an alternate solution indicates anything about the company's "code quality".
Bahamut 10y ago
Anecdotal evidence, but I have a friend who complained that some critical parts don't have tests, and a lot of developers are loathe to write them. He got bit by it when he made a change in an area with no test coverage and broke the spam filter.
tomkin 10y ago
It's important to understand that Facebook is incredibly large, and the fact that any software scaled with them to the point they are presently at, is a "on the shoulders of giants" situation that seems to actually be working well enough for them to become a billion dollar enterprise. Also, the building of infrastructure that takes on scale that's not yet been required is somewhat inefficient. But it does beg the question: If Facebook is limited by its infrastructure, what responsibilities do they have to build software that continues to scale for future organizations?
p0nce 10y ago
At this point it is clear that software size increase linearly with the size of your software team, whatever the problem you solve. This affects every organization and is something that should be actively fight against. Having accidental, unplanned, unaccounted costs should not be the default path.
charleslmunger 10y ago
"Exhibit C: Our site works when the engineers go on holiday" Is not all so different from "Fewer patients die when heart surgeons are on vacation". Of course your site is going to be more reliable when nobody is changing it! You should be worried if reliable isn't the steady state, and it requires constant changes to stay up!
- afc 10y ago
  I also took issue with this part of the article; it seems obvious to me that rolling out new features would directly cause reliability issues (and the only alternative, of not rolling out new features, is, obviously, a cure worse than the disease).
  - adrianratnapala 10y ago
    I am not sure that most Facebook users would actually agree about this. But then their view is slanted towards UI changes. Performance improvements and other subtleties are features too.
- joshvm 10y ago
  My immediate thinking was this as well: when they stop touching it, it's fine, which means that whatever bugs are checked in seem to be caught quickly. What's more, the periods of high reliability are during periods when you might expect there to be heavy load - assuming people don't check Facebook at work (har har). What counts as an incident? > Figure 1 includes data from an analysis of the timing of events severe enough to be considered an SLA (service-level agreement) violation. Each violation indicates an instance where our internal reliability goals were not met and caused an alert to be generated. Because our goals are strict most of these incidents are minor and not noticeable to users of the site. So these are minor issues. The parent article is paraphrasing at best, and jumping to conclusions at worst. From Facebook: > We believe this is not a result of carelessness on the part of people making changes but rather evidence that our infrastructure is largely self-healing in the face of non-human causes of errors such as machine failure. They then list a number of sane strategies to mitigate this. http://queue.acm.org/detail.cfm?id=2839461 http://queue.acm.org/detail.cfm?id=2839461
zo7 10y ago
Some of FB's ~18,000 Obj-C header files from a post linked in the article: [1] * FBFeedAwesomizerProfileListCardViewControllerListenerAnnouncer.h * FBBoostedComponentCreateInputDataCreativeObjectStorySpecLinkDataCallToActionValue.h * FBEventUpdateNotificationSubscriptionLevelMutationOptimisticPayloadFactoryProtocol-Protocol.h * FBGroupUpdateRequestToJoinSubscriptionLevelMutationOptimisticPayloadFactoryProtocol-Protocol.h * FBMemReactionAcornSportsContentSettingsSetShouldNotPushNotificationsResponsePayloadBuilder.h * FBProfileSetEventsCalendarSubscriptionStatusInputDataContextEventActionHistory.h * FBReactionUnitUserSettingsDisableUnitTypeMutationOptimisticPayloadFactoryProtocol-Protocol.h oh my god [1]: http://quellish.tumblr.com/post/126712999812/how-on-earth-the-facebook-ios-application-is-so http://quellish.tumblr.com/post/126712999812/how-on-earth-th...
- aljones 10y ago
  Those read like classes generated by tools. Or maybe I really don't want to think humans are capable of such names.
  - utexaspunk 10y ago
    Well, the humans could be tools...
  - oriolid 10y ago
    These are completely credible class names, just ask any Enterprise Java developer. I particularly enjoy FactoryProtocol-Protocol. Who would be satisfied with just a FactoryProtocol?
    - ryandrake 10y ago
      Yea, I was going to say, Java's almost got them beat with SimpleBeanFactoryAwareAspectInstanceFactory[1] 1: http://docs.spring.io/spring/docs/2.5.x/javadoc-api/org/springframework/aop/config/SimpleBeanFactoryAwareAspectInstanceFactory.html http://docs.spring.io/spring/docs/2.5.x/javadoc-api/org/spri...
      - 10y ago
jyriand 10y ago
Only Kent Beck can save them. :)
whack 10y ago
I do think that FB, and most large orgs, have code quality problems. But the article does a pretty bad job at making its case. > "That’s 429 people working, in some way, on the Facebook iOS app. Rather than take the obvious lesson that there are too many people working on this application, the presentation goes on to blame everything from git to Xcode for those 18,000 classes." How does the author know that 429 is too many? How does the author know that FB's goals and functionality can be best achieved with fewer people/classes. This just reads like a classic "Why is Google so big, I can do that in a weekend" comment (https://danluu.com/sounds-easy/ https://danluu.com/sounds-easy/) > "Our site works when the engineers go on holiday" This is pretty much universally true for any dynamic site where engineers are constantly adding new features. Change always comes with risks. Changing code is always more likely to break something in the very-short-term, compared to not changing code. I have a hobby site (shameless plug, http://thecaucus.net http://thecaucus.net) which has been running in auto-pilot for the past year, and almost never breaks, because there are virtually no moving pieces. The fact that the FB site breaks more often when engineers are making changes to it, is just repeating a universal law of software development. I do think that the organizational structures at most large companies are bloated, inefficient, non-transparent, and produce sub-par code. I had high hopes when I read the article's headline, but the arguments presented simply aren't very persuasive.
combatentropy 10y ago
I look at programming as all design, whether deliberate or not. Martin Fowler drove the point home for me (https://www.martinfowler.com/articles/newMethodology.html https://www.martinfowler.com/articles/newMethodology.html). He said that some people drew inspiration from construction, where there are designers and builders. One or two highly paid architects draw up the plans. Then you can hire a bunch of cheap labor to build it, say a bridge. This belief leads to a dichotomy in software companies, where one person is the "architect" and others are just "regular" programmers --- often outsourced to the lowest bidder. From Jack Reeves he cites the epiphany "that in fact the source code is a design document and that the construction phase is actually the use of the compiler and linker." There is little repetitive, mindless work in programming, because as most of you know, "anything that you can treat as construction can and should be automated." Therefore, "In software all the effort is design . . ."
- krick 10y ago
  I feel your sentiment, and usually oppose to the construction-driven approach you describe, but it is really easy to play devil's advocate here. I mean, let's just be realistic. There's nothing sacred in programming. The fault in your argument is that there's no design/construction dichotomy, but rather a gradient: everything can be viewed upon as a design from close enough distance, and everything can be a construction looking from the high above. For business people all the programming is just very, very low-level construction, and there's no design to it. Damn, even hiring programmers and creating companies is just a construction stage to them. And from the practical standpoint, the truth is that perceiving a lot of programming "just as a construction" just works. For the better or worse. It is cheaper and usually reliable enough to design the system in a way that a lot of the components would be plugins not depending one on the other that can be outsourced to anyone. Even if some fail, the system continues to work while you hire new people to rewrite some components. Sure, maybe it would be more efficient and "cool" to hire a really smart guy (or do it yourself, if you believe you can) to write it as really optimized monolith system, but if this isn't required, it is usually less risky to build something not so sophisticated, and improve it if need be. It is usually ok, that some parts of the system may fail. Fuck it, let's not lie to ourselves: there always are some parts of the system that fail, whether you think it's "acceptable" or not, let's just take it into account.
norswap 10y ago
I was doing iOS dev three years ago, and the Facebook lib for iOS was the worst piece of shit I ever had to deal with. Their app also used to be unbearably sluggish despite not doing anything too fancy.
- babesh 10y ago
  I thought the joke was that they had interns write that? I remember an API that would hold onto completion blocks and call them repeatedly instead of just once. They would also accidentally make non backwardly compatible changes to the API. They didn't care since that API period propelled them to more dominance and growth for awhile. After they shut down the viral channels, the developer ecosystem died...but it had proven its usefulness. The only issue now is developers don't trust them and they can't get enough developers to build atop them.
linkmotif 10y ago
While the author wrote this article FB made millions of dollars.
badpenny 10y ago
I recommend reading They Write the Right Stuff[1] to anybody interested in NASA's approach to writing software. [1] https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff
deleted 10y ago
[deleted]
ChrisRus 10y ago
> From Jack Reeves he cites the epiphany "that in fact the source code is a design document and that the construction phase is actually the use of the compiler and linker." Ha. That is laughable. If it were so, then we would be able to automatically graft features from one product to another. I contend this is exactly how SoC are designed and there's no reason why we shouldn't copy the methods of the digital design community (proven to scale or you wouldn't be reading this).
aqibgatoo 10y ago
bob martin a.k.a uncle bob always says that people consider patterns like mvvm,mvc,mvp etc as Architecture which actually are just for the ui layer.It happens commonly in mobile development.
pablo_fiesta 10y ago
Rub lavender oil on the servers, say "web-scale" 3 times, click you heals and everything will be OK. Oh wait, but the Russians.

502 points by setra 10y ago

neals 10y ago

[2015]

As pointed out this is from 2015 but I'd love to hear some updated. Has any Facebook employee created newer presentations that discussed, say, their 18,000 iOS classes and the fact that in a single week 429 people contributed to the same iOS app? I'd also love to hear an update to the "Our site works when the engineers go on holiday" claim.

arjie 10y ago
I'd bet that the 18000 classes is just Thrift-generated stuff or whatever.
- panic 10y ago
  You don't have to bet, you can see them here: http://quellish.tumblr.com/post/126712999812/how-on-earth-the-facebook-ios-application-is-so http://quellish.tumblr.com/post/126712999812/how-on-earth-th...
  - rimliu 10y ago
    Even with a list my imagination is still not powerful enough to understand how an app like FB needs 18000 classes :(
    - mmjaa 10y ago
      It needs them .. to keep all those programmers busy.
  - Sacho 10y ago
    what's up with the ZR* and ZZ* classes?
icebraining 10y ago
I'd also love to hear an update to the "Our site works when the engineers go on holiday" claim. Isn't this an inevitable outcome once your systems are actually reliable? I've been reading the Google SRE book and they make the same point: every new feature (read: change) is risky. Hence they delay deployments once their "error budget" starts running low. Seems to me that the only systems that don't work more reliably when the engineers are out are the ones which require hands on deck just to keep them running.

dep_b 10y ago

code quality can't handle our scale

userbinator 10y ago

It is ironic that a codebase with so many classes is likely to occur precisely because of dogmatic adherence to those "best practices" which were originally intended to improve code quality --- aggressive refactoring/modularisation being a likely culprit. What I think they really need is not more design, not more architecture, not more of anything but KISS and YAGNI.

afarrell 10y ago
The thing that having an overall architecture buys you is conceptual clarity. That lets someone much more easily understand what is going on. You need to have a minimal set of ideas that are well thought out and explained. Just saying "YAGNI" and "KISS" doesn't actually make the resulting code simpler. Why? You do have some things to build and refusing to lay out a plan for what that is going to be doesn't stop that. It just means that the complexity doesn't have a clear narrative
sidlls 10y ago
It seems to me your point about KISS and the author's observations are compatible. Facebook is successful for reasons that have almost nothing to do with their code, however, and as long as that's true there isn't a reason for them to change how they develop their software. In that sense, code quality doesn't actually matter (at least, until it does).

sushobhan 10y ago

I don't think all the things discussed are still relevant as it's a fairly old post. Also, "best practices" are not always best depending on the scalability of the problem itself.

_pmf_ 10y ago

If a company hails React as a sane way to develop software, you know where it stands.

peletiah 10y ago
Just trolling or could you argue that?
qaq 10y ago
OK so what should have they used instead?

sz4kerto 10y ago

It's interesting to note though that they have code quality problems but that doesn't prevent them from being massively successful.

bsaul 10y ago
It's a very hard to swallow pill for engineers, but many (if not most) startup or IT company success or failure is very independant of technological matters. Once you get to "average" quality, then business and product decisions become far more important than anything else.
- Gracana 10y ago
  And it only needs to be average, or approaching average, in your niche. There's a lot of truly awful enterprise software out there, and it gets sold on promises of features and support, not quality. I wish software engineering had the barriers and requirements of other engineering fields. I deal with industrial control systems, and we have the national electric code, UL, CE, NFPA 79, etc. With all that you can still make a machine that doesn't work well, but at least it's going to conform to certain conventions to make installation and maintenance easier, it will have a robust safety system, and it will be well-documented. Most fields of software engineering have no such minimum standards.
  - agentultra 10y ago
    > I wish software engineering had the barriers and requirements of other engineering fields. It does where I live you're required to be licensed if what you're doing passes these 3 criteria: 1. any act of planning, designing, composing, evaluating, advising, reporting, directing or supervising (or the managing of any such act) 2. that requires the application of engineering principles 3. concerns the safeguarding of life, health, property, economic interests, the public welfare or the environment, or the managing of any such act. This includes software development. It is illegal to use, "engineer," in your professional title unless your accredited. I think this is a good thing. Better software that works towards the public interest requires liability and professional accreditation. You can't take the, "move fast, break stuff" approach when your software is being used to monitor and control water filtration plants or food safety processes in a manufacturing facility. That works for the Facebook's of the world but they are by no means the be all and end all of professional software development.

rumcajz 10y ago

I believe "our site works when the engineers go on holiday" thing is not fair. Of course that application is less stable when it's actively modified. There's no way to make it more reliable on weekdays than on weekends except for stopping the development altogether or maybe deploying on weekends.

panic 10y ago
What other engineering discipline would say "there's no way to improve our reliability except stopping work altogether"? There's always a way to improve reliability. Arguably, with formal verification, you could ensure large parts of your system are perfectly reliable given simple assumptions. The problem isn't that it's impossible -- it's that it's more expensive than just hiring one more engineer to keep papering over the problems.
- icebraining 10y ago
  What other engineering discipline would say "there's no way to improve our reliability except stopping work altogether"? All of them. Are your roads more reliable when they're constantly being changed or when they are just being maintained? Is NASA achieving its reliability by constantly changing the designs of their ships, or by reusing the same design over and over? Arguably, with formal verification, you could ensure large parts of your system are perfectly reliable given simple assumptions. Yes. But a fixed formally verified system will still be more reliable than a formally verified system being constantly changed. What was said wasn't that FB couldn't be more reliable. It's that they are already so reliable that only new changes introduce problems. Sure you can still work on minimizing those problems, but that's a different point.
  - panic 10y ago
    I think I misread the OP. You're right that there's no straightforward way to make Facebook more reliable on weekdays than on weekends.
dsr_ 10y ago
No way at all? FB has a lot of servers and a lot of users. They have opportunities for quality practices that few other organizations get. Here's a simple example: Split the user population into 365* groups. Assign them to days of the year. Test new code only the group whose day has come up. Follow them until they stop having problems. Now you can deploy that to a month's worth of groups. All good? Deploy to everyone. Yes, that means that you can't have more than 365 changes in simultaneous development. Tough. *Yes, yes, leap years. Take a day off from deploying.

aamederen 10y ago

I believe, from time to time, you just need to stop developing new stuff for a period of time and dedicate the team to improve the overall quality, pay technical depts, hunt bugs, remove legacy code and refactor still existing ones.

muse900 10y ago
I agree with you. This is one of the main issues small-medium scale companies I've worked for have, they just have a dev team that has to keep on creating stuff constantly. Although I've never worked on a company with the size of FB, and I believe the same logic wouldn't apply to them. They have to keep creating stuff, as fb still has a lot of potential on becoming way more profitable.
icebraining 10y ago
Sounds essentially like GC. And like in GCs, it seems preferable to have pause-free processes than stop-the-world.
martinknafve 10y ago
Why not plan tasks like that continously and make it part of your normal iterations?
Cthulhu_ 10y ago
Even so, with an organization the size of Facebook, it'd still cause a lot of churn. I don't know if there's any hierarchy there, like, lead developers and architects that can align architecture and maintain quality and consistency.
maxlybbert 10y ago
I worked at a place where non-technical management didn't like hearing things had to be refactored occasionally. So the technical managers became very good at including refactoring of old code in estimates for new (related) features. This generally worked, except when a lot of refactoring was needed. Occasionally, we would learn after the fact that one of our designs, programs, etc. was fundamentally flawed in some way, but we couldn't fix it with a series of moderate changes. Those cases really need a dedicated refactoring effort.

peletiah 10y ago

Makes me wonder how their decision for an open-plan office is related with this...

amelius 10y ago

It also seems that improvements in code quality caused by React happened not because they were driven by management, but because a few "heroic" engineers thought it was an interesting side-project. At least, that's my impression from the publicly available talks they gave.

_pmf_ 10y ago

> The article moves on, without wondering whether releases regularly breaking your app are a normal part of the software engineering process. To be fair, that's absolutely their call to make. Nothing of value is ever lost if your platform does not provide any non-ephemeral value.

paganel 10y ago

I read Richard Gabriel's "Worse Is Better" essay in my first month hired as a professional programmer, that was almost 12 years ago. Since that time the lessons of that essay have only increased in significance for me, with this FB story as additional proof. Their back-end code might be bollocks, and I can certainly believe that judging by how sluggish their FB app feels on my phone, but the fact is that they've conquered the Internet (together with Google and a couple of other companies). It's a fact that I personally hate, but they're still winners in the end.

aaron-lebo 10y ago
But it's not better, it is worse as clearly put forward by this article. ;) We shouldn't attribute Facebook's success to poor quality code, unless we really believe that 429 people working that that slow sluggish app is productive. Do you? Maybe Facebook would be in a better situation than they are if they had a firm foundation? What happens when a competitor comes along and knocks on their door? That mess of code becomes an anchor. They succeeded not because "worse is better" but because ...they succeeded, maybe because they had the greatest social network in the world to expand within, or because of some other reason we don't know. But they didn't succeed because of worse is better, they succeeded in spite of it.
- paganel 10y ago
  > Do you? It's written between the article's lines, so to speak, but it seems like the bad code quality was generated in part by FB moving at a very fast pace. I think they wouldn't have been as successful as they are had they chosen a slower path with more "correct" code and architecture. > What happens when a competitor comes along and knocks on their door? They successfully managed to "eat" and integrate both Instagram and WahtsApp, both worthy contenders at the time, and they're now on their way to stealing Snap's lunch, so they've proved to be pretty resilient on this front. Not to forget how they successfully kept Google+ in check.
- jheriko 10y ago
  how would that success have been different if they had used different engineering practices? a big part of their success was delivering results... if they hadn't quickly crapped everything out of the door, how would facebook look today?

segmondy 10y ago

So? I've become very cynical since the past year or so because there is a lot of noise/articles on the net, all knowing what's best for you or wrong with what you are doing. Yet without context or true authority to talk about it. My question these days when I read such article, "What has the author done that is equivalent?" There is rarely anything to be found. Facebook might have a "code quality problem." But until you have worked at an organization that is big like Facebook please hold forth your tongues or fingers in this case. Startup principles don't apply nor academia. Facebook's "code problem" is what really does happen in the real world with such humongous business enterprises. Lot's of moving pieces in code, in people, in ideas and the amalgamation of these results in what most programmers will see as "code quality problem" yet the market sees as billions of dollars.

aaron-lebo 10y ago
Isn't that an appeal to authority? Because someone hasn't done X they don't have the right to criticize X? Facebook's "code problem" is what really does happen in the real world with such humongous business enterprises. But does it have to be this way?
- evgen 10y ago
  Until someone provided s counter-example the assumption is that yes, this is just what happens. We have entire industries of competing methodologies, some almost cult-like in their slavish devotion to "the one true process", and none have birthed any great successes. Why is that?
  - deleted 10y ago
    [deleted]
  - dustingetz 10y ago
    History is littered with counter-examples and great successes. Each generation of company is more efficient or solves more complicated problems than those before it. Facebook itself is an industry leader in open source innovation, which is why their product is so smooth. But not all companies specialize in hard tech, and all companies get old.

jheriko 10y ago

one look at hhvm tells me they have poor quality development practices. :P i get the impression that they use the struct keyword to avoid having to type public everywhere for instance...

watwut 10y ago

Reality check is that most companies, including most startups, attempting to create software of such size end up tangled in their own mess way sooner and fail. Keeping it maintainable at such scale is not as easy as keeping it maintainable when there are four of you.

planetjones 10y ago

Well when there is a codebase of the scale of Facebook's, with so many activate contributors of course there will be a code quality problem. I see so much stuff at Google breaking or becoming inconsistent too (away from the core search features). Even if they didn't move fast and break things, they would still have code quality issues. It's just the nature of a huge codebase like they have. And let's face facts - the development process needs to be suitable to the type of problem being solved. This is not a life and death situation and users would value new features, more than extreme reliability and consistency.

fishnchips 10y ago

I spent a few months at Facebook and left because of that. Not necessarily because of poor code quality but because of all the issues it was causing, and a culture of turning a blind eye to (or worse still, being proud of!) those issues. It's obviously true that quality of their code does not affect their market position, but it sure affects who they hire and who they retain.

joeletizia 10y ago
That may be true. But in the end, does it matter? Until we see someone taking market share away, I'd argue it doesn't. Facebook still retains top tier people regardless of any perceived quality problems. Sorry you left. I just started here, and I do agree that what I feel about code quality isn't generally shared here. But as I get older, I start realizing that in the end, as long as things work relatively well at our scale and in our problem domain (we're not medical device software, we can fail) alls well.
- feedjoelpie 10y ago
  I did some contract work for FB around when this article was written. I don't think that nobody cares about clean code. It's more like... results will vary widely based on who reviews you. And the governing philosophy is more focused on the automated checks and balances that try to keep you from doing harm to the overall system. While human code reviews were pretty lax, the machines would slap you around for all sorts of things, none of which would force you to write beautiful code. Personally, I like for humans to be pickier than they seemed to be at Facebook, but it really opened my eyes to the value of what I think of as a "keep your garbage in your own trash can" approach to quick-and-dirty coding. As long as you didn't spill your trash into the hallway, the machines were satisfied, and your reviewer didn't find your work to be _insane_, your work shipped. [EDIT: And I find that this can often be a wonderfully pragmatic way to work if you have the tools to keep you from doing harm.]
- Sevii 10y ago
  It matters because profits are lower. 400 devs on an iOS app is not free.

ptero 10y ago

Code quality is important, but for an organization the size of the FB good systems engineering is much more important than good software engineering. If the overall system is organized sanely, failures of subcomponents (which happen due to software bugs, hardware, humans, etc.) can be isolated, firewalled and fixed in reasonable time without bringing the system down.

chiefalchemist 10y ago

Move fast...and just leave things broken. Users be damned.

chiefalchemist 10y ago

But seriously (i.e., ref to my previous comment). All the tools. All the talent. All the leadership and management. All the money. And still even the mighty FB struggles. Point being: This shit might not be rocket scientist hard, but it ain't easy either. And when you don't have the war chest of the likes of FB you're in for an ongoing and never ending (quality) battle.

raygelogic 10y ago

this strikes me the same way as the story of the guys who figured out where to add armor to bombers during WW2. the places on the plane that had bullet holes after a mission clearly were operational enough to return to base. the places that never had any damage were probably critical, so that's where armor was added. if a company can make billions with a poor-quality codebase, clearly quality isn't a bottom-line concern. what is a concern is shipping the damn product.

dawidloubser 10y ago

I would like to argue that, in most of these cases, it's not "code quality" that is at fault, but "design quality" (before coding) - which is often absent entirely. The problem with design, in software, is not that most people forget to do it. It's that they never learn to do it. It always comes back to bite you. I don't want to start a discussion on design, and how most people mess it up because of lack of skill or experience therein. But hacker culture seems to be allergic to design, and hacker culture seems to be what everybody strives for these days.

lefty2 10y ago
I think you hit the nail on the head. My motto for developing code is "make it simple as possible, but no simpler". Whoever, created a design that used 18,000 classes did not follow that rule.

typetypetype 10y ago

What's the point of an app? Generating high quality code or generating revenue? Until you hit intersections where revenue can be increased by changing code, the priority will always be on new or changed features.

baggachipz 10y ago

> The “Hack” and “Move fast and break things” culture must make it very hard for developers to focus on quality. Well, yeah, that's what the "break things" part means. The problem is when people/companies try to have it both ways. "Move fast and have high quality" isn't possible.

thedevil 10y ago

It's hard to argue with the business sense of pushing out features quickly at the cost of code quality. But I've learned that that there are two kinds of development teams: (A) the teams that are "moving fast and breaking things" while "creating business value" and (B) the teams that are "only maintaining legacy code", "not getting anything done", "costing a lot of money" and "complaining too much" about code quality that team A wrote. As an engineer, I've learned that it's less work and more rewarding to be on the (A) teams than on the (B) teams.

nickpeterson 10y ago
The difference is that Team A can leave and a project is delayed. If Team B all left the company would fold.
- ceronman 10y ago
  Both are necessary. If Team A leaves, and no more business value is produced, and the devs only care about beautiful code, the company will fail too. Both kinds of teams are needed. Or even better, teams should not be divided like this, everyone should do their shares of both breaking things, and creating value and cleaning the mess and scaling the things.
- scaryclam 10y ago
  Funnily I've actually seen a Team B leave. There was much panic and thinking it would all go horribly wrong, and indeed it was madness for the first week. Then the people who took over started to stabilise some of the worst parts of the system and it all sort of fell into place after that. Fast forward a couple of months, the project was pretty much hands off and we all realised that keeping a whole team on that project had pretty much been a giant waste of money. They did nothing but create fires for themselves to fix. The Team B's exodus was actually a really great thing for the company. I suppose the thing to learn here is, that you need redundancy in your teams. If we hadn't had other devs to step in and take over, it would have been a bit different. Still, I'm not convinced that a handful of contractors couldn't have been brought in to the same result, just more expensive.

mybrid 10y ago

It is worth noting that when something goes viral we still don't know why. Facebook represents the ultimate in viral success. Given we do not understand why things go viral then it stands to reason that the underlying software methodology is not causation. This means that the fickle finger of fate could easily be visited upon Facebook the same as with MySpace and Facebook will be in no control. This is independent of the any software methodology they use. Claims made without evidence can be dismissed without evidence.

strken 10y ago

Can someone explain the example with the ramdisk? It's not clear to me how a ramdisk is relevant to the problems put forward in the paper, i.e. disk is too slow, they rely on lazy page allocation, and they have insufficient ram to hold two copies of everything simultaneously.

alex- 10y ago

> "when Facebook employees are not actively making changes . . . the site experiences higher levels of reliability." This seems like a great thing to me. i.e. The system is stable and the error budget is being used to facilitate change.

dwils098 10y ago

links to slides are all dead... in parent articles and reference articles

bobbyi_settv 10y ago

> The second exhibit is from Facebook Research... wouldn’t you just write your disk files to a ramdisk? Surely they noticed this too I'm not sure the author understand what a Research team is and what it does. Trying out a new solution to an existing problem is sort of their job. I'm also not sure how a research team publishing a paper discussing an alternate solution indicates anything about the company's "code quality".

Bahamut 10y ago

Anecdotal evidence, but I have a friend who complained that some critical parts don't have tests, and a lot of developers are loathe to write them. He got bit by it when he made a change in an area with no test coverage and broke the spam filter.

tomkin 10y ago

It's important to understand that Facebook is incredibly large, and the fact that any software scaled with them to the point they are presently at, is a "on the shoulders of giants" situation that seems to actually be working well enough for them to become a billion dollar enterprise. Also, the building of infrastructure that takes on scale that's not yet been required is somewhat inefficient. But it does beg the question: If Facebook is limited by its infrastructure, what responsibilities do they have to build software that continues to scale for future organizations?

p0nce 10y ago

At this point it is clear that software size increase linearly with the size of your software team, whatever the problem you solve. This affects every organization and is something that should be actively fight against. Having accidental, unplanned, unaccounted costs should not be the default path.

charleslmunger 10y ago

"Exhibit C: Our site works when the engineers go on holiday" Is not all so different from "Fewer patients die when heart surgeons are on vacation". Of course your site is going to be more reliable when nobody is changing it! You should be worried if reliable isn't the steady state, and it requires constant changes to stay up!

afc 10y ago
I also took issue with this part of the article; it seems obvious to me that rolling out new features would directly cause reliability issues (and the only alternative, of not rolling out new features, is, obviously, a cure worse than the disease).
- adrianratnapala 10y ago
  I am not sure that most Facebook users would actually agree about this. But then their view is slanted towards UI changes. Performance improvements and other subtleties are features too.
joshvm 10y ago
My immediate thinking was this as well: when they stop touching it, it's fine, which means that whatever bugs are checked in seem to be caught quickly. What's more, the periods of high reliability are during periods when you might expect there to be heavy load - assuming people don't check Facebook at work (har har). What counts as an incident? > Figure 1 includes data from an analysis of the timing of events severe enough to be considered an SLA (service-level agreement) violation. Each violation indicates an instance where our internal reliability goals were not met and caused an alert to be generated. Because our goals are strict most of these incidents are minor and not noticeable to users of the site. So these are minor issues. The parent article is paraphrasing at best, and jumping to conclusions at worst. From Facebook: > We believe this is not a result of carelessness on the part of people making changes but rather evidence that our infrastructure is largely self-healing in the face of non-human causes of errors such as machine failure. They then list a number of sane strategies to mitigate this. http://queue.acm.org/detail.cfm?id=2839461 http://queue.acm.org/detail.cfm?id=2839461

zo7 10y ago

Some of FB's ~18,000 Obj-C header files from a post linked in the article: [1] * FBFeedAwesomizerProfileListCardViewControllerListenerAnnouncer.h * FBBoostedComponentCreateInputDataCreativeObjectStorySpecLinkDataCallToActionValue.h * FBEventUpdateNotificationSubscriptionLevelMutationOptimisticPayloadFactoryProtocol-Protocol.h * FBGroupUpdateRequestToJoinSubscriptionLevelMutationOptimisticPayloadFactoryProtocol-Protocol.h * FBMemReactionAcornSportsContentSettingsSetShouldNotPushNotificationsResponsePayloadBuilder.h * FBProfileSetEventsCalendarSubscriptionStatusInputDataContextEventActionHistory.h * FBReactionUnitUserSettingsDisableUnitTypeMutationOptimisticPayloadFactoryProtocol-Protocol.h oh my god [1]: http://quellish.tumblr.com/post/126712999812/how-on-earth-the-facebook-ios-application-is-so http://quellish.tumblr.com/post/126712999812/how-on-earth-th...

aljones 10y ago
Those read like classes generated by tools. Or maybe I really don't want to think humans are capable of such names.
- utexaspunk 10y ago
  Well, the humans could be tools...
- oriolid 10y ago
  These are completely credible class names, just ask any Enterprise Java developer. I particularly enjoy FactoryProtocol-Protocol. Who would be satisfied with just a FactoryProtocol?
  - ryandrake 10y ago
    Yea, I was going to say, Java's almost got them beat with SimpleBeanFactoryAwareAspectInstanceFactory[1] 1: http://docs.spring.io/spring/docs/2.5.x/javadoc-api/org/springframework/aop/config/SimpleBeanFactoryAwareAspectInstanceFactory.html http://docs.spring.io/spring/docs/2.5.x/javadoc-api/org/spri...
    - 10y ago

jyriand 10y ago

Only Kent Beck can save them. :)

whack 10y ago

I do think that FB, and most large orgs, have code quality problems. But the article does a pretty bad job at making its case. > "That’s 429 people working, in some way, on the Facebook iOS app. Rather than take the obvious lesson that there are too many people working on this application, the presentation goes on to blame everything from git to Xcode for those 18,000 classes." How does the author know that 429 is too many? How does the author know that FB's goals and functionality can be best achieved with fewer people/classes. This just reads like a classic "Why is Google so big, I can do that in a weekend" comment (https://danluu.com/sounds-easy/ https://danluu.com/sounds-easy/) > "Our site works when the engineers go on holiday" This is pretty much universally true for any dynamic site where engineers are constantly adding new features. Change always comes with risks. Changing code is always more likely to break something in the very-short-term, compared to not changing code. I have a hobby site (shameless plug, http://thecaucus.net http://thecaucus.net) which has been running in auto-pilot for the past year, and almost never breaks, because there are virtually no moving pieces. The fact that the FB site breaks more often when engineers are making changes to it, is just repeating a universal law of software development. I do think that the organizational structures at most large companies are bloated, inefficient, non-transparent, and produce sub-par code. I had high hopes when I read the article's headline, but the arguments presented simply aren't very persuasive.

combatentropy 10y ago

I look at programming as all design, whether deliberate or not. Martin Fowler drove the point home for me (https://www.martinfowler.com/articles/newMethodology.html https://www.martinfowler.com/articles/newMethodology.html). He said that some people drew inspiration from construction, where there are designers and builders. One or two highly paid architects draw up the plans. Then you can hire a bunch of cheap labor to build it, say a bridge. This belief leads to a dichotomy in software companies, where one person is the "architect" and others are just "regular" programmers --- often outsourced to the lowest bidder. From Jack Reeves he cites the epiphany "that in fact the source code is a design document and that the construction phase is actually the use of the compiler and linker." There is little repetitive, mindless work in programming, because as most of you know, "anything that you can treat as construction can and should be automated." Therefore, "In software all the effort is design . . ."

krick 10y ago
I feel your sentiment, and usually oppose to the construction-driven approach you describe, but it is really easy to play devil's advocate here. I mean, let's just be realistic. There's nothing sacred in programming. The fault in your argument is that there's no design/construction dichotomy, but rather a gradient: everything can be viewed upon as a design from close enough distance, and everything can be a construction looking from the high above. For business people all the programming is just very, very low-level construction, and there's no design to it. Damn, even hiring programmers and creating companies is just a construction stage to them. And from the practical standpoint, the truth is that perceiving a lot of programming "just as a construction" just works. For the better or worse. It is cheaper and usually reliable enough to design the system in a way that a lot of the components would be plugins not depending one on the other that can be outsourced to anyone. Even if some fail, the system continues to work while you hire new people to rewrite some components. Sure, maybe it would be more efficient and "cool" to hire a really smart guy (or do it yourself, if you believe you can) to write it as really optimized monolith system, but if this isn't required, it is usually less risky to build something not so sophisticated, and improve it if need be. It is usually ok, that some parts of the system may fail. Fuck it, let's not lie to ourselves: there always are some parts of the system that fail, whether you think it's "acceptable" or not, let's just take it into account.

norswap 10y ago

I was doing iOS dev three years ago, and the Facebook lib for iOS was the worst piece of shit I ever had to deal with. Their app also used to be unbearably sluggish despite not doing anything too fancy.

babesh 10y ago
I thought the joke was that they had interns write that? I remember an API that would hold onto completion blocks and call them repeatedly instead of just once. They would also accidentally make non backwardly compatible changes to the API. They didn't care since that API period propelled them to more dominance and growth for awhile. After they shut down the viral channels, the developer ecosystem died...but it had proven its usefulness. The only issue now is developers don't trust them and they can't get enough developers to build atop them.

linkmotif 10y ago

While the author wrote this article FB made millions of dollars.

badpenny 10y ago

I recommend reading They Write the Right Stuff[1] to anybody interested in NASA's approach to writing software. [1] https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff

deleted 10y ago

[deleted]

ChrisRus 10y ago

> From Jack Reeves he cites the epiphany "that in fact the source code is a design document and that the construction phase is actually the use of the compiler and linker." Ha. That is laughable. If it were so, then we would be able to automatically graft features from one product to another. I contend this is exactly how SoC are designed and there's no reason why we shouldn't copy the methods of the digital design community (proven to scale or you wouldn't be reading this).

aqibgatoo 10y ago

bob martin a.k.a uncle bob always says that people consider patterns like mvvm,mvc,mvp etc as Architecture which actually are just for the ui layer.It happens commonly in mobile development.

pablo_fiesta 10y ago

Rub lavender oil on the servers, say "web-scale" 3 times, click you heals and everything will be OK. Oh wait, but the Russians.