8 ms·

How Facebook Ships Code

344 points by atularora 16y ago

kemiller 16y ago
Shocked... shocked I am that that buggy pile of crap has no QA.
intranation 16y ago
Sounds like utter chaos for any seasoned professional. No QA? No real product managers (or at least none with any teeth)? Design as optional resource? Ops managing user engagement metrics? Where's the security audit? Scary, even for a developer like myself.
- abyssknight 16y ago
  I'm a little blown away at this myself. I knew they were extremely agile, able to make changes on a whim, but this sounds a bit crazy. Might check in with my buddy who works at FB to see if this matches up with reality. The sheer stress of working in an environment where perfection is not only desired, but required, would drive me crazy. Some of the processes, though, sound like they would be a dream. Working on things I want to work on would make me 100x more productive. Of course, it also means some things may never get done.
  - EllieAsksWhy 16y ago
    It would be interesting to get some verification, because no one could work in this environment for very long. Regardless of compensation. The stress and associated error rates that would accompany it would've been a problem long ago for Facebook. Particularly with no QA. I cannot imagine Facebook existing and prospering for this many years without any QA team.
- brown9-2 16y ago
  Perhaps this is a good example of how much of that professional "seasoning" is actually unnecessary or gets in the way of building successful products and high-caliber responsible teams, i.e. "throwing over the wall to QA" etc. edit: Can I ask why this is being downvoted? Because I am disagreeing with the parent poster? I don't think I am raising poor points or somehow not contributing to the discussion. Facebook seems to not do a lot of "standard" things because they view them as counter to their goals of building their product. I am merely stating that perhaps there is a lesson in there about the value of what some of the convential wisdom in our industry dictates.
  - kenjackson 16y ago
    Upmodded... with that said, I do think that this is very industry dependent. They are a free product that people don't necessarily "rely" on. If status messages were severely delayed or dropped, it wouldn't be a huge deal (assuming they fixed it when noticed). There's very little that they can't fix after deployed, and they have no SLA with consumers (advertisers may be different). In essence they have 500 million testers. This is a bit harder to do with a database or compiler.
    - kemiller 16y ago
      App developers and other API users do in fact rely on them. Their unreliability is part of what's convincing many sites to rethink that.
- jcfrei 16y ago
  I'm especially wondering: who will take this platform to the next level? Is there anybody besides mark zuckerberg in charge for designing and testing out new ideas. I don't mean just new features like questions or places, but really significant changes like a facebook phone, or a new friends model to better graph you real network.
ramanujam 16y ago
Some inaccuracies pointed out by another FB engineer http://www.reddit.com/r/programming/comments/f3u0n/how_facebook_ships_code/ http://www.reddit.com/r/programming/comments/f3u0n/how_faceb...
- neilk 16y ago
  Copying the main points here, from Reddit user "epriest": The article is has a few inaccuracies: There is mandatory code review for all changes (i.e., by one or more engineers). I think the article is just saying that Zuck doesn't look at every change personally. People do not get called out for introducing bugs. They only get called out if they ask for changes to go out with the release but aren't around to support them in case something goes wrong (and haven't found someone to cover for you). Getting blamed will NOT get you fired. We are extremely forgiving in this respect, and most of the senior engineers have pushed at least one horrible thing, myself included. As far as I know, no one has ever been fired for making mistakes of this nature. We have automated testing, including "push-blocking" tests which must pass before the release goes out. We absolutely do not believe "most engineers are capable of writing bug-free code", much less that this is a reasonable notion to base a business upon. If this is a real quote, it comes from a very junior engineer. It is empirically untrue. The nine push phases are not concentric. There are three concentric phases (p1 = internal release, p2 = small external release, p3 = full external release). The other six phases are auxiliary tiers like our internal tools, video upload hosts, etc.
  - pak 16y ago
    OK, so according to him there's code review and some automated tests. But that still doesn't address wide open access to the live DB by brand-new devs, and the complete disregard for having a dedicated QA team. Why QA? I would think FB engineers are more busy doing real work at work than fooling around on their own site (heh, the irony) and finding out that Junior Dev 39's change to the thingamabob broke chat for the umpteenth time in IE7. And as many people have pointed out here, they are a platform for many other businesses now--there's no dogfooding that--and the API stinks in terms of stability and documentation.
deleted 16y ago
[deleted]
ernestipark 16y ago
This seems to fit how I view Facebook as a whole. I find some of their technology extremely impressive such as their blazing fast search or HipHop. However, a lot of their user experience is extremely flawed and there is a big lack of consistency and intuitiveness. Examples: clicking your friend requests on the right side of the home page pops something up that tells you to click somewhere else, events are no longer notified so you have to manually check to see if you received new ones, the distinction between adding a page to your interests versus Like'ing it is not well defined, or if it is, many users are unaware, making an event for a Facebook Page is very convoluted... etc etc. I feel that issues like these are the reason why product managers and designers exist. I think Facebook is starting to really fall behind in this regard as more and more features start to clutter and complicate the site.
EGreg 16y ago
Yeah, I wouldn't want to run my company like this: re: surprise at lack of QA or automated unit tests — “most engineers are capable of writing bug-free code. it’s just that they don’t have an incentive to do so at most companies. when there’s a QA department, it’s easy to just throw it over to them to find the errors.” It does explain how facebook ships with bad documentation etc. -- well, now we have enough knowledge to know how to compete with it :) Kasparov often remarked how a good process is more important than the actual participants. An average human and pretty good computer with a great system won the championship against great computers and against great grandmasters. Google believes in this, and their products are very well engineered, with full documentation, videos etc. (although admittedly, many haven't taken off). Yahoo definitely understands this. But these are the same companies that are losing to facebook because of social. If google and yahoo understood the dynamics of social, we would all be better off.
kenjackson 16y ago
I'm really surprised that all engineers have free access to the Facebook database. That's really scary. I know at other organizations, with much less sensitive data, this stuff was under lock and key. I have a hard time believing many aren't abusing this, just out of human nature ("did she mention me to any of her friends?").
- ssclafani 16y ago
  Multiple employees were fired for accessing private data in the past. I suspect that it is a serious taboo at the company now to even joke about it.
  - pak 16y ago
    So what you're saying is that now anybody can still do it, but everybody knows not to talk about it. I'm gathering that this will never change.
    - hugh3 16y ago
      Whenever I meet anyone who works at facebook, my first instinct is to extricate myself from that conversation. Otherwise we might get to know each other, and become friends, and I'd never be sure that they weren't looking me up. My strategy for avoiding the prying eyes of facebook engineers is to be personally uninteresting to any facebook engineers. This might not work so well if I were a chick.
  - kenjackson 16y ago
    That seems way to risky to me. You basically have to watch server logs for some interesting patterns. And of course, if they're smart they simply oversubscribe in a query (or temporally space out a query), and then pinpoint the data they are looking for locally. Given they have 2000+ employees this is just a process that is looking to be constantly abused. I surely don't expect people aren't talking about it, but I'm nearly as confident that it is happening in a way that most people aren't aware of. No evidence, but again, I've seen people do worse, with less access.
- 16y ago
theletterd 16y ago
It seems, from the article at least, that the're very much a culture of blame, which I find quite surprising. I've never been a fan of the practices of practices such as making people wear a hat if they break the build and the like. It seems counter-productive to getting things built and shipped, and just seems to make an unhappy workplace (in my mind, at least)
- epriest 16y ago
  This part of the article is simply untrue. The culture is extremely forgiving of mistakes. (I don't speak for my employer, Facebook.)
clojurerocks 16y ago
The more i read about facebook engineering the less impressed i am. It used to be interesting to hear about their engineering challenges. Now its just scary to hear anything about them. And i use it as a rulebook of how not to do youre infrastructure. Im amazed facebook works at all frankly. I also think its one of the ugliest sites ive ever used. Its what 5 years old now and they still dont even have some basic functionality on it. Plus as another person said its often broken.
pak 16y ago
So about 1000 people have unfettered access to all of the live data, but there are absolutely no safeguards against them modifying it, copying it elsewhere, or peeking at it? That's terrifying. I'm under the impression that Google has more a stringent philosophy about this, and even then, one of the few SRE's that they do give broad access to was caught stalking teens via GMail data. Imagine the range of bad behavior an FB engineer can get away with after just 4 weeks of "Boot Camp". Part of the reason I never made a real FB account was my general feeling in 2005 that securing a MySQL/PHP site against internal abuse takes incredible effort, and they wouldn't ever bother to do it. According to this article, that's likely true. And the "everybody can modify anything anytime" philosophy with little QA explains why, after 6 years, FB is still a gaping maw of security holes.
- holman 16y ago
  after boot camp, all engineers get access to live DB (comes with standard lecture about “with great power comes great responsibility” and a clear list of “fire-able offenses”, e.g., sharing private user data) This is through a third-party, so I'm not shaking the finger at Facebook outright yet, but the wording of that — sharing private user data — is kind of frightening. What about merely accessing private user data? Like you mentioned, I know Google is super-paranoid about even simple data access. And now someone from Facebook chimes in and replies to this and talks more about their developer privacy safeguards and makes us all feel better. Go!
  - pollockmania 16y ago
    > And now someone from Facebook chimes in and replies to this and talks more about their developer privacy safeguards and makes us all feel better. Go! Former Facebook dev here. That's all I have to say in response.
    - pak 16y ago
      Since this is your alias, I hope you won't mind my curiosity--when did you leave the company, and do you have any thoughts about leaving?
ajsharp 16y ago
“most engineers are capable of writing bug-free code. it’s just that they don’t have an incentive to do so at most companies. when there’s a QA department, it’s easy to just throw it over to them to find the errors.” Bull. Shit. This one line makes me think that this entire post is completely hear-say. A 500 person engineering team that writes bug-free code on a system as a large as facebook? Give me a break.
- krobertson 16y ago
  Agreed. Anyone who works with FB apps/apis surely notices their lack of QA leads to frequent and often times routine breakage.
  - peeps 16y ago
    To be fair, the ridiculously low quality of their code makes me think that it's true that they don't really bother with testing or QA. It explains a lot, really.
  - lukeschlather 16y ago
    I have my Facebook set on Spanish to keep my Spanish language neurons firing even when I'm surrounded by Gringos, and their localization definitely has frequent bugs. There's been some issues for months with the birthday wall post aggregation they've been testing in production. I've frequently seen stuff like "So-and-so has written on so-and-so's muro for their birthday." So yeah, lack of QA => lots of bugs.
    - nroman 16y ago
      Almost all of Facebook's internationalization is crowd-sourced. What you're seeing probably isn't a bug, the string just hasn't been translated by a user yet.
    - sidupadhyay 16y ago
      Facebook's language translations are actually crowdsourced. It's funny that so many people would approve errors like this. There is a brief blog post about it: http://blog.facebook.com/blog.php?post=10056937130 http://blog.facebook.com/blog.php?post=10056937130 The whole system is pretty interesting. They maintain a mapping of every token on the site to every language and only user translators verify portions.
CWIZO 16y ago
What does it mean to be "publicly shamed" in this context?
siddhant 16y ago
Multiple mentions of "public shaming". Doesn't seem so good. A good engineer is anyway going to have a sense of shame inside when he/she screws up. I really don't think there's any need for "public shaming".
- EllieAsksWhy 16y ago
  So, so true. The tendency for perfectionism necessary to write code is already sufficient to instill a professional ethic as far as quality of work. Public shaming would be intolerable, I'd think.
- gaius 16y ago
  You don't get shamed for a bug in your code - you get shamed if you aren't willing to go on-call to support your own code. That IMHO is a very healthy attitude.
kunjaan 16y ago
I would like to know what are the other items in the 'clear list of “fire-able offenses” , e.g., sharing private user data)"
krummas 16y ago
The key point for me in this article was: "resourcing for projects is purely voluntary." As someone who works for an online gambling company where we are not even allowed to use the product we are building (legal/trust issues i guess), it would _rock_ to be able to have this kind of impact on features
flyt 16y ago
More info directly from a Facebook Engineer about their testing process: http://www.quora.com/What-kind-of-automated-testing-does-Facebook-do/answer/Steven-Grimm http://www.quora.com/What-kind-of-automated-testing-does-Fac...
cduruk 16y ago
Is the author of this piece an engineer working at Facebook? This part makes me think not: >I’m fascinated by the way Facebook operates. It’s a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried). These are notes gathered from talking with many friends at Facebook about how the company develops and release software. So I'd take whatever is written here with a grain of salt. My communication with friends working at Facebook yielded similar thoughts but nothing that comes to what's written that implies a callous recklessness. I know for a fact that they have some code-review tools and blocking tests. Anyway, my point is that the author doesn't seem to be embedded too deeply in the engineering at Facebook and his notes are, while not outright false, definitely misleading.
sanj 16y ago
This is very close to how we develop at TripAdvisor, though we have a suite of tests that get run during a branch merge and code review (of at least one more engineer) of anything going into the livesite code base. We also have very small QA team. They work to make sure that the buttons that make us money aren't screwed up. The key to this sort of uber-agile development is that you have very, very talented engineers, a release process flexible enough to deal with errors, and a management that buys into moving so fast that mistakes will happen. I have to say that it is simultaneously exhilarating, humbling and a little terrifying to work like this. Luckily, mostly the first two.
- rhizome 16y ago
  I know all of this certainly has worked and does work for you, but the first thing I thought about when reading "small QA team" and "the key...talented engineers" is that QA is not very important to the company. A QA team that only checks the money buttons tells me that traditional QA has been incorporated into the developer positions themselves, which creates a conflict of interest. Couching my conclusion in the topic of this post, it would seem that Facebook gives extraordinary freedom to developers, but also hangs everything around the developers neck. Screwing up in this area sounds pretty fatal, and not an aspect of a company I would want to work for. I'm kinda old, though.
  - brown9-2 16y ago
    the first thing I thought about when reading "small QA team" and "the key...talented engineers" is that QA is not very important to the company Couldn't the other corollary of this be that the company/development team has invested heavily in automated testing?
- timcederman 16y ago
  Very small QA team...unless you count all the feature, cross-browser and multi-language testing done by the product management team. I'm still not sure having the product team responsible for hours and hours of black-box QA testing each week was an effective use of resources.
patrickk 16y ago
"If lots of [employees] are flocking to a new business unit, that's a good sign that the opportunity is a good one. . . . If a business unit can't attract people very easily, that's a good sign that it's a business Enron shouldn't be in." - Jeff Skilling, former president of Enron http://www.gladwell.com/2002/2002_07_22_a_talent.htm http://www.gladwell.com/2002/2002_07_22_a_talent.htm Contrast that with the Facebook approach: "Resourcing for projects is purely voluntary. -a PM lobbies group of engineers, tries to get them excited about their ideas. -Engineers decide which ones sound interesting to work on. -Engineer talks to their manager, says “I’d like to work on these 5 things this week.” -Engineering Manager mostly leaves engineers’ preferences alone, may sometimes ask that certain tasks get done first. -Engineers handle entire feature themselves — front end javascript, backend database code, and everything in between. If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on. Same for Architect help. But in general, expectation is that engineers will handle everything they need themselves." Good god. Does this strike anyone else as disturbing? Surely every single piece of work being taken on should have the USERS needs and concerns as top priority and not sexy stuff that can attract sufficient engineering interest? What if there's important problems that are really bothering lots of users and a PM can't get anybody interested (or no-one decides to take on the problem?) Here's a complaint from a guy about maintaining a personal page and fan page: http://www.stevepavlina.com/blog/2011/01/leaving-facebook/ http://www.stevepavlina.com/blog/2011/01/leaving-facebook/ Some quotes from that piece: "As a programmer myself, I can’t fathom that it would take much technical and design effort to address these issues, and Facebook is flooded with complaints from users begging them to fix these headaches. From my perspective as a Facebook user with a very active personal page and fan page, I can’t help but get the impression that Facebook deliberately wants to make some basic admin tasks (like blocking spammers) difficult or impossible in order to compel you to spend more time on the site. There doesn’t seem to be any other logical reason for these glaring design flaws that I can comprehend, other than pure incompetence, and based on their success in other areas, it seems more likely that these choices are deliberate." "Surely someone on their team is aware of all the complaints and requests to fix the broken elements. So why do they seem to ignore what appear to be such glaring (and fixable) problems?" "I thought that Facebook would be an interesting place to share inspirational messages and build more community around growth-oriented people. But the current implementation of Facebook can’t handle the way I’ve been trying to use it without creating more headaches than it’s worth, and their momentum appears to be headed in the wrong direction for me to expect that these problems would be fixed anytime soon." "So I’ve crossed the threshold where Facebook’s value isn’t worth the hassle to use it. I concluded that the best choice was to simply drop the service altogether and invest my time elsewhere." Who will take on these issues? Again from the Malcom Gladwell article linked above: "You might expect a C.E.O. to say that if a business unit can't attract customers very easily that's a good sign it's a business the company shouldn't be in. A company's business is supposed to be shaped in the direction that its managers find most profitable. But at Enron the needs of the customers and the shareholders were secondary to the needs of its stars." Facebook should wake up in my opinion. They did really well to reach 600m users (or whatever the figure is now) but if they want to stay there they should get their priorities straight. EDIT: Grammar
anthony_franco 16y ago
"no QA at all, zero" As someone who develops on top of the Facebook Platform, I'm not surprised. Huge, obvious bugs that affect many applications are released far too often.
clojurerocks 16y ago
Serious question. Why doesnt Facebook do any engineering talks anymore? Some of the coolest talks on infrstructure from the past few years have been from FB. And yet the only thing i can remember really hearing about their stack recently is that they were using HBase. Which really came off as more about marketing and hype about some products they were working on then truly geeky stuff thats come out from FB in the past. Anybody know? I wonder if its because Fb is no longer a startup? Which might the reason why we never hear about Youtube or Twitter anymore either. Oh my other comment was just meant in jest. But thanks for downvoting it and killing all the worthless and meaningless HN karma points i had accrued!
- deleted 16y ago
  [deleted]
- daveman692 16y ago
  http://www.facebook.com/Engineering?v=app_2392950137 http://www.facebook.com/Engineering?v=app_2392950137
- deleted 16y ago
  [deleted]
didip 16y ago
Given the current sad state of Facebook Platform, it must be because no one is working on it. Or it is labeled "unsexy" to work on it.
- bjclark 16y ago
  Well, the lack of QA certainly explains a lot.
tommi 16y ago
Funny how this article has 33 main bullets points out of which 13 are or contain corrections.
Aloisius 16y ago
It sounds like they've institutionalized the herding of cats as a project management system.
deleted 16y ago
[deleted]
kschua 16y ago
"all engineers go through 4 to 6 week “Boot Camp” training where they learn the Facebook system by fixing bugs" I really like this part. Many new staffs prefer to come in and do the "sexy" enhancements instead of the "mundane" support. This leads to their lack of understanding of the system and poor design. I had always believe that new starters should do support work for a while to gain an understanding of the overall system.
jkuria 16y ago
"very engineering driven culture. 'product managers are essentially useless here.' is a quote from an engineer. engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime. during monthly cross-team meetings, the engineers are the ones who present progress reports. product marketing and product management attend these meetings, but if they are particularly outspoken, there is actually feedback to the leadership that 'product spoke too much at the last meeting.' they really want engineers to publicly own products and be the main point of contact for the things they built." This sounds like Mark Zuckerberg living his 'revenge of the nerds' dream in his Facebook nirvana! It certainly doesn't sound like good management practice!
- maxawaytoolong 16y ago
  It certainly doesn't sound like good management practice! Why not? I worked for silicon valley startups for 15 years and not once did I meet a "product manager" who did anything worthwhile.
  - pzb 16y ago
    Product managers are very useful when you are not your own customer. Good product managers are a proxy for your customers, providing you information on the customer wants and needs. Good product managers work with the engineers to prioritize projects; it is not a dictator position, but does help ensure that the less sexy projects get done as well as the uber-cool sexy projects.
keyle 16y ago
This just looks like some sort of marketing propaganda to me.
InclinedPlane 16y ago
Estimated Joel Test score for facebook: about 5 out of 12. http://www.joelonsoftware.com/articles/fog0000000043.html http://www.joelonsoftware.com/articles/fog0000000043.html
- nikolayav 16y ago
  As a Facebook engineer, I'd say the score is 10 or 11, depending on your interpretation of the tools item.
bootload 16y ago
"... What do you think? Would “developer-driven culture” work at your company? ..." Hasn't everyone read, "Microsoft Secrets" ~ http://www.amazon.com/MICROSOFT-SECRETS-Powerful-Software-Technology/dp/0028740483 http://www.amazon.com/MICROSOFT-SECRETS-Powerful-Software-Te... ...because what the author describes is pretty much the MS dev process (cf CH4 Defining products and development processes). Reading JOS, "How to be a program manager" also shows how MS PM's worked in conjunction with developers ~ http://www.joelonsoftware.com/items/2009/03/09.html http://www.joelonsoftware.com/items/2009/03/09.html MS sure knows/knew a thing or two about organising large groups of people to produce software & output product in a corporate setting.

344 points by atularora 16y ago

Shocked... shocked I am that that buggy pile of crap has no QA.

Sounds like utter chaos for any seasoned professional. No QA? No real product managers (or at least none with any teeth)? Design as optional resource? Ops managing user engagement metrics? Where's the security audit? Scary, even for a developer like myself.

abyssknight 16y ago
I'm a little blown away at this myself. I knew they were extremely agile, able to make changes on a whim, but this sounds a bit crazy. Might check in with my buddy who works at FB to see if this matches up with reality. The sheer stress of working in an environment where perfection is not only desired, but required, would drive me crazy. Some of the processes, though, sound like they would be a dream. Working on things I want to work on would make me 100x more productive. Of course, it also means some things may never get done.
- EllieAsksWhy 16y ago
  It would be interesting to get some verification, because no one could work in this environment for very long. Regardless of compensation. The stress and associated error rates that would accompany it would've been a problem long ago for Facebook. Particularly with no QA. I cannot imagine Facebook existing and prospering for this many years without any QA team.
brown9-2 16y ago
Perhaps this is a good example of how much of that professional "seasoning" is actually unnecessary or gets in the way of building successful products and high-caliber responsible teams, i.e. "throwing over the wall to QA" etc. edit: Can I ask why this is being downvoted? Because I am disagreeing with the parent poster? I don't think I am raising poor points or somehow not contributing to the discussion. Facebook seems to not do a lot of "standard" things because they view them as counter to their goals of building their product. I am merely stating that perhaps there is a lesson in there about the value of what some of the convential wisdom in our industry dictates.
- kenjackson 16y ago
  Upmodded... with that said, I do think that this is very industry dependent. They are a free product that people don't necessarily "rely" on. If status messages were severely delayed or dropped, it wouldn't be a huge deal (assuming they fixed it when noticed). There's very little that they can't fix after deployed, and they have no SLA with consumers (advertisers may be different). In essence they have 500 million testers. This is a bit harder to do with a database or compiler.
  - kemiller 16y ago
    App developers and other API users do in fact rely on them. Their unreliability is part of what's convincing many sites to rethink that.
jcfrei 16y ago
I'm especially wondering: who will take this platform to the next level? Is there anybody besides mark zuckerberg in charge for designing and testing out new ideas. I don't mean just new features like questions or places, but really significant changes like a facebook phone, or a new friends model to better graph you real network.

ramanujam 16y ago

Some inaccuracies pointed out by another FB engineer http://www.reddit.com/r/programming/comments/f3u0n/how_facebook_ships_code/ http://www.reddit.com/r/programming/comments/f3u0n/how_faceb...

neilk 16y ago
Copying the main points here, from Reddit user "epriest": The article is has a few inaccuracies: There is mandatory code review for all changes (i.e., by one or more engineers). I think the article is just saying that Zuck doesn't look at every change personally. People do not get called out for introducing bugs. They only get called out if they ask for changes to go out with the release but aren't around to support them in case something goes wrong (and haven't found someone to cover for you). Getting blamed will NOT get you fired. We are extremely forgiving in this respect, and most of the senior engineers have pushed at least one horrible thing, myself included. As far as I know, no one has ever been fired for making mistakes of this nature. We have automated testing, including "push-blocking" tests which must pass before the release goes out. We absolutely do not believe "most engineers are capable of writing bug-free code", much less that this is a reasonable notion to base a business upon. If this is a real quote, it comes from a very junior engineer. It is empirically untrue. The nine push phases are not concentric. There are three concentric phases (p1 = internal release, p2 = small external release, p3 = full external release). The other six phases are auxiliary tiers like our internal tools, video upload hosts, etc.
- pak 16y ago
  OK, so according to him there's code review and some automated tests. But that still doesn't address wide open access to the live DB by brand-new devs, and the complete disregard for having a dedicated QA team. Why QA? I would think FB engineers are more busy doing real work at work than fooling around on their own site (heh, the irony) and finding out that Junior Dev 39's change to the thingamabob broke chat for the umpteenth time in IE7. And as many people have pointed out here, they are a platform for many other businesses now--there's no dogfooding that--and the API stinks in terms of stability and documentation.

deleted 16y ago

[deleted]

ernestipark 16y ago

This seems to fit how I view Facebook as a whole. I find some of their technology extremely impressive such as their blazing fast search or HipHop. However, a lot of their user experience is extremely flawed and there is a big lack of consistency and intuitiveness. Examples: clicking your friend requests on the right side of the home page pops something up that tells you to click somewhere else, events are no longer notified so you have to manually check to see if you received new ones, the distinction between adding a page to your interests versus Like'ing it is not well defined, or if it is, many users are unaware, making an event for a Facebook Page is very convoluted... etc etc. I feel that issues like these are the reason why product managers and designers exist. I think Facebook is starting to really fall behind in this regard as more and more features start to clutter and complicate the site.

EGreg 16y ago

Yeah, I wouldn't want to run my company like this: re: surprise at lack of QA or automated unit tests — “most engineers are capable of writing bug-free code. it’s just that they don’t have an incentive to do so at most companies. when there’s a QA department, it’s easy to just throw it over to them to find the errors.” It does explain how facebook ships with bad documentation etc. -- well, now we have enough knowledge to know how to compete with it :) Kasparov often remarked how a good process is more important than the actual participants. An average human and pretty good computer with a great system won the championship against great computers and against great grandmasters. Google believes in this, and their products are very well engineered, with full documentation, videos etc. (although admittedly, many haven't taken off). Yahoo definitely understands this. But these are the same companies that are losing to facebook because of social. If google and yahoo understood the dynamics of social, we would all be better off.

kenjackson 16y ago

I'm really surprised that all engineers have free access to the Facebook database. That's really scary. I know at other organizations, with much less sensitive data, this stuff was under lock and key. I have a hard time believing many aren't abusing this, just out of human nature ("did she mention me to any of her friends?").

ssclafani 16y ago
Multiple employees were fired for accessing private data in the past. I suspect that it is a serious taboo at the company now to even joke about it.
- pak 16y ago
  So what you're saying is that now anybody can still do it, but everybody knows not to talk about it. I'm gathering that this will never change.
  - hugh3 16y ago
    Whenever I meet anyone who works at facebook, my first instinct is to extricate myself from that conversation. Otherwise we might get to know each other, and become friends, and I'd never be sure that they weren't looking me up. My strategy for avoiding the prying eyes of facebook engineers is to be personally uninteresting to any facebook engineers. This might not work so well if I were a chick.
- kenjackson 16y ago
  That seems way to risky to me. You basically have to watch server logs for some interesting patterns. And of course, if they're smart they simply oversubscribe in a query (or temporally space out a query), and then pinpoint the data they are looking for locally. Given they have 2000+ employees this is just a process that is looking to be constantly abused. I surely don't expect people aren't talking about it, but I'm nearly as confident that it is happening in a way that most people aren't aware of. No evidence, but again, I've seen people do worse, with less access.
16y ago

theletterd 16y ago

It seems, from the article at least, that the're very much a culture of blame, which I find quite surprising. I've never been a fan of the practices of practices such as making people wear a hat if they break the build and the like. It seems counter-productive to getting things built and shipped, and just seems to make an unhappy workplace (in my mind, at least)

epriest 16y ago
This part of the article is simply untrue. The culture is extremely forgiving of mistakes. (I don't speak for my employer, Facebook.)

clojurerocks 16y ago

The more i read about facebook engineering the less impressed i am. It used to be interesting to hear about their engineering challenges. Now its just scary to hear anything about them. And i use it as a rulebook of how not to do youre infrastructure. Im amazed facebook works at all frankly. I also think its one of the ugliest sites ive ever used. Its what 5 years old now and they still dont even have some basic functionality on it. Plus as another person said its often broken.

pak 16y ago

So about 1000 people have unfettered access to all of the live data, but there are absolutely no safeguards against them modifying it, copying it elsewhere, or peeking at it? That's terrifying. I'm under the impression that Google has more a stringent philosophy about this, and even then, one of the few SRE's that they do give broad access to was caught stalking teens via GMail data. Imagine the range of bad behavior an FB engineer can get away with after just 4 weeks of "Boot Camp". Part of the reason I never made a real FB account was my general feeling in 2005 that securing a MySQL/PHP site against internal abuse takes incredible effort, and they wouldn't ever bother to do it. According to this article, that's likely true. And the "everybody can modify anything anytime" philosophy with little QA explains why, after 6 years, FB is still a gaping maw of security holes.

holman 16y ago
after boot camp, all engineers get access to live DB (comes with standard lecture about “with great power comes great responsibility” and a clear list of “fire-able offenses”, e.g., sharing private user data) This is through a third-party, so I'm not shaking the finger at Facebook outright yet, but the wording of that — sharing private user data — is kind of frightening. What about merely accessing private user data? Like you mentioned, I know Google is super-paranoid about even simple data access. And now someone from Facebook chimes in and replies to this and talks more about their developer privacy safeguards and makes us all feel better. Go!
- pollockmania 16y ago
  > And now someone from Facebook chimes in and replies to this and talks more about their developer privacy safeguards and makes us all feel better. Go! Former Facebook dev here. That's all I have to say in response.
  - pak 16y ago
    Since this is your alias, I hope you won't mind my curiosity--when did you leave the company, and do you have any thoughts about leaving?

ajsharp 16y ago

“most engineers are capable of writing bug-free code. it’s just that they don’t have an incentive to do so at most companies. when there’s a QA department, it’s easy to just throw it over to them to find the errors.” Bull. Shit. This one line makes me think that this entire post is completely hear-say. A 500 person engineering team that writes bug-free code on a system as a large as facebook? Give me a break.

krobertson 16y ago
Agreed. Anyone who works with FB apps/apis surely notices their lack of QA leads to frequent and often times routine breakage.
- peeps 16y ago
  To be fair, the ridiculously low quality of their code makes me think that it's true that they don't really bother with testing or QA. It explains a lot, really.
- lukeschlather 16y ago
  I have my Facebook set on Spanish to keep my Spanish language neurons firing even when I'm surrounded by Gringos, and their localization definitely has frequent bugs. There's been some issues for months with the birthday wall post aggregation they've been testing in production. I've frequently seen stuff like "So-and-so has written on so-and-so's muro for their birthday." So yeah, lack of QA => lots of bugs.
  - nroman 16y ago
    Almost all of Facebook's internationalization is crowd-sourced. What you're seeing probably isn't a bug, the string just hasn't been translated by a user yet.
  - sidupadhyay 16y ago
    Facebook's language translations are actually crowdsourced. It's funny that so many people would approve errors like this. There is a brief blog post about it: http://blog.facebook.com/blog.php?post=10056937130 http://blog.facebook.com/blog.php?post=10056937130 The whole system is pretty interesting. They maintain a mapping of every token on the site to every language and only user translators verify portions.

CWIZO 16y ago

What does it mean to be "publicly shamed" in this context?

siddhant 16y ago

Multiple mentions of "public shaming". Doesn't seem so good. A good engineer is anyway going to have a sense of shame inside when he/she screws up. I really don't think there's any need for "public shaming".

EllieAsksWhy 16y ago
So, so true. The tendency for perfectionism necessary to write code is already sufficient to instill a professional ethic as far as quality of work. Public shaming would be intolerable, I'd think.
gaius 16y ago
You don't get shamed for a bug in your code - you get shamed if you aren't willing to go on-call to support your own code. That IMHO is a very healthy attitude.

kunjaan 16y ago

I would like to know what are the other items in the 'clear list of “fire-able offenses” , e.g., sharing private user data)"

krummas 16y ago

The key point for me in this article was: "resourcing for projects is purely voluntary." As someone who works for an online gambling company where we are not even allowed to use the product we are building (legal/trust issues i guess), it would _rock_ to be able to have this kind of impact on features

flyt 16y ago

More info directly from a Facebook Engineer about their testing process: http://www.quora.com/What-kind-of-automated-testing-does-Facebook-do/answer/Steven-Grimm http://www.quora.com/What-kind-of-automated-testing-does-Fac...

cduruk 16y ago

Is the author of this piece an engineer working at Facebook? This part makes me think not: >I’m fascinated by the way Facebook operates. It’s a very unique environment, not easily replicated (nor would their system work for all companies, even if they tried). These are notes gathered from talking with many friends at Facebook about how the company develops and release software. So I'd take whatever is written here with a grain of salt. My communication with friends working at Facebook yielded similar thoughts but nothing that comes to what's written that implies a callous recklessness. I know for a fact that they have some code-review tools and blocking tests. Anyway, my point is that the author doesn't seem to be embedded too deeply in the engineering at Facebook and his notes are, while not outright false, definitely misleading.

sanj 16y ago

This is very close to how we develop at TripAdvisor, though we have a suite of tests that get run during a branch merge and code review (of at least one more engineer) of anything going into the livesite code base. We also have very small QA team. They work to make sure that the buttons that make us money aren't screwed up. The key to this sort of uber-agile development is that you have very, very talented engineers, a release process flexible enough to deal with errors, and a management that buys into moving so fast that mistakes will happen. I have to say that it is simultaneously exhilarating, humbling and a little terrifying to work like this. Luckily, mostly the first two.

rhizome 16y ago
I know all of this certainly has worked and does work for you, but the first thing I thought about when reading "small QA team" and "the key...talented engineers" is that QA is not very important to the company. A QA team that only checks the money buttons tells me that traditional QA has been incorporated into the developer positions themselves, which creates a conflict of interest. Couching my conclusion in the topic of this post, it would seem that Facebook gives extraordinary freedom to developers, but also hangs everything around the developers neck. Screwing up in this area sounds pretty fatal, and not an aspect of a company I would want to work for. I'm kinda old, though.
- brown9-2 16y ago
  the first thing I thought about when reading "small QA team" and "the key...talented engineers" is that QA is not very important to the company Couldn't the other corollary of this be that the company/development team has invested heavily in automated testing?
timcederman 16y ago
Very small QA team...unless you count all the feature, cross-browser and multi-language testing done by the product management team. I'm still not sure having the product team responsible for hours and hours of black-box QA testing each week was an effective use of resources.

patrickk 16y ago

"If lots of [employees] are flocking to a new business unit, that's a good sign that the opportunity is a good one. . . . If a business unit can't attract people very easily, that's a good sign that it's a business Enron shouldn't be in." - Jeff Skilling, former president of Enron http://www.gladwell.com/2002/2002_07_22_a_talent.htm http://www.gladwell.com/2002/2002_07_22_a_talent.htm Contrast that with the Facebook approach: "Resourcing for projects is purely voluntary. -a PM lobbies group of engineers, tries to get them excited about their ideas. -Engineers decide which ones sound interesting to work on. -Engineer talks to their manager, says “I’d like to work on these 5 things this week.” -Engineering Manager mostly leaves engineers’ preferences alone, may sometimes ask that certain tasks get done first. -Engineers handle entire feature themselves — front end javascript, backend database code, and everything in between. If they want help from a Designer (there are a limited staff of dedicated designers available), they need to get a Designer interested enough in their project to take it on. Same for Architect help. But in general, expectation is that engineers will handle everything they need themselves." Good god. Does this strike anyone else as disturbing? Surely every single piece of work being taken on should have the USERS needs and concerns as top priority and not sexy stuff that can attract sufficient engineering interest? What if there's important problems that are really bothering lots of users and a PM can't get anybody interested (or no-one decides to take on the problem?) Here's a complaint from a guy about maintaining a personal page and fan page: http://www.stevepavlina.com/blog/2011/01/leaving-facebook/ http://www.stevepavlina.com/blog/2011/01/leaving-facebook/ Some quotes from that piece: "As a programmer myself, I can’t fathom that it would take much technical and design effort to address these issues, and Facebook is flooded with complaints from users begging them to fix these headaches. From my perspective as a Facebook user with a very active personal page and fan page, I can’t help but get the impression that Facebook deliberately wants to make some basic admin tasks (like blocking spammers) difficult or impossible in order to compel you to spend more time on the site. There doesn’t seem to be any other logical reason for these glaring design flaws that I can comprehend, other than pure incompetence, and based on their success in other areas, it seems more likely that these choices are deliberate." "Surely someone on their team is aware of all the complaints and requests to fix the broken elements. So why do they seem to ignore what appear to be such glaring (and fixable) problems?" "I thought that Facebook would be an interesting place to share inspirational messages and build more community around growth-oriented people. But the current implementation of Facebook can’t handle the way I’ve been trying to use it without creating more headaches than it’s worth, and their momentum appears to be headed in the wrong direction for me to expect that these problems would be fixed anytime soon." "So I’ve crossed the threshold where Facebook’s value isn’t worth the hassle to use it. I concluded that the best choice was to simply drop the service altogether and invest my time elsewhere." Who will take on these issues? Again from the Malcom Gladwell article linked above: "You might expect a C.E.O. to say that if a business unit can't attract customers very easily that's a good sign it's a business the company shouldn't be in. A company's business is supposed to be shaped in the direction that its managers find most profitable. But at Enron the needs of the customers and the shareholders were secondary to the needs of its stars." Facebook should wake up in my opinion. They did really well to reach 600m users (or whatever the figure is now) but if they want to stay there they should get their priorities straight. EDIT: Grammar

anthony_franco 16y ago

"no QA at all, zero" As someone who develops on top of the Facebook Platform, I'm not surprised. Huge, obvious bugs that affect many applications are released far too often.

clojurerocks 16y ago

Serious question. Why doesnt Facebook do any engineering talks anymore? Some of the coolest talks on infrstructure from the past few years have been from FB. And yet the only thing i can remember really hearing about their stack recently is that they were using HBase. Which really came off as more about marketing and hype about some products they were working on then truly geeky stuff thats come out from FB in the past. Anybody know? I wonder if its because Fb is no longer a startup? Which might the reason why we never hear about Youtube or Twitter anymore either. Oh my other comment was just meant in jest. But thanks for downvoting it and killing all the worthless and meaningless HN karma points i had accrued!

deleted 16y ago
[deleted]
daveman692 16y ago
http://www.facebook.com/Engineering?v=app_2392950137 http://www.facebook.com/Engineering?v=app_2392950137
deleted 16y ago
[deleted]

didip 16y ago

Given the current sad state of Facebook Platform, it must be because no one is working on it. Or it is labeled "unsexy" to work on it.

bjclark 16y ago
Well, the lack of QA certainly explains a lot.

tommi 16y ago

Funny how this article has 33 main bullets points out of which 13 are or contain corrections.

Aloisius 16y ago

It sounds like they've institutionalized the herding of cats as a project management system.

deleted 16y ago

[deleted]

kschua 16y ago

"all engineers go through 4 to 6 week “Boot Camp” training where they learn the Facebook system by fixing bugs" I really like this part. Many new staffs prefer to come in and do the "sexy" enhancements instead of the "mundane" support. This leads to their lack of understanding of the system and poor design. I had always believe that new starters should do support work for a while to gain an understanding of the overall system.

jkuria 16y ago

"very engineering driven culture. 'product managers are essentially useless here.' is a quote from an engineer. engineers can modify specs mid-process, re-order work projects, and inject new feature ideas anytime. during monthly cross-team meetings, the engineers are the ones who present progress reports. product marketing and product management attend these meetings, but if they are particularly outspoken, there is actually feedback to the leadership that 'product spoke too much at the last meeting.' they really want engineers to publicly own products and be the main point of contact for the things they built." This sounds like Mark Zuckerberg living his 'revenge of the nerds' dream in his Facebook nirvana! It certainly doesn't sound like good management practice!

maxawaytoolong 16y ago
It certainly doesn't sound like good management practice! Why not? I worked for silicon valley startups for 15 years and not once did I meet a "product manager" who did anything worthwhile.
- pzb 16y ago
  Product managers are very useful when you are not your own customer. Good product managers are a proxy for your customers, providing you information on the customer wants and needs. Good product managers work with the engineers to prioritize projects; it is not a dictator position, but does help ensure that the less sexy projects get done as well as the uber-cool sexy projects.

keyle 16y ago

This just looks like some sort of marketing propaganda to me.

InclinedPlane 16y ago

Estimated Joel Test score for facebook: about 5 out of 12. http://www.joelonsoftware.com/articles/fog0000000043.html http://www.joelonsoftware.com/articles/fog0000000043.html

nikolayav 16y ago
As a Facebook engineer, I'd say the score is 10 or 11, depending on your interpretation of the tools item.

bootload 16y ago

"... What do you think? Would “developer-driven culture” work at your company? ..." Hasn't everyone read, "Microsoft Secrets" ~ http://www.amazon.com/MICROSOFT-SECRETS-Powerful-Software-Technology/dp/0028740483 http://www.amazon.com/MICROSOFT-SECRETS-Powerful-Software-Te... ...because what the author describes is pretty much the MS dev process (cf CH4 Defining products and development processes). Reading JOS, "How to be a program manager" also shows how MS PM's worked in conjunction with developers ~ http://www.joelonsoftware.com/items/2009/03/09.html http://www.joelonsoftware.com/items/2009/03/09.html MS sure knows/knew a thing or two about organising large groups of people to produce software & output product in a corporate setting.