7 ms·
Engineering management at Facebook
- gabaix 16y agoimpressive insights.
- spitfire 16y agoHe's wrong. You should not focus on tools, focus on people and ideas. The people will then make the tools (and throw them away when they cause too much friction) as needed. But Facebook is still young, they're still learning. Unfortunately they don't seem to be learning from the past, which means they get to repeat everyone else's mistakes.
- jasonwatkinspdx 16y agoI think you're both in agreement. I believe the point about tools was not to enshrine any particular tool, but rather, to create a culture where internal tools are viewed as being the most important technical contribution. This is backwards from many technology companies where internal tooling is often given a lower priority and the work assigned to less skilled engineers. This is often done with noble intent, like to try to create the best value possible for customers via improving customer facing features. But I think that proves a short term strategy rather than focusing on enabling your organization as a whole. You'd be surprised how many resources become free for feature work when your tools and infrastructure are so good that you don't wast effort on what could be automated or prevented.
- yishan 16y agoHi, I'm the author. This comment is correct. My intention is not to prioritize tools over thinking about anything else. If you read the essay linked from it, I said "your most talented engineers should be working on your tools." So when I say "highest priority," I mean development priority, i.e. you shouldn't have your best people working on features and your second-tier people doing tool work, like many organizations do.
- spitfire 16y agoYou might like to short-circuit a lot of this learning. There was a fighter pilot who studied this for 40 years named John Boyd. He architected the f15, 16, 18, A-10 and the first iraq invasion. His academic work short circuits what all the dotcoms (including facebook) are slowly learning through trial and error. If you want a starting place Robert corams book on boyd is excellent, while Frans Osinga's thesis Science, strategy and war is essential reading for the hard subject matter. Here's destruction and creation. http://www.goalsys.com/books/documents/DESTRUCTION_AND_CREATION.pdf http://www.goalsys.com/books/documents/DESTRUCTION_AND_CREAT...
- spitfire 16y agoSorry I responded too quickly. People, Ideas, Hardware (technology) In that order are what you should focus. Near 100% of managements time should be spent on people and ideas. Get that right and the people will take care of the technology. One tool to implement that is a mission order (No civilian counterpart), which is where you define an intent to be achieved and operating parameters to be met but not how that intent is to be carried out. parameters are mutually agreed on - and the subordinate has the right to refuse. Usually you leave most of the definition of the intent up to the subordinate. Like "build a database useful for storing graph data" or "Find a way to improve ads customer acquisition". EFAS culture - An implementation of blitzkrieg for business gleaned from interviews done in the 70's implements this nicely.
- unoti 16y agoPerhaps it's just word definitions you and the author are disagreeing on. Making the organization more efficient is a big win, we can probably all agree. Making the people more efficient through improved systems is a big win, we can probably all agree. Letting the people who perform the process figure out how to make it better is a vitally important part of that, though, as you point out. But making tools and systems improvement a priority doesn't necessarily mean that we're going to elevate the systems to the point that they become sacred cows even when they get in the way of getting things done.
- spitfire 16y agoWhat about making people better at improving systems? That's what I'm talking about and what I've seen. Operating inside the OODA loop gives you exponential results. Technology is just an artifact to assist people. When you get to the point where people truly are first you create and throw away systems with ease. and you get real speed. But hey, this is HN. So it's all just internet masturbation anyways.
- Tamerlin 16y agoNo, he isn't. Tools need to be a high priority, or else they end up being crap. Amazon, Disney, Northrup Grumman, SAIC... all of the big companies I've worked for had the same problem: their tools were terrible. At most of them it just meant that HR ended up wasting time dealing with crappy tools, which is no big loss, since HR is a usually a waste of oxygen anyway. At Disney and Amazon, it meant that the people producing consumer-facing content wasted a lot of time fighting the tools that were supposed to enable their jobs, and a lot of developers had to waste a lot of time supporting the tools that weren't doing their job. For example, just two and a half years ago, one of the buyers at Amazon showed me how she set up the relationships so that when you look at an iPod, you get a collection of compatible accessories. She had to create relationships for EVERY permutation between each accessory and for each accessory to the iPod. By hand. In a spreadsheet (Excel, I think.) Then she uploaded it to the site, and every once in a while a relationship would mysteriously fail, because of another relationship managed in another tool that she had no access to conflicted with it.
- sdizdar 16y agoThere is actually a problem when organizations are focusing too much on tools. In order to efficiently address a disruptive innovation, a corporations need to change their processes and since tools are connected to processes, these good tools make that change even harder. Yes, it good to have good tools for developing the current product, but organization needs to be ready to eliminate internal tools without hesitations.
- Tamerlin 16y agoI agree -- becoming too attached to existing tools is a major drag industry-wide. Even if it's well-designed in the beginning, which from what I've seen is extremely rare, it tends to devolve into a mess given enough time. To get maximal value out of the tools, everyone has to be willing to toss them when they start to becoming a big enough impediment. (Deciding on that isn't always easy, of course.)
- Swannie 16y agoIt is a case of letting the best people determine when a new tool will speed up a process and having the environment where they can take the time to write the tool. Too often the environment is hostile to what might be a risky investigation. I've spent a week investigating a tool to potentially save four weeks. I failed, as I hadn't sufficiently understood the complexity of the problem. But ultimately that failure led to a better understanding. I changed the way we carried out the task, saving us a week anyway. My project manager at the time thought that week was a total waste of time. (Thankfully he was later "let go" when 360 degrees of people who had contact were saying the same things). If I do that task again, I will have a go at v2 of the tool, and probably get it right. Too many people think tools are a waste of developer time. It appears that the best tools where I've worked have been created in developer's free time, and then used widely by the team, almost to the shock of the management. Only then do the developers get a little work time to maintain or even improve them. But yes, you can go the other way. In the largest organisation I have worked in, there was the "infra" team. Their job was to provide the environments, from source control to db instances (Oracle ones), test environments (multi VM set-ups), etc. However their removal from the actual development team resulted in tools and processes which took about as long as for us to do these things with our own tools... the same ones they'd taken to improve.
- danielharan 16y ago"You will begin to get the (objectively) best candidates" If "it's [everyone's] job to say no-hire" when they're "not sure" about a candidate, I'd like to know what is done to avoid systemic bias in hiring decisions. Anyone here from FB able to comment? How diverse is the work force, especially compared to applicants?
- yishan 16y agoHi, I'm the author. Someone alerted me that this showed up on HN, so I figured I should come by to clarify. I believe my statement might be clear. What I mean by that is "it's [everyone's] job to say no-hire," as opposed to anyone feeling like "well, I'm not sure about this candidate, but I'll let someone else reject them because I don't want to be perceived as mean." One problem with hiring is that everyone must be willing to uphold the standard - if you have too many people who are never willing to be the one to say no-hire, the standard will fall - everyone has to be willing to take the responsibility for saying no. That said, systemic bias is an orthogonal (and real) issue. Being "not sure" should be applied to "does this candidate measure up" not "is this candidate a lot like me?"
- cpeterso 16y agoSaying "no hire" can be hard for some people, but becomes easier if your team adopts the attitude that "maybe means no". It's probably better for the team to miss out hiring some promising "maybe" candidates than later trying to fire a bad one.
- danielharan 16y agoGood to know. Does Facebook collect data about how they're doing internally to avoid these systemic biases? Is this tracked? Although it appears orthogonal, if everyone starts making hiring decisions without any checks and balances that an HR person would put in, I'd assume more bias to creep in. At the very least, I'd want the "standard" to be clearly defined.
- tgflynn 16y agoOne of the things that strikes me about what I've heard about hiring practices is that they all seem primarily focussed on keeping the wrong people out rather than on finding the right people. Does anyone consider the possibility that there might be really capable people who don't fit the model that N developers have of what a good candidate should be ?
- unoti 16y agoHis thoughts on internal tools and support are very insightful. It's why I am typically sad when a company outsources its support. When a company does its own support with internal personnel, it has a vested interest in making the customers happy and also serving them as efficiently as possible. When a company outsources its technical support, there is no real incentive for the outsource partner to let the company know how the processes and tools can be improved and automated. In fact, quite the opposite. If doubling the business means double the support costs, that works out great for the outsourcing partner. Perhaps it's still possible to outsource the support work, but still care deeply about the details of how that work is done, and how the process can be improved/automated with systems. I haven't seen it work that way, however.
- deleted 16y ago[deleted]
- dacort 16y agoThe processes section intrigues. As the "CTO" in a 4-person startup, I'm constantly juggling between getting $hit done and documenting what the heck I did to get $hit done. Seems to be a fine balance.
- DavidMcLaughlin 16y agoYou can say shit here.
- superhacker 16y agoAre you Scrooge McDuck?
- randfish 16y agoI like much of the advice, but the style and presentation of the message is, like so many things I've seen associated with Facebook, lacking any sense of humility. There's no "we did this because it worked for us and we're sharing in hopes that it may help others." Instead it's "Other people think this. They are wrong." Or "A commonly held belief is X. It is false." This hubris has certainly been a powerful ally to Facebook's founder, but like so many other powerful people, companies, governments and organizations that came before them, I can't help but think it will ultimately lead to demise (unless tempered).
- Nitramp 16y agoWhile hubris can be a problem, I'm very much a fan of stating your opinion clear and bold. One might be wrong, but astute readers should be able to judge ideas for themselves. Speaking clearly will also make it much easier for people to see if you're wrong. I think giving clear statements of your opinion and being proven wrong is much better than obscuring your point with "maybe" and "personally" and "IMHO". That's depending on the context of course, as you don't want to unnecessarily offend people (which is bad in general, but also counterproductive). But in a general analysis like this, not attacking particular people's shortcomings? Go for it.
- joelhaus 16y agoA reminder to "think for yourself" can be helpful, and does not necessarily imply a lack of clarity or boldness. Sometimes a story is told so well, that you get emotionally wrapped up and forget to take a step back and make a fair-minded analysis. Not everyone is an astute reader. You also included some conditional statements with, "depending on the context" and "But in a general analysis". I see no problem with acknowledging that things aren't always black & white.
- derefr 16y ago> Not everyone is an astute reader. I have to single this out: must we all prepare our rhetoric for the remedial education of those who were never made self-sufficient in the processing of external knowledge into internal beliefs?
- wooster 16y agoI spent a few years at Apple writing internal tools. IMHO, this is the way to have an effect on a company completely disproportionate to any other activity I know of. The tools I wrote at Apple have enabled projects which, AFAIK, are completely unheard-of at any other tech company. That said, don't expect a payoff proportionate to the effect of the tool. From a company owner's perspective, however, excellent tools can provide an advantage that is difficult for competitors to match. That's worth an awful lot.
- leftnode 16y agoYup, I've done this as well at current and past jobs. I'll usually just invent a new project, it gets the boss's attention and they like it, so I get to work on that rather than the normal boring stuff. It's great to invent your own projects at work.
- wizard_2 16y agoDitto - the firt few tools were small and only used by my team, the next few were big and used by everybody eventually, working on round 3...
- sdfghrjtr 16y ago●---------------------------------------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●------ http://goo.gl/l5v0b http://goo.gl/l5v0b ------● ●---------------------------------------●
- Isamu 16y agoSo the problem of "hiring the best" is solved by making hiring your top priority. Likewise the problems of development are solved by making tools your top priority. Presumably along with everything else that is a top priority, like making something insanely great. I'm sorry, I just sensitive to how many times I've seen "top priority" in somebody's management presentation, as if it solves something. That said, I mostly agree with the gist of what he's saying here.
- eitally 16y agoI know what you mean, but you may be surprised to know how many people feel like their organization doesn't have any top priorities, so overkill on the hyperbole isn't necessarily a bad thing. One of the issues in large organizations is trying to engage staff strongly enough so they really do feel a personal responsibility for the success of the company. It's hard, and very few large corporations do it well.
- Isamu 16y agoI agree this is a real problem. I guess I've come from organizations where everything is declared a top priority, therefore nothing is. And nothing can be pushed aside for a true priority. In particular, it is damaging for a leader to declare a top priority and then go on the same as usual, not making extra efforts to invest in that priority, because of course the reality is that so many things are important, and they lack the will to focus at the expense of something else. If something is that important, you'll direct significant resources to the goal and I imagine the author of the article is sincere in doing that. Many organizations however are resource-constrained and must make tough choices. It's just mismanagement to declare lots of top priorities - heck, you don't have to say anything, just show us how you are allocating the money and people, and adjusting your expectations of everybody's schedules and deadlines.
- eitally 16y agoI read this with my head nodding the whole time. :(
- sdizdar 16y agoWith all due respect to Facebook and many great engineers at Facebook, if you look at quality of Facebook API ( documentation, bugs, reliability, compatibility), I don't think they followed "hiring the best" in the division working on Facebook API.
- dasil003 16y agoI disagree. The fact is that Facebook does not optimize for documentation, reliability or compatibility. The reason is because their goal is to be as agile as possible, and iterate extremely quickly. It's true that this makes life terrible for developers, but that's not a problem because Facebook does not have a shortage of developers. Over the past 3.5 years the Facebook platform has evolved so quickly that a lot of bugs simply disappeared by deprecation; if they had spent more time on solid architecture they would have left a lot on the table business-wise. Also, I'm not sure if you used the Facebook API when it was new, but I remember in 2007 being distinctly blown away by how functional it was and how far ahead of the curve they were on web APIs. You can argue that "the best" engineers wouldn't churn out something so ephemeral, but that's subjective.
- Swannie 16y agoA number of good responses about writing good tools. That's a no brainer. It's scary to how many project managers I've had to explain why I have allocated 20% of my project planning to writing a new tool. They see it as wasted time because they don't think that we will be redoing the task again... when it's something sales try and sell with every project?! The biggest thing that struck me were the strong statements about technical managers. I think the sentiment is correct, technically experienced managers are great. But it reads like you expect all technical managers to be up to date on their coding? Or just be competent at writing pseudo-code? Hopefully it is the latter!