13 ms·

Let a thousand flowers bloom, then rip 999 of them out

231 points by logic 11y ago

lucio 11y ago
I was constantly being reminded of Taylor, while reading the article. Is this the scientific management of software engineering? It looks a little too constraining for a rapid-changing discipline as SE.
nowarninglabel 11y ago
The question that has been arising for me, is what amount of resources do you devote to engineering effectiveness in a relatively small engineering org (~25 people)? It seems like the presented formula would say to not devote any resources, but there seems to be some good return on some effort put towards it. Right now, I've just considered it some percentage of my job to work on.
- JoshTriplett 11y ago
  If you have only 25 people, you have few enough that you can reasonably discuss just about everything with the whole team. So, you can reasonably talk about what kind of "engineering effectiveness" work you need, and its priority relative to the other tasks on your list. How much time you spend on it then depends on where the task falls on https://xkcd.com/1205/ https://xkcd.com/1205/ as well as how much time you can spare from your current workloads.
Rifu 11y ago
That was an enjoyable read. As mentioned in the article, here's the link to the same talk he gave at Facebook's @scale [0], though the sound gets ridiculously out of sync after a bit. [0]https://www.youtube.com/watch?v=8IyXcLFO9ns https://www.youtube.com/watch?v=8IyXcLFO9ns
aaronbrethorst 11y ago
GOOOOAAAAALLLL! Fail Whale. GOOOOAAAAALLLL! Fail Whale. Oh yeah, fail whale. I guess Twitter really has come a long way in the past five years. Despite all of the shit they get for not building out their product in a visually-engaging fashion, I have to say that I can't remember the last time their product failed me. Props.
- saosebastiao 11y ago
  That's true, but I can't help but feel like it was an extremely low bar to hurdle. If "35.6 million tweets during the Brazil/Germany final" is to be believed, that is only 5k tweets per second on average...maybe 50k peak due to heterogeneous volume during a sporting event. Call me uninformed if that's what it is, but that seems like something that should be easily handled by any decently architected system on the JVM. I couldn't even count the number of shit systems at Amazon that process that kind of volume with less than 4 9's of availability.
  - aaronbrethorst 11y ago
    http://highscalability.com/blog/2013/7/8/the-architecture-twitter-uses-to-deal-with-150m-active-users.html http://highscalability.com/blog/2013/7/8/the-architecture-tw...
    - saosebastiao 11y ago
      Facebook is getting 43x the QPS of twitter, and they are doing it on MySQL. http://highscalability.com/blog/2010/11/4/facebook-at-13-million-queries-per-second-recommends-minimiz.html http://highscalability.com/blog/2010/11/4/facebook-at-13-mil...
      - random3 11y ago
        You're quoting a post which is 5 years old and comparing it with one that's two years old.
CamperBob2 11y ago
Amusingly, that was Chairman Mao's approach as well.
- zyx321 11y ago
  Yeah, from the title you'd expect an article about Chinese politics. Turns out it's just an Effectiveness Engineer using fancy phrases without understanding their historical significance.
  - task_queue 11y ago
    This happens so often in this community that it's cringeworthy. When I got to the first mention of Twitter I hit the back button.
  - GFK_of_xmaspast 11y ago
    And people say the humanities aren't important.
jackgavigan 11y ago
> Twitter’s first line was written in 2006 by our now interim CEO, Jack Dorsey. Really? Not Noah Glass? Also, no mention of Pivotal Labs role in helping Twitter get rid of the fail whale...
- sulam 11y ago
  What role did they have? Unless you mean the pivots that Twitter hired full time? I worked at Twitter during the entire transition from the monorail to the current architecture and there was ~0 Pivotal Labs involvement in that process. I know they had a big impact on the earlier phase of Twitter but when it comes to getting rid of the fail whale a large part of the credit goes to Marius and Raffi, neither one of whom were Pivots and I can't think of a single Pivot that built a service at Twitter that is in production today.
hueving 11y ago
This article repeats the assertion that there is no such thing as a 10x engineer but doesn't site anything that shows such. It has become a popular meme but I know counterexamples from people I have worked with that were significantly more effective (10x minimum) engineers than others. Does anyone have anything that shows they don't exist? I'm starting to think it's something average engineers like to repeat to convince themselves that there isn't anyone really 'better' than them. Sort of a reincarnation of the condescending "there are no winners or losers" thing we do with childrens sports.
- x5n1 11y ago
  Though this is true. No one knows everything and the chain is as strong as its weakest link. Ultimately 10x programmers have to end up working with 1x 2x 3x.. you get the idea. They also have to deal with tools and bad decision making and bureaucracy and real life. Success is often defined by how well everyone and the market come together. Was Jack a 10x programmer, I doubt that very much. The focus should be on how to make everyone better. Not how each programmer can show off his skills at the cost to other people who might not be as good as him. In fact that's why programmers in the Valley aren't making a million a piece. They don't stick together, they compete to outdo each other, and management moves in for the kill.
  - hueving 11y ago
    Oh, I didn't intend to underplay the importance of having an effective team. It just feels like we are pretending that extremely skilled people don't exist and I'm not sure why. I know a principal engineer at (giant networking company) that does clear close to 1 million a year after bonuses and stock because of the special project he is on and he was invited to that project because he is so effective.
- InclinedPlane 11y ago
  I've come to dislike the idea of the "10x engineer" partly because it puts software development too much in the realm of factory work or even craft-work, which it is not. Fundamentally software creation is creative and inventive. It is not a matter of stamping out a certain number of widgets per day. It is not a matter of churning through some number of bugs, lines of code, function points, or what-have-you. It is design work, it is creation, it is art and artisanry. To purport to measure developers against one another along some linear measure is to miss the point entirely. It is the same as asking whether Mark Twain and Walt Whitman are "10x writers", or whether fillet mignon is 10x better than a PB&J. Not only are these things non-linear, they exhibit complexity that is irreducible to such simple measures, which is why we try to avoid the attempt (although somehow we refuse to learn that lesson when it comes to movies, television, and video games, but I digress). The top tier of developers are far more than 10x more valuable than the average developer. Not because they produce 10x more lines of code, or "crush" 10x as many bugs or sprint points, but because they build. better. systems. Period. They write better code. They build the right stuff. They promote better team cohesion. And they execute on difficult projects more effective. And they don't just make one order of magnitude difference, often they make multiple orders of magnitude difference. Here are three fairly common scenarios involving such "keystone" developers: There's a large and crucial code base which is owned and maintained by an entire team, but there is one developer who has a much different relationship to the code than the other people on the team. They wrote most of the original code, though it has grown much since then, and they know far more of it than any other single person on the team. Tasks related to the code which would require much more time and often the collaboration of multiple team members can be done alone and with much less time by this person. If they leave the project will live on, and it may even flourish, but the development velocity and the capabilities of the team will be significantly diminished compared to when the key developer was around. There's a core feature on a very important product that is in development, and there's one developer who seems to have an outsized impact on it. They've gone the extra mile to make sure the feature is designed and implemented well, they've put in the hard work to get the feature shipped with excellent quality (both in the code behind the feature and in the way it works and feels to end-users). They've made sure that the right work has gotten done and the most important core aspects have shipped. And the end result is a phenomenal product that attracts and retains users, engenders customer loyalty, and ultimately results in a massive amount of revenue for the company. There's a tiny team or a startup working on an ambitious product. Despite their limited resources they produce something of significant value and quality, something that normally would require a much vaster number of average developers working for much longer to achieve a similar result. This leads to runaway success. These scenarios happen all the time in the industry, and they speak to a gulf in capabilities between the most capable devs and the industry average. And it's not necessarily about "productivity" it's about getting the right work done, done well, and done efficiently. You put a top tier developer and a middle of the road developer on the same task for a week or two and the difference isn't that one has written a lot more code than the other, the difference is that one developer has something that works and something that you wouldn't mind keeping around. This is another thing. Good developers learn and build from their past projects. Which means that over time the developer is getting better, and also what they're able to deliver based solely on their own work grows exponentially, because it builds and builds and builds, like compound interest. Whereas mediocre developers are building foundations of sand, they can only build so high before everything comes crashing back down and they have to tear down and replace what they've built before with something a little bit better. Mediocre developers tend to spend their time running in place. P.S. Another huge factor is culture, mentoring, leadership by example, and work environment. Top tier developers raise everyone around them to a higher level through their example and their interactions with others. Learning from better developers is a hugely important way that devs get better at their job. And merely getting into a mindset that getting better is possible and valuable is important in its own right. Additionally, working with talented engineers, people we look up to, people we take inspiration from, etc. are important reasons why devs show up to work every day. When an "anchor" dev with a lot of talent leaves a team or a company it can trigger an avalanche of talent leaving the org as other talented engineers decide that the work environment isn't as worthwhile as it once was and they leave as well, accelerating the process until a lot of the talent has evaporated from the org. This is a very common process in the industry because the people who tend to be the most talented and the most intrinsically motivated tend to have the easiest time finding other jobs and tend to be the most sensitive to the quality of the work environment. Unless you're immersed in the work environment and sensitive to these things you may not pick up how drastic these changes can be until years down the road when the impact of a team that has lost all its talent finds itself vastly less able to execute on important projects effectively. And while I've talked about a lot of "soft" subjects such as "art" and "work environment", these things absolutely have an impact on the bottom line.
diminish 11y ago
At first I expected an article on Twitter platform for 3rd parties where 1000 flowers bloom and 999 of them get ripped. Twitter has a horrible, slow moving, bloated interface. I m facing countless examples of disturbances every day. Maybe a single platform based on PHP could be better, if Facebook is so usable?
- MrBra 11y ago
  Really, Facebook? Yesterday I spent half an hour on my mobile just to find out where the "view profile as it would look to someone else" page had been moved.
bbrazil 11y ago
> Once your engineering org gets to be a certain size the benefits you can obtain by investing in making all your engineers slightly more productive start to swamp the slight gains that one team might get from doing things their own, slightly different way. I like to think of this idea as group vs. individual velocity. Yes, you personally might be able to go faster by introducing new technologies - but everyone else is slowed down by having to learn it. Knowing which few of these technologies will be an overall win is the tricky bit.
plesiv 11y ago
Minor side-issue: 2014 WC final Germany/Argentina; Brazil/Germany was in semi-finals.
not-a-10xer 11y ago
>Engineers’ effectiveness, on the other hand, is hard to measure. We don’t even really know what makes people productive; thus we talk about 10x engineers as though that’s a thing when even the studies that lead to the notion of a 10x engineer pointed more strongly to the notion of a 10x office. Management has a vested interest in pushing the meme that there are no 10xers: it gives them an excuse for not paying these engineers what they're actually worth. The rest of us just go along with it because it assuages the inadequacy we naturally feel when we compare ourselves with someone like Linus Torvalds or John Carmack. Clearly there are 10xers, and some of them even post here: Walter Bright, Bryan Cantrill, and Ted Unangst. Edit: I'm also not surprised Seibel would say something like this given his past demonstration of lacking social awareness by titling his book of programmer interviews "Coders at Work"--a title that one of his interviewees, L. Peter Deutsch, rightly objected to, much to Seibel's shock. Deutsch in 1984 co-authored the seminal paper on building efficient virtual machines for dynamic languages through just-in-time compilation and inline caching, techniques which ultimately gave rise to the JVM. He deserves an immense amount of respect for this and other accomplishments, and we do him and our profession a great disservice by publicly referring to him as a "coder" or anything else that elicits something less than awe from the general public. Could you imagine doctors referring to some highly-regarded brain surgeon as a "cutter" or "butcher?" No, instead they talk them up as "brilliant" or even god-like, yet we do the equivalent with "coder" (and allow the media to do it too) all the time.
- stdbrouw 11y ago
  Albert Einstein and Richard Feynman were 10x physicists. Then there are probably a couple more 9x physicists, many of them with Nobel prizes under their belt. A lot more 5x physicists. A whole mass of 2x physicists. Another whole mass of 0.5x physicists. I don't think anyone wants to seriously dispute that there's a bell curve of aptitude in pretty much any human endeavor, whether that aptitude came naturally or was nurtured due to hard work or both. But the 10x gospel we see in software engineering is an entirely different beast: the claim is that there's a small group of people who "really get it" and that everyone else "just isn't made of the right stuff". It assumes both a kind of determinism (you do not become good, you either are good or you're not) and a hard dichotomy (there's no ordinary engineers, just bad ones and great ones). Of course management would prefer to make ordinary engineers better than to let a couple of brilliant minds work their magic. But that's just common sense: https://en.wikisource.org/wiki/Why_I_Never_Hire_Brilliant_Men https://en.wikisource.org/wiki/Why_I_Never_Hire_Brilliant_Me...
creyer 11y ago
Nice article, but I can't agree with everything inside. 1. Engineers are not computers, so saving 5 minutes a day will not increase productivity by 1%. Most engineers work with passion, and don't look at the clock. They are focused on delivering a certain task, and they will make their best to deliver it, even with the interruptions, even if this will require them to work 10 minutes more at the end of the day. 2. The benefit of the tools is a clear win, but as these tools will not change so much over time, after the initial boost of productivity, the effect will no longer be noticeable. You can't increase performance each month compared to the last month just by using the right tools. So I think the EE team make sense to be created "on demand" and not as a permanent solution. Time should be a parameter in the effectiveness model 3. If you grow to a large scale enterprise, as the people are not computers and reaching agreement between large number of individuals is not easy, the bigger the number of people in the EE team the harder will be to approach solutions and test them.
- creshal 11y ago
  > they will make their best to deliver it, even with the interruptions, even if this will require them to work 10 minutes more at the end of the day. I'd say giving your engineers 10 minutes more free time at the end of the day will give long-term returns too. Engineers are not computers, they need time to unwind after work like everyone else.
  - pc86 11y ago
    You're missing the point entirely.
    - AnimalMuppet 11y ago
      creshal has a different point. That does not mean that he/she missed the other point.
- panic 11y ago
  As organizations grow, it can also become harder to maintain the passion you mention. This is a huge factor that people usually don't account for in their models.
davedx 11y ago
I'm really interested in the "standardization" part near the end. Is the end goal to have Twitter building everything with Java, or everything with Scala, or something like that? It does seem to go against the grain of "use the best tool for the job".
anon4 11y ago
Offtopic, but why is "monorepo" pluralised as "monorepi", while "repo" is traditionally pluralised as "repos"?
- thristian 11y ago
  I'm guessing the author is poking fun at the idea of having more than one 'monorepo' by giving it a deliberately silly plural.
  - pavlov 11y ago
    The plural of "bordello" is "bordelli", so there's that. (And that's all the Italian I know.)
retrogradeorbit 11y ago
> Alex Roetter, now our SVP of Engineering, then an engineer, personally led the effort to convert what Scala code the ads team had already written into Java. I'm having trouble understanding why on earth you would rewrite Scala code to Java. The Scala code runs on the JVM and can interoperate with the Java, so what is the point?
- mseebach 11y ago
  That's cool if the Scala bits are mostly static bit, not so much if they see active development. When (say) 5% of your codebase is Scala and the rest is Java, and your engineering team prefers working in Java, those 5% become stumbling blocks. The context switching slows you down and increases the risk of introducing errors. Also, a tool chain that will support two languages in parallel, while obviously not impossible, is necessarily more complex than only supporting one.
- flurdy 11y ago
  Maintenance and when adding more features to it. Obviously runs and interops fine, but for less paper cuts when they are modifying it or near to it, it makes sense to refactor it to a common base. And removing the scala plugin from poms will make the project(s) simpler. Though as a Scala dev, not the direction I would have chosen, ever.
- technomancy 11y ago
  I'm sure faster compile times are at least part of it.
presty 11y ago
is it just me or this article points to a huge lack of technical leadership inside twitter? I wonder how much allowing things drag for so long cost the company? millions in salaries?
dunkelheit 11y ago
The article discusses the kinds of problems that wildly successful teams of 1000-s of engineers have. As such it is kind of irrelevant for most of us - not everybody will grow their team to 1000 engineers or assume the role of VP Eng in a large organization. A more practical matter to consider, are these problems worth worrying about from the start or is "let the thousand flowers bloom" strategy part of the twitter's success?
jordigh 11y ago
I wish people would learn what the hundred flowers campaign was and not use such dumb and insensitive titles for their blog posts. It's like calling it "the final solution to the twitter problem".
- iak8god 11y ago
  Yikes. I'm a little ashamed to admit I had to look this up: https://en.wikipedia.org/wiki/Hundred_Flowers_Campaign https://en.wikipedia.org/wiki/Hundred_Flowers_Campaign Hopefully this blog post's author was unaware of the hundred flowers campaign. In any case it's very helpful of you to point this out so this meme doesn't get passed along by those of us ignorant of this bit of history.
  - deleted 11y ago
    [deleted]
- Confusion 11y ago
  I wish people would learn that words and phrases can acquire meanings of their own, independent of original meanings, origins and associations. "Let a thousand flowers bloom" has an entirely clear and positive meaning for the vast majority of English speakers.
  - jordigh 11y ago
    And Thailand does love Hitler: https://en.wikipedia.org/wiki/Nazi_imagery_in_Thailand https://en.wikipedia.org/wiki/Nazi_imagery_in_Thailand
    - Confusion 11y ago
      Do you really believe Mao was the first to ever use something equivalent to the phrase 'Let a hundred flowers bloom' in any language? That thought, and that expression, has many sources older than him.
      - 11y ago
deleted 11y ago
[deleted]
deleted 11y ago
[deleted]
im3w1l 11y ago
> E = (eng - ee) * (1 + b * ee^s ) He posits that model and draws many conclusions from it. But how reasonable is? I would expect the improvements in efficiency factor to decay much faster. If everything is done exactly right, there is an optimal efficiency factor. With no ee's there are deviations from that perfect process. As more ee's are added, more deviations are fixed and you get closer to optimal efficiency. These assumptions would mean that efficiency should converge as number of ee's approach infinity. My model suggestion would be E = (eng - ee) * (1 - b * s^(-ee) )
franciscop 11y ago
Once you get 4.000 Engineers in EE, it is worth considering creating some tools specifically for EE. Let's call this team Effective and Efficient Engineering, thus EEE.
Mz 11y ago
I strongly suspect the mathematical formula suggesting that 10,000 engineers could be as productive as 45,000 with the right support is a case of "in theory, but not reality." When tree farming was invented, the initial crop was wonderful, but they soon had to come up with a term meaning "forest death" for the sickly, underdeveloped trees that followed. A monoculture of the same plants quickly strips the soil of essential nutrients and if you cannot identify those nutrients and aggressively resupply them, you will soon find that the trees you wanted to grow and harvest will no longer thrive and somnetimes will no longer survive. A forest can produce healthy trees for generations because of the diversity of plants growing therein. Different plants use different nutrients and some replenish the nutrients used by neighboring plants. Obviously, software tools are not plants. But I have my suspicions that the diversity of code he describes developed in part for reasons like "different engineers happened to be experienced with different languages, tools, etc" but there may be cases in there where it developed that way because it was the best way to handle it. So when you start standardizing things, you may be killing off things that are critical to the health of the codebase and the company. I can see value in having someone dedicated to serving the needs of the engineers so they can be more productive. I can see a need to clean things up. But complexity itself often has inherent value that cannot be replaced or improved upon with a simpler solution. Sometimes, simplifying things amounts to throwing the baby out with the bath water. I hope they don't wind up doing that. https://en.m.wikipedia.org/wiki/Forest_dieback https://en.m.wikipedia.org/wiki/Forest_dieback
wpietri 11y ago
I mostly like this post, but there's something that troubles me about it. One of the distinctions I learned from Lean literature was that bureaucracies fall on a spectrum between controlling and supportive. I had honestly only ever experienced the former: rules and mandates to make me do what other people valued. But I came to realize that there are other possibilities. For example, when my team jointly makes a checklist we follow, that's a kind of supportive bureaucracy. I could imagine scaling that up, but it's certainly rare. In this post is a hunger for power that makes me uncomfortable. Sure, any short-term mess can be solved by bossing people around. But I've seen places where centralized quality groups become strongly counterproductive. Every mandate they issue is intended to clean up a mess. But as developers become disempowered, they do less and less cleaning on their own. Hopefully that won't happen at Twitter; there's a theme of wanting to build trust. But there's also the theme of ripping stuff out, of controlling the chaos, of whipping things into shape. That sounds much better to the person holding the whip than to those getting whipped with it.
wpeterson 11y ago
Once upon a time, there were multiple repos and multiple tools for different jobs. Then Twitter grew to thousands of engineers, many of whom were from Google. Now everything is being converted to enterprise Java in a single, monolithic repo. Is that an improvement or just bureaucratic standardization and cargo-culting from engineers raised in the google3 monolith?
throwaway4123 11y ago
I was discussing this article with another ex-Twitter employee yesterday. I quit in the middle of this monorepo clusterfuck, and my views are very different - I believe most of what happened was a Battle of the Egos. I can name names but its not worth the trouble, Twitter was a generous employer etc. He remembers most of this as a Battle of the Languages ie. Scala vs Java vs Ruby thing. Seibel has this 3rd version, having joined much later in 2013. The list of ex-Twitter engineers is probably in the thousands at this point, so you are going to hear a whole different set of narratives.Twitter is Roshomon 2.0. One thing is certain. The lack of profitability is very much due to throwing lots of cash at lots of engineers who wrote and rewrote the same bits until kingdom come, driving each other and the market nuts as to what all the fuss was about.

231 points by logic 11y ago

lucio 11y ago

I was constantly being reminded of Taylor, while reading the article. Is this the scientific management of software engineering? It looks a little too constraining for a rapid-changing discipline as SE.

nowarninglabel 11y ago

The question that has been arising for me, is what amount of resources do you devote to engineering effectiveness in a relatively small engineering org (~25 people)? It seems like the presented formula would say to not devote any resources, but there seems to be some good return on some effort put towards it. Right now, I've just considered it some percentage of my job to work on.

JoshTriplett 11y ago
If you have only 25 people, you have few enough that you can reasonably discuss just about everything with the whole team. So, you can reasonably talk about what kind of "engineering effectiveness" work you need, and its priority relative to the other tasks on your list. How much time you spend on it then depends on where the task falls on https://xkcd.com/1205/ https://xkcd.com/1205/ as well as how much time you can spare from your current workloads.

Rifu 11y ago

That was an enjoyable read. As mentioned in the article, here's the link to the same talk he gave at Facebook's @scale [0], though the sound gets ridiculously out of sync after a bit. [0]https://www.youtube.com/watch?v=8IyXcLFO9ns https://www.youtube.com/watch?v=8IyXcLFO9ns

aaronbrethorst 11y ago

GOOOOAAAAALLLL! Fail Whale. GOOOOAAAAALLLL! Fail Whale. Oh yeah, fail whale. I guess Twitter really has come a long way in the past five years. Despite all of the shit they get for not building out their product in a visually-engaging fashion, I have to say that I can't remember the last time their product failed me. Props.

saosebastiao 11y ago
That's true, but I can't help but feel like it was an extremely low bar to hurdle. If "35.6 million tweets during the Brazil/Germany final" is to be believed, that is only 5k tweets per second on average...maybe 50k peak due to heterogeneous volume during a sporting event. Call me uninformed if that's what it is, but that seems like something that should be easily handled by any decently architected system on the JVM. I couldn't even count the number of shit systems at Amazon that process that kind of volume with less than 4 9's of availability.
- aaronbrethorst 11y ago
  http://highscalability.com/blog/2013/7/8/the-architecture-twitter-uses-to-deal-with-150m-active-users.html http://highscalability.com/blog/2013/7/8/the-architecture-tw...
  - saosebastiao 11y ago
    Facebook is getting 43x the QPS of twitter, and they are doing it on MySQL. http://highscalability.com/blog/2010/11/4/facebook-at-13-million-queries-per-second-recommends-minimiz.html http://highscalability.com/blog/2010/11/4/facebook-at-13-mil...
    - random3 11y ago
      You're quoting a post which is 5 years old and comparing it with one that's two years old.

CamperBob2 11y ago

Amusingly, that was Chairman Mao's approach as well.

zyx321 11y ago
Yeah, from the title you'd expect an article about Chinese politics. Turns out it's just an Effectiveness Engineer using fancy phrases without understanding their historical significance.
- task_queue 11y ago
  This happens so often in this community that it's cringeworthy. When I got to the first mention of Twitter I hit the back button.
- GFK_of_xmaspast 11y ago
  And people say the humanities aren't important.

jackgavigan 11y ago

> Twitter’s first line was written in 2006 by our now interim CEO, Jack Dorsey. Really? Not Noah Glass? Also, no mention of Pivotal Labs role in helping Twitter get rid of the fail whale...

sulam 11y ago
What role did they have? Unless you mean the pivots that Twitter hired full time? I worked at Twitter during the entire transition from the monorail to the current architecture and there was ~0 Pivotal Labs involvement in that process. I know they had a big impact on the earlier phase of Twitter but when it comes to getting rid of the fail whale a large part of the credit goes to Marius and Raffi, neither one of whom were Pivots and I can't think of a single Pivot that built a service at Twitter that is in production today.

hueving 11y ago

This article repeats the assertion that there is no such thing as a 10x engineer but doesn't site anything that shows such. It has become a popular meme but I know counterexamples from people I have worked with that were significantly more effective (10x minimum) engineers than others. Does anyone have anything that shows they don't exist? I'm starting to think it's something average engineers like to repeat to convince themselves that there isn't anyone really 'better' than them. Sort of a reincarnation of the condescending "there are no winners or losers" thing we do with childrens sports.

x5n1 11y ago
Though this is true. No one knows everything and the chain is as strong as its weakest link. Ultimately 10x programmers have to end up working with 1x 2x 3x.. you get the idea. They also have to deal with tools and bad decision making and bureaucracy and real life. Success is often defined by how well everyone and the market come together. Was Jack a 10x programmer, I doubt that very much. The focus should be on how to make everyone better. Not how each programmer can show off his skills at the cost to other people who might not be as good as him. In fact that's why programmers in the Valley aren't making a million a piece. They don't stick together, they compete to outdo each other, and management moves in for the kill.
- hueving 11y ago
  Oh, I didn't intend to underplay the importance of having an effective team. It just feels like we are pretending that extremely skilled people don't exist and I'm not sure why. I know a principal engineer at (giant networking company) that does clear close to 1 million a year after bonuses and stock because of the special project he is on and he was invited to that project because he is so effective.
InclinedPlane 11y ago
I've come to dislike the idea of the "10x engineer" partly because it puts software development too much in the realm of factory work or even craft-work, which it is not. Fundamentally software creation is creative and inventive. It is not a matter of stamping out a certain number of widgets per day. It is not a matter of churning through some number of bugs, lines of code, function points, or what-have-you. It is design work, it is creation, it is art and artisanry. To purport to measure developers against one another along some linear measure is to miss the point entirely. It is the same as asking whether Mark Twain and Walt Whitman are "10x writers", or whether fillet mignon is 10x better than a PB&J. Not only are these things non-linear, they exhibit complexity that is irreducible to such simple measures, which is why we try to avoid the attempt (although somehow we refuse to learn that lesson when it comes to movies, television, and video games, but I digress). The top tier of developers are far more than 10x more valuable than the average developer. Not because they produce 10x more lines of code, or "crush" 10x as many bugs or sprint points, but because they build. better. systems. Period. They write better code. They build the right stuff. They promote better team cohesion. And they execute on difficult projects more effective. And they don't just make one order of magnitude difference, often they make multiple orders of magnitude difference. Here are three fairly common scenarios involving such "keystone" developers: There's a large and crucial code base which is owned and maintained by an entire team, but there is one developer who has a much different relationship to the code than the other people on the team. They wrote most of the original code, though it has grown much since then, and they know far more of it than any other single person on the team. Tasks related to the code which would require much more time and often the collaboration of multiple team members can be done alone and with much less time by this person. If they leave the project will live on, and it may even flourish, but the development velocity and the capabilities of the team will be significantly diminished compared to when the key developer was around. There's a core feature on a very important product that is in development, and there's one developer who seems to have an outsized impact on it. They've gone the extra mile to make sure the feature is designed and implemented well, they've put in the hard work to get the feature shipped with excellent quality (both in the code behind the feature and in the way it works and feels to end-users). They've made sure that the right work has gotten done and the most important core aspects have shipped. And the end result is a phenomenal product that attracts and retains users, engenders customer loyalty, and ultimately results in a massive amount of revenue for the company. There's a tiny team or a startup working on an ambitious product. Despite their limited resources they produce something of significant value and quality, something that normally would require a much vaster number of average developers working for much longer to achieve a similar result. This leads to runaway success. These scenarios happen all the time in the industry, and they speak to a gulf in capabilities between the most capable devs and the industry average. And it's not necessarily about "productivity" it's about getting the right work done, done well, and done efficiently. You put a top tier developer and a middle of the road developer on the same task for a week or two and the difference isn't that one has written a lot more code than the other, the difference is that one developer has something that works and something that you wouldn't mind keeping around. This is another thing. Good developers learn and build from their past projects. Which means that over time the developer is getting better, and also what they're able to deliver based solely on their own work grows exponentially, because it builds and builds and builds, like compound interest. Whereas mediocre developers are building foundations of sand, they can only build so high before everything comes crashing back down and they have to tear down and replace what they've built before with something a little bit better. Mediocre developers tend to spend their time running in place. P.S. Another huge factor is culture, mentoring, leadership by example, and work environment. Top tier developers raise everyone around them to a higher level through their example and their interactions with others. Learning from better developers is a hugely important way that devs get better at their job. And merely getting into a mindset that getting better is possible and valuable is important in its own right. Additionally, working with talented engineers, people we look up to, people we take inspiration from, etc. are important reasons why devs show up to work every day. When an "anchor" dev with a lot of talent leaves a team or a company it can trigger an avalanche of talent leaving the org as other talented engineers decide that the work environment isn't as worthwhile as it once was and they leave as well, accelerating the process until a lot of the talent has evaporated from the org. This is a very common process in the industry because the people who tend to be the most talented and the most intrinsically motivated tend to have the easiest time finding other jobs and tend to be the most sensitive to the quality of the work environment. Unless you're immersed in the work environment and sensitive to these things you may not pick up how drastic these changes can be until years down the road when the impact of a team that has lost all its talent finds itself vastly less able to execute on important projects effectively. And while I've talked about a lot of "soft" subjects such as "art" and "work environment", these things absolutely have an impact on the bottom line.

diminish 11y ago

At first I expected an article on Twitter platform for 3rd parties where 1000 flowers bloom and 999 of them get ripped. Twitter has a horrible, slow moving, bloated interface. I m facing countless examples of disturbances every day. Maybe a single platform based on PHP could be better, if Facebook is so usable?

MrBra 11y ago
Really, Facebook? Yesterday I spent half an hour on my mobile just to find out where the "view profile as it would look to someone else" page had been moved.

bbrazil 11y ago

> Once your engineering org gets to be a certain size the benefits you can obtain by investing in making all your engineers slightly more productive start to swamp the slight gains that one team might get from doing things their own, slightly different way. I like to think of this idea as group vs. individual velocity. Yes, you personally might be able to go faster by introducing new technologies - but everyone else is slowed down by having to learn it. Knowing which few of these technologies will be an overall win is the tricky bit.

plesiv 11y ago

Minor side-issue: 2014 WC final Germany/Argentina; Brazil/Germany was in semi-finals.

not-a-10xer 11y ago

>Engineers’ effectiveness, on the other hand, is hard to measure. We don’t even really know what makes people productive; thus we talk about 10x engineers as though that’s a thing when even the studies that lead to the notion of a 10x engineer pointed more strongly to the notion of a 10x office. Management has a vested interest in pushing the meme that there are no 10xers: it gives them an excuse for not paying these engineers what they're actually worth. The rest of us just go along with it because it assuages the inadequacy we naturally feel when we compare ourselves with someone like Linus Torvalds or John Carmack. Clearly there are 10xers, and some of them even post here: Walter Bright, Bryan Cantrill, and Ted Unangst. Edit: I'm also not surprised Seibel would say something like this given his past demonstration of lacking social awareness by titling his book of programmer interviews "Coders at Work"--a title that one of his interviewees, L. Peter Deutsch, rightly objected to, much to Seibel's shock. Deutsch in 1984 co-authored the seminal paper on building efficient virtual machines for dynamic languages through just-in-time compilation and inline caching, techniques which ultimately gave rise to the JVM. He deserves an immense amount of respect for this and other accomplishments, and we do him and our profession a great disservice by publicly referring to him as a "coder" or anything else that elicits something less than awe from the general public. Could you imagine doctors referring to some highly-regarded brain surgeon as a "cutter" or "butcher?" No, instead they talk them up as "brilliant" or even god-like, yet we do the equivalent with "coder" (and allow the media to do it too) all the time.

stdbrouw 11y ago
Albert Einstein and Richard Feynman were 10x physicists. Then there are probably a couple more 9x physicists, many of them with Nobel prizes under their belt. A lot more 5x physicists. A whole mass of 2x physicists. Another whole mass of 0.5x physicists. I don't think anyone wants to seriously dispute that there's a bell curve of aptitude in pretty much any human endeavor, whether that aptitude came naturally or was nurtured due to hard work or both. But the 10x gospel we see in software engineering is an entirely different beast: the claim is that there's a small group of people who "really get it" and that everyone else "just isn't made of the right stuff". It assumes both a kind of determinism (you do not become good, you either are good or you're not) and a hard dichotomy (there's no ordinary engineers, just bad ones and great ones). Of course management would prefer to make ordinary engineers better than to let a couple of brilliant minds work their magic. But that's just common sense: https://en.wikisource.org/wiki/Why_I_Never_Hire_Brilliant_Men https://en.wikisource.org/wiki/Why_I_Never_Hire_Brilliant_Me...

creyer 11y ago

Nice article, but I can't agree with everything inside. 1. Engineers are not computers, so saving 5 minutes a day will not increase productivity by 1%. Most engineers work with passion, and don't look at the clock. They are focused on delivering a certain task, and they will make their best to deliver it, even with the interruptions, even if this will require them to work 10 minutes more at the end of the day. 2. The benefit of the tools is a clear win, but as these tools will not change so much over time, after the initial boost of productivity, the effect will no longer be noticeable. You can't increase performance each month compared to the last month just by using the right tools. So I think the EE team make sense to be created "on demand" and not as a permanent solution. Time should be a parameter in the effectiveness model 3. If you grow to a large scale enterprise, as the people are not computers and reaching agreement between large number of individuals is not easy, the bigger the number of people in the EE team the harder will be to approach solutions and test them.

creshal 11y ago
> they will make their best to deliver it, even with the interruptions, even if this will require them to work 10 minutes more at the end of the day. I'd say giving your engineers 10 minutes more free time at the end of the day will give long-term returns too. Engineers are not computers, they need time to unwind after work like everyone else.
- pc86 11y ago
  You're missing the point entirely.
  - AnimalMuppet 11y ago
    creshal has a different point. That does not mean that he/she missed the other point.
panic 11y ago
As organizations grow, it can also become harder to maintain the passion you mention. This is a huge factor that people usually don't account for in their models.

davedx 11y ago

I'm really interested in the "standardization" part near the end. Is the end goal to have Twitter building everything with Java, or everything with Scala, or something like that? It does seem to go against the grain of "use the best tool for the job".

anon4 11y ago

Offtopic, but why is "monorepo" pluralised as "monorepi", while "repo" is traditionally pluralised as "repos"?

thristian 11y ago
I'm guessing the author is poking fun at the idea of having more than one 'monorepo' by giving it a deliberately silly plural.
- pavlov 11y ago
  The plural of "bordello" is "bordelli", so there's that. (And that's all the Italian I know.)

retrogradeorbit 11y ago

> Alex Roetter, now our SVP of Engineering, then an engineer, personally led the effort to convert what Scala code the ads team had already written into Java. I'm having trouble understanding why on earth you would rewrite Scala code to Java. The Scala code runs on the JVM and can interoperate with the Java, so what is the point?

mseebach 11y ago
That's cool if the Scala bits are mostly static bit, not so much if they see active development. When (say) 5% of your codebase is Scala and the rest is Java, and your engineering team prefers working in Java, those 5% become stumbling blocks. The context switching slows you down and increases the risk of introducing errors. Also, a tool chain that will support two languages in parallel, while obviously not impossible, is necessarily more complex than only supporting one.
flurdy 11y ago
Maintenance and when adding more features to it. Obviously runs and interops fine, but for less paper cuts when they are modifying it or near to it, it makes sense to refactor it to a common base. And removing the scala plugin from poms will make the project(s) simpler. Though as a Scala dev, not the direction I would have chosen, ever.
technomancy 11y ago
I'm sure faster compile times are at least part of it.

presty 11y ago

is it just me or this article points to a huge lack of technical leadership inside twitter? I wonder how much allowing things drag for so long cost the company? millions in salaries?

dunkelheit 11y ago

The article discusses the kinds of problems that wildly successful teams of 1000-s of engineers have. As such it is kind of irrelevant for most of us - not everybody will grow their team to 1000 engineers or assume the role of VP Eng in a large organization. A more practical matter to consider, are these problems worth worrying about from the start or is "let the thousand flowers bloom" strategy part of the twitter's success?

jordigh 11y ago

I wish people would learn what the hundred flowers campaign was and not use such dumb and insensitive titles for their blog posts. It's like calling it "the final solution to the twitter problem".

iak8god 11y ago
Yikes. I'm a little ashamed to admit I had to look this up: https://en.wikipedia.org/wiki/Hundred_Flowers_Campaign https://en.wikipedia.org/wiki/Hundred_Flowers_Campaign Hopefully this blog post's author was unaware of the hundred flowers campaign. In any case it's very helpful of you to point this out so this meme doesn't get passed along by those of us ignorant of this bit of history.
- deleted 11y ago
  [deleted]
Confusion 11y ago
I wish people would learn that words and phrases can acquire meanings of their own, independent of original meanings, origins and associations. "Let a thousand flowers bloom" has an entirely clear and positive meaning for the vast majority of English speakers.
- jordigh 11y ago
  And Thailand does love Hitler: https://en.wikipedia.org/wiki/Nazi_imagery_in_Thailand https://en.wikipedia.org/wiki/Nazi_imagery_in_Thailand
  - Confusion 11y ago
    Do you really believe Mao was the first to ever use something equivalent to the phrase 'Let a hundred flowers bloom' in any language? That thought, and that expression, has many sources older than him.
    - 11y ago

deleted 11y ago

[deleted]

deleted 11y ago

[deleted]

im3w1l 11y ago

> E = (eng - ee) * (1 + b * ee^s ) He posits that model and draws many conclusions from it. But how reasonable is? I would expect the improvements in efficiency factor to decay much faster. If everything is done exactly right, there is an optimal efficiency factor. With no ee's there are deviations from that perfect process. As more ee's are added, more deviations are fixed and you get closer to optimal efficiency. These assumptions would mean that efficiency should converge as number of ee's approach infinity. My model suggestion would be E = (eng - ee) * (1 - b * s^(-ee) )

franciscop 11y ago

Once you get 4.000 Engineers in EE, it is worth considering creating some tools specifically for EE. Let's call this team Effective and Efficient Engineering, thus EEE.

Mz 11y ago

I strongly suspect the mathematical formula suggesting that 10,000 engineers could be as productive as 45,000 with the right support is a case of "in theory, but not reality." When tree farming was invented, the initial crop was wonderful, but they soon had to come up with a term meaning "forest death" for the sickly, underdeveloped trees that followed. A monoculture of the same plants quickly strips the soil of essential nutrients and if you cannot identify those nutrients and aggressively resupply them, you will soon find that the trees you wanted to grow and harvest will no longer thrive and somnetimes will no longer survive. A forest can produce healthy trees for generations because of the diversity of plants growing therein. Different plants use different nutrients and some replenish the nutrients used by neighboring plants. Obviously, software tools are not plants. But I have my suspicions that the diversity of code he describes developed in part for reasons like "different engineers happened to be experienced with different languages, tools, etc" but there may be cases in there where it developed that way because it was the best way to handle it. So when you start standardizing things, you may be killing off things that are critical to the health of the codebase and the company. I can see value in having someone dedicated to serving the needs of the engineers so they can be more productive. I can see a need to clean things up. But complexity itself often has inherent value that cannot be replaced or improved upon with a simpler solution. Sometimes, simplifying things amounts to throwing the baby out with the bath water. I hope they don't wind up doing that. https://en.m.wikipedia.org/wiki/Forest_dieback https://en.m.wikipedia.org/wiki/Forest_dieback

wpietri 11y ago

I mostly like this post, but there's something that troubles me about it. One of the distinctions I learned from Lean literature was that bureaucracies fall on a spectrum between controlling and supportive. I had honestly only ever experienced the former: rules and mandates to make me do what other people valued. But I came to realize that there are other possibilities. For example, when my team jointly makes a checklist we follow, that's a kind of supportive bureaucracy. I could imagine scaling that up, but it's certainly rare. In this post is a hunger for power that makes me uncomfortable. Sure, any short-term mess can be solved by bossing people around. But I've seen places where centralized quality groups become strongly counterproductive. Every mandate they issue is intended to clean up a mess. But as developers become disempowered, they do less and less cleaning on their own. Hopefully that won't happen at Twitter; there's a theme of wanting to build trust. But there's also the theme of ripping stuff out, of controlling the chaos, of whipping things into shape. That sounds much better to the person holding the whip than to those getting whipped with it.

wpeterson 11y ago

Once upon a time, there were multiple repos and multiple tools for different jobs. Then Twitter grew to thousands of engineers, many of whom were from Google. Now everything is being converted to enterprise Java in a single, monolithic repo. Is that an improvement or just bureaucratic standardization and cargo-culting from engineers raised in the google3 monolith?

throwaway4123 11y ago

I was discussing this article with another ex-Twitter employee yesterday. I quit in the middle of this monorepo clusterfuck, and my views are very different - I believe most of what happened was a Battle of the Egos. I can name names but its not worth the trouble, Twitter was a generous employer etc. He remembers most of this as a Battle of the Languages ie. Scala vs Java vs Ruby thing. Seibel has this 3rd version, having joined much later in 2013. The list of ex-Twitter engineers is probably in the thousands at this point, so you are going to hear a whole different set of narratives.Twitter is Roshomon 2.0. One thing is certain. The lack of profitability is very much due to throwing lots of cash at lots of engineers who wrote and rewrote the same bits until kingdom come, driving each other and the market nuts as to what all the fuss was about.