10 ms·

Twitter SVP Chris Fry Breaks Down How His Engineering Org Works

83 points by adidash 13y ago

jtchang 13y ago
Chris mentions the 3 things that motivate people. Good video about this from RSA Animate if you haven't seen it yet: http://www.youtube.com/watch?v=u6XAPnuFjJc http://www.youtube.com/watch?v=u6XAPnuFjJc
- noddingham 13y ago
  I've been trying to use Dan Pink's ideas on motivation for a while now, it was good to hear someone at his level mention them. Hadn't watched the video in a while, thanks for linking it.
amix 13y ago
While Twitter a is cool service I can't imagine what 1000 engineers can do all day long? Based on this maybe they are building their organization wrongly and following their footsteps isn't the correct way to follow.
- chatmasta 13y ago
  But think what they have the power to do.
- noelwelsh 13y ago
  If you've worked on online software you might have some appreciation of the masses of unseen services that are essential for running the system. All web apps, except in the early days, are icebergs. Payments, analytics, control panels, monitoring, deployment etc. -- there is just a huge amount of stuff that goes on that the user typically doesn't see.
  - xorgar831 13y ago
    Their monitoring dev team alone is 22 [1], and they specifically mention decomposition of services as you say. [1] https://blog.twitter.com/2013/observability-at-twitter https://blog.twitter.com/2013/observability-at-twitter
  - amix 13y ago
    I have actually built and scaled a system similar to Twitter called Plurk (which has millions of users, billions of data items and similar use cases as Twitter). We built this with a team of a few developers and one sys-admin. While Twitter is at a much larger scale, I really doubt you need 1000 engineers to solve that problem, because even at Plurk's scale we built automated systems for scaling, monitoring, backups etc. (e.g. sharded the databases, used master-master MySQL setups that supported auto-failovers, nagios for alerts etc.) You may say: but they have mobile apps? But so do a lot of other companies that don't have 1000 engineers. You need a few really great mobile devs to build great mobile apps, again based on my experience. I think what has happened for them is that their structure isn't that good and they have scaled too fast (and in this process they have hired a lot of B and C people that aren't that good). It's a least my conclusion based on the limited information I know about them.
    - lifeisstillgood 13y ago
      I for one would be (honestly) interested in reading your article in the same vein as the original - what would you say is and is not necessary in his take?
    - Irishsteve 13y ago
      Take a look at https://github.com/twitter https://github.com/twitter this should give some insight into where they may be spending their time. There are a few projects that would be well known (Storm, Kestrel, Bootstrap, FlockDB). Although I'm 100% if they were created at Twitter or just hired in.
    - hythloday 13y ago
      If Plurk has 5 developers serving 1 million daily users, and Twitter has 1000 developers serving 100 million daily active users, you only have to believe that scaling by 2 orders of magnitude is twice as hard for it to be a tie in terms of developer:user ratio. (I suspect the write load is rather different - the figures I can find for Plurk suggest 8 billion messages in total - Twitter writes half a billion a day.)
      - 13y ago
- chavesn 13y ago
  Come on. We're supposed to doubt the merits of this entire article about Twitter's philosophy and general large-organization engineering practices because someone outside can't imagine how so many people fill their day? Because maybe a thing which we can't hope to gain an accurate perspective of indicates they're doing everything wrong? This reminds me of the story of a walk from SF to LA illustrating why software estimates are always so wrong: http://www.quora.com/Engineering-Management/Why-are-software-development-task-estimations-regularly-off-by-a-factor-of-2-3/answer/Michael-Wolfe http://www.quora.com/Engineering-Management/Why-are-software... Instead shouldn't we just evaluate the content on its own merits?
  - beagle3 13y ago
    Look at e.g. Instagram or imgur or reddit or HN for a lean team. It's hard to compare without intimate knowledge of what either team faces, but without considering alternatives, many things that shouldn't look plausible actually do, e.g. The $5,000 hammers the army supposedly bought, the (incorrect, but oft repeated) space pen story, or the clusterfuck that is US healthcare.
- a3voices 13y ago
  They probably make awesome unit test scripts.
- hythloday 13y ago
  This is a bit of an unfair request, but it would be interesting to have you break down the engineering groups you think exist at Twitter and estimate their size.
- deleted 13y ago
  [deleted]
- reeses 13y ago
  This is pretty common in the growth phase of a company, especially one with easily scalable demand such as a software company. As others have pointed out, a lot of projects have been built from 'scratch' due to Twitter's needs and/or culture. This alone has a heavy carrying cost, as they now have the burden of having at least two maintainers for mission critical applications and services. It doesn't mean the engineer-tuples are bijective onto the various projects, but there are only so many tickets a person can resolve. At Amazon, we grew to what we then thought was an insane size in IT (and across the company – we had horrid little offices all over the place in Seattle). This was boosted by a number of acquisitions, such as drugstore.com and homegrocer.com. In 2000, we had a small round of layoffs, then they had another round in 2001. I would expect something similar at Twitter if they don't move horizontally. I would say that, given the competitive HR landscape, it's quite possible they intentionally overhired to secure strong engineers and will rank and yank those who have not turned out to be performers. They're still in an annealing phase, but, honestly, I'm unqualified to speculate on their internal alignment, growth plans, or any RIFs they have planned in the near future.
nitinupparpelli 13y ago
Snapchat has released an official post about the recent leak of 4.6M usernames and phone numbers from its servers.
jph 13y ago
He highlights Twitter EventParrot and MagicRecs, both compelling experiments: EventParrot for news alerts: http://www.theverge.com/2013/10/10/4823278/twitter-eventparrot-account-news-alerts-rumor http://www.theverge.com/2013/10/10/4823278/twitter-eventparr... MagicRecs for notifications: http://www.theverge.com/2013/9/24/4767290/twitter-will-notify-you-about-popular-people-and-their-tweets http://www.theverge.com/2013/9/24/4767290/twitter-will-notif...
random2 13y ago
"Breaks Down How His Engineering Org Works" is an exaggeration. "Offers a glimpse" would have been more appropriate: 1000+ engineers that include (perhaps a majority) of people working on reliability/operations. Mobility - engineers _can_ move every quarter to a team that has an open position Peer-feedback (360?) promotions based system (this is rather popular) I guess I was hoping for something more about the actual structure and interactions, but it's mostly saying it's not too centralized, nor too distributed and that it's "like a school" and they're using lean/agile methodologies.
mathattack 13y ago
I like their counterpoint to the idea that you have to dump all the managers. It really is about managerial effectiveness. I also like their idea of a free market for transfers within the company. This is a big departure from many large companies where your boss has to sign off on your move. Ultimately that creates an environment where you need a job offer in hand to switch.
- yawz 13y ago
  Well... it looks like there needs to be an opening first (which is the obvious thing, I guess) and also who decides when there is a single place but 4 applicants? It sounds like the team would decide. After what? interviews?
  - mathattack 13y ago
    That's usually how it works for internal transfers. If the person is well known enough, then the team (or manager) may decide without as formal an interview process.
narsil 13y ago
Regarding the dig at stack ranking: it's really just Yahoo now. Microsoft abandoned it 2 months ago. [1] [1] http://www.theverge.com/2013/11/12/5094864/microsoft-kills-stack-ranking-internal-structure http://www.theverge.com/2013/11/12/5094864/microsoft-kills-s...
- T_Rex 13y ago
  eBay also has stack ranking.
- vellum 13y ago
  It might only be temporary at Yahoo. If they have a lot of C players, they might be calculating that there won't be a lot of false positives in the early purges.
- reeses 13y ago
  Every company is stack ranking and every good manager (and many bad ones) have their teams ranked in the event that they need to cut or have the opportunity to improve the team through training, promotion, etc. ("What would you do if I gave you four open positions?") You have that guy who you're putting on plan because he's late with his deliverables, doesn't really 'get' the purpose of the department/group/company, and kvetches all the time about one thing or another? You've ranked him. That woman you're trying to mentor as much as possible so you can promote her, maybe even to your peer level? Ranked. There are people who are essential to an organization at different times. Keeping the lights on? I've cut everything but devops and maintenance, including myself. Sometimes you can save someone, but you can't transfer a problem.
coldcode 13y ago
1000 engineers is not a very lean organization. Maybe they sit around all day and try to figure out some way for Twitter to actually make money? Or maybe they count everyone tech related as an engineer, like designers and QA and build automation, etc? Google says Twitter has 2,300 employees.
alpastor 13y ago
He has no eyebrows.

83 points by adidash 13y ago

Chris mentions the 3 things that motivate people. Good video about this from RSA Animate if you haven't seen it yet: http://www.youtube.com/watch?v=u6XAPnuFjJc http://www.youtube.com/watch?v=u6XAPnuFjJc

noddingham 13y ago
I've been trying to use Dan Pink's ideas on motivation for a while now, it was good to hear someone at his level mention them. Hadn't watched the video in a while, thanks for linking it.

amix 13y ago

While Twitter a is cool service I can't imagine what 1000 engineers can do all day long? Based on this maybe they are building their organization wrongly and following their footsteps isn't the correct way to follow.

chatmasta 13y ago
But think what they have the power to do.
noelwelsh 13y ago
If you've worked on online software you might have some appreciation of the masses of unseen services that are essential for running the system. All web apps, except in the early days, are icebergs. Payments, analytics, control panels, monitoring, deployment etc. -- there is just a huge amount of stuff that goes on that the user typically doesn't see.
- xorgar831 13y ago
  Their monitoring dev team alone is 22 [1], and they specifically mention decomposition of services as you say. [1] https://blog.twitter.com/2013/observability-at-twitter https://blog.twitter.com/2013/observability-at-twitter
- amix 13y ago
  I have actually built and scaled a system similar to Twitter called Plurk (which has millions of users, billions of data items and similar use cases as Twitter). We built this with a team of a few developers and one sys-admin. While Twitter is at a much larger scale, I really doubt you need 1000 engineers to solve that problem, because even at Plurk's scale we built automated systems for scaling, monitoring, backups etc. (e.g. sharded the databases, used master-master MySQL setups that supported auto-failovers, nagios for alerts etc.) You may say: but they have mobile apps? But so do a lot of other companies that don't have 1000 engineers. You need a few really great mobile devs to build great mobile apps, again based on my experience. I think what has happened for them is that their structure isn't that good and they have scaled too fast (and in this process they have hired a lot of B and C people that aren't that good). It's a least my conclusion based on the limited information I know about them.
  - lifeisstillgood 13y ago
    I for one would be (honestly) interested in reading your article in the same vein as the original - what would you say is and is not necessary in his take?
  - Irishsteve 13y ago
    Take a look at https://github.com/twitter https://github.com/twitter this should give some insight into where they may be spending their time. There are a few projects that would be well known (Storm, Kestrel, Bootstrap, FlockDB). Although I'm 100% if they were created at Twitter or just hired in.
  - hythloday 13y ago
    If Plurk has 5 developers serving 1 million daily users, and Twitter has 1000 developers serving 100 million daily active users, you only have to believe that scaling by 2 orders of magnitude is twice as hard for it to be a tie in terms of developer:user ratio. (I suspect the write load is rather different - the figures I can find for Plurk suggest 8 billion messages in total - Twitter writes half a billion a day.)
    - 13y ago
chavesn 13y ago
Come on. We're supposed to doubt the merits of this entire article about Twitter's philosophy and general large-organization engineering practices because someone outside can't imagine how so many people fill their day? Because maybe a thing which we can't hope to gain an accurate perspective of indicates they're doing everything wrong? This reminds me of the story of a walk from SF to LA illustrating why software estimates are always so wrong: http://www.quora.com/Engineering-Management/Why-are-software-development-task-estimations-regularly-off-by-a-factor-of-2-3/answer/Michael-Wolfe http://www.quora.com/Engineering-Management/Why-are-software... Instead shouldn't we just evaluate the content on its own merits?
- beagle3 13y ago
  Look at e.g. Instagram or imgur or reddit or HN for a lean team. It's hard to compare without intimate knowledge of what either team faces, but without considering alternatives, many things that shouldn't look plausible actually do, e.g. The $5,000 hammers the army supposedly bought, the (incorrect, but oft repeated) space pen story, or the clusterfuck that is US healthcare.
a3voices 13y ago
They probably make awesome unit test scripts.
hythloday 13y ago
This is a bit of an unfair request, but it would be interesting to have you break down the engineering groups you think exist at Twitter and estimate their size.
deleted 13y ago
[deleted]
reeses 13y ago
This is pretty common in the growth phase of a company, especially one with easily scalable demand such as a software company. As others have pointed out, a lot of projects have been built from 'scratch' due to Twitter's needs and/or culture. This alone has a heavy carrying cost, as they now have the burden of having at least two maintainers for mission critical applications and services. It doesn't mean the engineer-tuples are bijective onto the various projects, but there are only so many tickets a person can resolve. At Amazon, we grew to what we then thought was an insane size in IT (and across the company – we had horrid little offices all over the place in Seattle). This was boosted by a number of acquisitions, such as drugstore.com and homegrocer.com. In 2000, we had a small round of layoffs, then they had another round in 2001. I would expect something similar at Twitter if they don't move horizontally. I would say that, given the competitive HR landscape, it's quite possible they intentionally overhired to secure strong engineers and will rank and yank those who have not turned out to be performers. They're still in an annealing phase, but, honestly, I'm unqualified to speculate on their internal alignment, growth plans, or any RIFs they have planned in the near future.

nitinupparpelli 13y ago

Snapchat has released an official post about the recent leak of 4.6M usernames and phone numbers from its servers.

jph 13y ago

He highlights Twitter EventParrot and MagicRecs, both compelling experiments: EventParrot for news alerts: http://www.theverge.com/2013/10/10/4823278/twitter-eventparrot-account-news-alerts-rumor http://www.theverge.com/2013/10/10/4823278/twitter-eventparr... MagicRecs for notifications: http://www.theverge.com/2013/9/24/4767290/twitter-will-notify-you-about-popular-people-and-their-tweets http://www.theverge.com/2013/9/24/4767290/twitter-will-notif...

random2 13y ago

"Breaks Down How His Engineering Org Works" is an exaggeration. "Offers a glimpse" would have been more appropriate: 1000+ engineers that include (perhaps a majority) of people working on reliability/operations. Mobility - engineers _can_ move every quarter to a team that has an open position Peer-feedback (360?) promotions based system (this is rather popular) I guess I was hoping for something more about the actual structure and interactions, but it's mostly saying it's not too centralized, nor too distributed and that it's "like a school" and they're using lean/agile methodologies.

mathattack 13y ago

I like their counterpoint to the idea that you have to dump all the managers. It really is about managerial effectiveness. I also like their idea of a free market for transfers within the company. This is a big departure from many large companies where your boss has to sign off on your move. Ultimately that creates an environment where you need a job offer in hand to switch.

yawz 13y ago
Well... it looks like there needs to be an opening first (which is the obvious thing, I guess) and also who decides when there is a single place but 4 applicants? It sounds like the team would decide. After what? interviews?
- mathattack 13y ago
  That's usually how it works for internal transfers. If the person is well known enough, then the team (or manager) may decide without as formal an interview process.

narsil 13y ago

Regarding the dig at stack ranking: it's really just Yahoo now. Microsoft abandoned it 2 months ago. [1] [1] http://www.theverge.com/2013/11/12/5094864/microsoft-kills-stack-ranking-internal-structure http://www.theverge.com/2013/11/12/5094864/microsoft-kills-s...

T_Rex 13y ago
eBay also has stack ranking.
vellum 13y ago
It might only be temporary at Yahoo. If they have a lot of C players, they might be calculating that there won't be a lot of false positives in the early purges.
reeses 13y ago
Every company is stack ranking and every good manager (and many bad ones) have their teams ranked in the event that they need to cut or have the opportunity to improve the team through training, promotion, etc. ("What would you do if I gave you four open positions?") You have that guy who you're putting on plan because he's late with his deliverables, doesn't really 'get' the purpose of the department/group/company, and kvetches all the time about one thing or another? You've ranked him. That woman you're trying to mentor as much as possible so you can promote her, maybe even to your peer level? Ranked. There are people who are essential to an organization at different times. Keeping the lights on? I've cut everything but devops and maintenance, including myself. Sometimes you can save someone, but you can't transfer a problem.

coldcode 13y ago

1000 engineers is not a very lean organization. Maybe they sit around all day and try to figure out some way for Twitter to actually make money? Or maybe they count everyone tech related as an engineer, like designers and QA and build automation, etc? Google says Twitter has 2,300 employees.

alpastor 13y ago

He has no eyebrows.