5 ms·

I wondered this before joining a large tech company. As it turns out, a lot of engineering is done on experimentation to optimize the product. Only a small port

I wondered this before joining a large tech company. As it turns out, a lot of engineering is done on experimentation to optimize the product. Only a small portion of users might see the experiment, and many don't succeed. An example might be complying with food and alcohol law in a single region, which you would never see unless you lived there. Another example might be a complete experimentally gated refactor that shaves 1 second off loading time. A lot of effort would go into simply making sure the change can be turned off, and verifying with unit tests, which takes time and effort. These projects might be pointless on a small app, but in a large business might be worth millions to the company. Another aspect is your changes often have to be approved by a lot of people, which is not the case on small projects. Your projects are also analyzed by data scientists, to make sure they're not harming the business - and if they are, your work might be thrown out, which contributes to even less visible change to outsiders. All of these in concert slow things down, and necessitate staff.

madamelic 7y ago
>Another aspect is your changes often have to be approved by a lot of people, which is not the case on small projects. So you need more engineers because you hired more middle managers? That's a little absurd in my opinion but I am likely wrong in this assumption. It just seems like the reaction to "We have lots of approvals" should be figuring out how to reduce the amount of approvals needed, not hiring more engineers to increase velocity.
- neltnerb 7y ago
  I think you need as many approvals as your risk tolerance allows for... they're not pointless. If they are pointless then sure, get rid of them. But I don't know how you'd guess that number without deep knowledge of the business. It is sometimes better to go slower than you could and be more confident that the product is acceptable. Approvals can but don't necessarily help with that.
- deminature 7y ago
  The approvals usually come from other engineers. It stems from having to modify highly-sensitive code which usually has an owner you need to work with to get an approval.
- ianceicys 7y ago
  You're not hiring more middle managers because you want too, no large companies have to promote engineers to senior engineering managers to handle the complexities of approvals required by compliance such as SOC2,SOC3,ISO27001,FedRAMP, GDPR, etc. I've worked at companies were every line of code has to be reviewed and approved by a lawyer. Yes a lawyer (usually a former engineer that the company paid to go to law school) is reviewing some dumb little javascript or python script for license and legal compliance.
  - sjs7007 7y ago
    > I've worked at companies were every line of code has to be reviewed and approved by a lawyer. Yes a lawyer (usually a former engineer that the company paid to go to law school) is reviewing some dumb little javascript or python script for license and legal compliance. Jesus, what company was that? And how about libraries included... Will this lawyer now also need to go through that?
- ratww 7y ago
  It's not just approvals from managers, sometimes you need approval from a group of peer engineers just to create new routes in an (internal) API, or to launch a new cron job that will upload an Excel spreadsheet in marketing once a day with data from Google Forms. Also, UX wants to see the final result on their screen before they approve your build, so before deploy you have to push into a test cluster, but there are only 10 of those, and you need to wait until one of them is free. And of course your Jenkins server takes 2h to build and fails sometimes, so pushing small changes takes 2h because you have to babysit the build. The testing cluster above also takes 2h to build. CTO believes in engineers taking care of their code, so SRE is not allowed to handle the builds. And then there's an average of 2h of meetings per day (I'm not including the 40 minute daily scrum because your squad has 15 people). Half of them to discuss the production incident you had because you didn't see the Slack alerts when you were in another meeting. The other half is chapter and guild meetings where you don't talk, but participation is mandatory. On Mondays you have sprint planning, so forget about coding. Engineering fact: A 300 watt amplifier is only twice as loud (+10 dB) as a 30 watt amplifier. The same math works for engineering teams.
  - ThrowMeAwayOkay 7y ago
    Right here. Everyone read the above, this answers the question of why so many?
- twblalock 7y ago
  The approvals often exist for valid reasons. You can't just get rid of them. You can try to streamline the approval process but that only gets you so far.
- dan_quixote 7y ago
  It sounds wasteful until you've been with a company that outgrew its management structure. I spent many years at a place that grew from 200 to 800 people in about 3 years. We prided ourselves on the flat org and high level of responsibility. But as we approached the 500 mark, the cracks really started to show: - Things truly requiring management handling (like raises, big expenses, hiring, etc.) took FOREVER in the best case because of the bottleneck at the top. More commonly, they got lost/dropped without explanation. - Flow of sensitive information was slow and inconsistent because the only way to communicate with a flat org is all-or-nothing. - Specialist engineers found themselves trying to track too many projects before they realized the overload was a problem (this was me) - The president (and effective sole owner) was still deeply involved in all projects and regularly exercised veto rights. As the project load increased, the vetos landed later and later - some times after we'd already spent 50% of the budget! This left our customers up a creek and our reputation suffered.
- _fs 7y ago
  It appears that Middle managers are downvoting you in force.
  - deminature 7y ago
    Middle managers have nothing to do with the approvals I was talking about. It's entirely eng-driven.
- hluska 7y ago
  The best way I’ve ever heard this explained is to imagine a software business as a machine. Early on, it’s easy to trust - the machine is small, there aren’t many working parts and dependencies are all manageable. Making things even easier, the founders generally have control over hiring and they’re often still at the hiring friends/colleagues stage of their evolution. Then, the machine gets bigger and the dependency graph gets more complicated. Suddenly, if you change one small part of your module, you run a very real chance of ruining something someone in a faraway department relies upon. So, the machine evolves to have things like coding standards and more formalized review processes. Unfortunately, the machine is composed of humans and humans aren’t as simple as just pulling it all apart, greasing it up and putting it back into service. So every single time you add a review step, there’s a chance it genuinely adds to quality (or removes risk). But, there’s also a chance that that change was about ego and organizational power. Then, in five to ten years, they hire MBAs to come in and lay people off. Edit - Good heavens, this sounds bleak.
- majormajor 7y ago
  You start out with a very light process. Then you have outages, or you discover that not every part of the system with appropriate care, etc. Then you put some process into place. That slows everyone down a bit... your business keeps growing, though, so "more slightly less efficient people" can capture more of the market than "fewer more efficient people" can because you have a calculation where something like 1000.4 is still greater than 101.0, so if that additional stuff getting done is worth the cost of more people, you keep doing it! And you can cycle through this a few times. And yes, people will also cycle through "we need to reduce bottlenecks" and such, but at this point, you've been successful, you've got a bunch of money coming in, and pissing off your clients is something to be avoided, since individual efficiency is not the end goal by itself.
- TulliusCicero 7y ago
  > So you need more engineers because you hired more middle managers? Companies often implement more process for good reason. The antithesis of old Facebook's Move Fast And Break Things is to have policies that ensure good security, stability, accessibility, privacy, internationalization, meeting legal requirements, etc. and that usually means processes to ensure that eager engineers and product managers actually meet those policies, instead of rushing to launch. It's easy to scoff at process when you're a small company or startup whose mistakes will go largely unnoticed unless you suddenly go viral. If you're a Big Tech Company, accidentally messing up security or privacy somewhat means getting a front page article on The Verge detailing your crimes. Working at Google now, there's plenty of process involves in developing and launching a thing, but I can't really say any of the pieces are without merit.
coffeemug 7y ago
That's a really charitable explanation and I think it explains some percentage of labor, but not nearly all of it (not even the majority of it, IMO). Yes, there is more going on behind the scenes in every company than you'd ever know about. But I think the majority of labor follows the pareto principle-- there are maybe 100 employees in every company of, say, 1000 people that do nearly all the work, and 900 employees that do very little to nothing at all at best (and produce negative value at worst). Why that is the case in every business, I haven't been able to figure out. I suspect it's because when businesses reach that size, the efficient market effects become greatly overstated in the short term, so these businesses naturally acquire a lot of fat. One piece of evidence for this is that private equity exists. You can buy a business, fire a quarter of the people, then sell it and make a lot of money without hurting the operations at all.
- deminature 7y ago
  Yes, you're probably right to some extent. A lot of efficiency and velocity is exchanged in these large scale environments for risk-aversion in the form of heavy testing, supporting the ability to revert changes and requiring approvals. I'm not personally a fan of it, but I get why it works this way. The financial loss if you break something adds up in an excruciating fashion on a per hour basis.
- throwawaymsft 7y ago
  Check the incentives. For corporate managers, your salary is generally proportional to the number of people you manage. Who cares if your 1000-person org doesn't accomplish anything, you're a big shot now. On a sports team, coaches are rewarded for championships. There is no room for waste, and underperforming people get cut.
  - listenallyall 7y ago
    >> no room for waste Most sports teams rosters are restricted by league rules. However coaching staffs are not, and they have grown tremendously in recent years, as have college athletic department staffs. When someone actually does take a hard look, these staffs seem exceptionally bloated and wasteful. https://www.baltimoresun.com/news/bs-xpm-2006-10-31-0610310106-story.html https://www.baltimoresun.com/news/bs-xpm-2006-10-31-06103101... When Vince Lombardi began coaching the Green Bay Packers in 1959, he had four assistants. This year, Denver Broncos coach Mike Shanahan has 21. http://www.espn.com/espn/page2/story/_/page/easterbrook%2F101207_tuesday_morning_quarterback/sportCat/nfl http://www.espn.com/espn/page2/story/_/page/easterbrook%2F10... Ohio State lists 458 people in its athletic department. There are 192 faculty members in Ohio State's English department, with a support staff of about 50. https://www.newsobserver.com/sports/college/acc/article163255778.html https://www.newsobserver.com/sports/college/acc/article16325... One assistant coach is called the recruiting coordinator, but he’s backed by a Director of Player Personnel, an Assistant Director of Player Personnel, a Coordinator of On-Campus Recruiting and a Recruiting Assistant. There are three video coordinators.
zelly 7y ago
The real answer is that it's a publicly traded "tech" company that has to maintain a charade of growth. They can either give their profits back to shareholders (violating the growth charade) or they can hire more people and hope they figure out how to create actual growth. Corona spoke up and pointed out the Emperor had no clothes.

madamelic 7y ago

>Another aspect is your changes often have to be approved by a lot of people, which is not the case on small projects. So you need more engineers because you hired more middle managers? That's a little absurd in my opinion but I am likely wrong in this assumption. It just seems like the reaction to "We have lots of approvals" should be figuring out how to reduce the amount of approvals needed, not hiring more engineers to increase velocity.

neltnerb 7y ago
I think you need as many approvals as your risk tolerance allows for... they're not pointless. If they are pointless then sure, get rid of them. But I don't know how you'd guess that number without deep knowledge of the business. It is sometimes better to go slower than you could and be more confident that the product is acceptable. Approvals can but don't necessarily help with that.
deminature 7y ago
The approvals usually come from other engineers. It stems from having to modify highly-sensitive code which usually has an owner you need to work with to get an approval.
ianceicys 7y ago
You're not hiring more middle managers because you want too, no large companies have to promote engineers to senior engineering managers to handle the complexities of approvals required by compliance such as SOC2,SOC3,ISO27001,FedRAMP, GDPR, etc. I've worked at companies were every line of code has to be reviewed and approved by a lawyer. Yes a lawyer (usually a former engineer that the company paid to go to law school) is reviewing some dumb little javascript or python script for license and legal compliance.
- sjs7007 7y ago
  > I've worked at companies were every line of code has to be reviewed and approved by a lawyer. Yes a lawyer (usually a former engineer that the company paid to go to law school) is reviewing some dumb little javascript or python script for license and legal compliance. Jesus, what company was that? And how about libraries included... Will this lawyer now also need to go through that?
ratww 7y ago
It's not just approvals from managers, sometimes you need approval from a group of peer engineers just to create new routes in an (internal) API, or to launch a new cron job that will upload an Excel spreadsheet in marketing once a day with data from Google Forms. Also, UX wants to see the final result on their screen before they approve your build, so before deploy you have to push into a test cluster, but there are only 10 of those, and you need to wait until one of them is free. And of course your Jenkins server takes 2h to build and fails sometimes, so pushing small changes takes 2h because you have to babysit the build. The testing cluster above also takes 2h to build. CTO believes in engineers taking care of their code, so SRE is not allowed to handle the builds. And then there's an average of 2h of meetings per day (I'm not including the 40 minute daily scrum because your squad has 15 people). Half of them to discuss the production incident you had because you didn't see the Slack alerts when you were in another meeting. The other half is chapter and guild meetings where you don't talk, but participation is mandatory. On Mondays you have sprint planning, so forget about coding. Engineering fact: A 300 watt amplifier is only twice as loud (+10 dB) as a 30 watt amplifier. The same math works for engineering teams.
- ThrowMeAwayOkay 7y ago
  Right here. Everyone read the above, this answers the question of why so many?
twblalock 7y ago
The approvals often exist for valid reasons. You can't just get rid of them. You can try to streamline the approval process but that only gets you so far.
dan_quixote 7y ago
It sounds wasteful until you've been with a company that outgrew its management structure. I spent many years at a place that grew from 200 to 800 people in about 3 years. We prided ourselves on the flat org and high level of responsibility. But as we approached the 500 mark, the cracks really started to show: - Things truly requiring management handling (like raises, big expenses, hiring, etc.) took FOREVER in the best case because of the bottleneck at the top. More commonly, they got lost/dropped without explanation. - Flow of sensitive information was slow and inconsistent because the only way to communicate with a flat org is all-or-nothing. - Specialist engineers found themselves trying to track too many projects before they realized the overload was a problem (this was me) - The president (and effective sole owner) was still deeply involved in all projects and regularly exercised veto rights. As the project load increased, the vetos landed later and later - some times after we'd already spent 50% of the budget! This left our customers up a creek and our reputation suffered.
_fs 7y ago
It appears that Middle managers are downvoting you in force.
- deminature 7y ago
  Middle managers have nothing to do with the approvals I was talking about. It's entirely eng-driven.
hluska 7y ago
The best way I’ve ever heard this explained is to imagine a software business as a machine. Early on, it’s easy to trust - the machine is small, there aren’t many working parts and dependencies are all manageable. Making things even easier, the founders generally have control over hiring and they’re often still at the hiring friends/colleagues stage of their evolution. Then, the machine gets bigger and the dependency graph gets more complicated. Suddenly, if you change one small part of your module, you run a very real chance of ruining something someone in a faraway department relies upon. So, the machine evolves to have things like coding standards and more formalized review processes. Unfortunately, the machine is composed of humans and humans aren’t as simple as just pulling it all apart, greasing it up and putting it back into service. So every single time you add a review step, there’s a chance it genuinely adds to quality (or removes risk). But, there’s also a chance that that change was about ego and organizational power. Then, in five to ten years, they hire MBAs to come in and lay people off. Edit - Good heavens, this sounds bleak.
majormajor 7y ago
You start out with a very light process. Then you have outages, or you discover that not every part of the system with appropriate care, etc. Then you put some process into place. That slows everyone down a bit... your business keeps growing, though, so "more slightly less efficient people" can capture more of the market than "fewer more efficient people" can because you have a calculation where something like 1000.4 is still greater than 101.0, so if that additional stuff getting done is worth the cost of more people, you keep doing it! And you can cycle through this a few times. And yes, people will also cycle through "we need to reduce bottlenecks" and such, but at this point, you've been successful, you've got a bunch of money coming in, and pissing off your clients is something to be avoided, since individual efficiency is not the end goal by itself.
TulliusCicero 7y ago
> So you need more engineers because you hired more middle managers? Companies often implement more process for good reason. The antithesis of old Facebook's Move Fast And Break Things is to have policies that ensure good security, stability, accessibility, privacy, internationalization, meeting legal requirements, etc. and that usually means processes to ensure that eager engineers and product managers actually meet those policies, instead of rushing to launch. It's easy to scoff at process when you're a small company or startup whose mistakes will go largely unnoticed unless you suddenly go viral. If you're a Big Tech Company, accidentally messing up security or privacy somewhat means getting a front page article on The Verge detailing your crimes. Working at Google now, there's plenty of process involves in developing and launching a thing, but I can't really say any of the pieces are without merit.

coffeemug 7y ago

That's a really charitable explanation and I think it explains some percentage of labor, but not nearly all of it (not even the majority of it, IMO). Yes, there is more going on behind the scenes in every company than you'd ever know about. But I think the majority of labor follows the pareto principle-- there are maybe 100 employees in every company of, say, 1000 people that do nearly all the work, and 900 employees that do very little to nothing at all at best (and produce negative value at worst). Why that is the case in every business, I haven't been able to figure out. I suspect it's because when businesses reach that size, the efficient market effects become greatly overstated in the short term, so these businesses naturally acquire a lot of fat. One piece of evidence for this is that private equity exists. You can buy a business, fire a quarter of the people, then sell it and make a lot of money without hurting the operations at all.

deminature 7y ago
Yes, you're probably right to some extent. A lot of efficiency and velocity is exchanged in these large scale environments for risk-aversion in the form of heavy testing, supporting the ability to revert changes and requiring approvals. I'm not personally a fan of it, but I get why it works this way. The financial loss if you break something adds up in an excruciating fashion on a per hour basis.
throwawaymsft 7y ago
Check the incentives. For corporate managers, your salary is generally proportional to the number of people you manage. Who cares if your 1000-person org doesn't accomplish anything, you're a big shot now. On a sports team, coaches are rewarded for championships. There is no room for waste, and underperforming people get cut.
- listenallyall 7y ago
  >> no room for waste Most sports teams rosters are restricted by league rules. However coaching staffs are not, and they have grown tremendously in recent years, as have college athletic department staffs. When someone actually does take a hard look, these staffs seem exceptionally bloated and wasteful. https://www.baltimoresun.com/news/bs-xpm-2006-10-31-0610310106-story.html https://www.baltimoresun.com/news/bs-xpm-2006-10-31-06103101... When Vince Lombardi began coaching the Green Bay Packers in 1959, he had four assistants. This year, Denver Broncos coach Mike Shanahan has 21. http://www.espn.com/espn/page2/story/_/page/easterbrook%2F101207_tuesday_morning_quarterback/sportCat/nfl http://www.espn.com/espn/page2/story/_/page/easterbrook%2F10... Ohio State lists 458 people in its athletic department. There are 192 faculty members in Ohio State's English department, with a support staff of about 50. https://www.newsobserver.com/sports/college/acc/article163255778.html https://www.newsobserver.com/sports/college/acc/article16325... One assistant coach is called the recruiting coordinator, but he’s backed by a Director of Player Personnel, an Assistant Director of Player Personnel, a Coordinator of On-Campus Recruiting and a Recruiting Assistant. There are three video coordinators.

zelly 7y ago

The real answer is that it's a publicly traded "tech" company that has to maintain a charade of growth. They can either give their profits back to shareholders (violating the growth charade) or they can hire more people and hope they figure out how to create actual growth. Corona spoke up and pointed out the Emperor had no clothes.