6 ms·

Mistakes Google made in scaling its organization

73 points by plinkplonk 16y ago

Hominem 16y ago
Honestly that sounds typical of a large organization, everyone wants to put their fingerprint on a project when it's way up. Once everyone gets their hands on it, every vp packs the team with their buddies, burdens you with no-show contractors from a firm they have a "relationship" with. Then everyone starts scuttling away to the next big thing. Of course someone gets bored one day, and decides to redefine the entire scope of the project, pushes it as the next big thing, then everyone come scuttling back.
- cageface 16y ago
  Exactly. It's a scramble for symbolic ownership that obscures the fact that everybody is increasingly reluctant to take real ownership. Accountability doesn't scale well.
ccarpenterg 16y ago
That is not a mistake. It's a corporation.
- staunch 16y ago
  That is not a mistake. It's a large corporation. But I repeat myself.
SoftwareMaven 16y ago
I've never seen success when an organization organizes around business function. Invariably, the function takes precedence over the product.
- CaptainZapp 16y ago
  I'd enter Oracle as a counter example. Personally, however, this is not something I would want to achieve, but to each his own I guess.
  - jbooth 16y ago
    Maybe a corollary for businesses whose product is based on lock-in. Poor software design is sort of the point as long as you keep getting your hooks into people.
birken 16y ago
I think calling Google processes "mistakes" is a little strong. For example, I recall at one point at Google where they were trying to improve latency on the products, so you weren't allowed to launch a new feature if it hurt latency. I could imagine somebody who just made an awesome feature that made an application 5% slower complain about how stupid that policy is. However, they don't realize that however awesome their feature is, speed of applications is a hidden cost that everybody bears, and sometimes the priority for a company may differ with a specific developers vision. Google wants their products to be fast, wants them to be available, and wants them to be secure. I'm not saying that everything they do is great and their processes and checklists are optimal, but when they have processes designed to do a certain thing, it isn't fair to say their processes are a mistake because they don't do something else.
- lemmsjid 16y ago
  Very much agreed. When an organization becomes large it needs to make tough choices. In an organization built around product-centric teams, it may be easier to release products more quickly and the average employee may perceive a smoother course from concept to execution, but the organization will quickly discover that horizontal changes become extremely difficult. The frustrated product developer who can't get products out is replaced by the frustrated operations guy who can't patch unsecure systems fast enough. Which one is worse? For Google, I would say the latter. While it always seems like a tragedy to hear about "great" products getting killed, pre-birth, in the corporate mill, I much prefer my stable, high uptime, browser-performant Google to a Google that releases way more product than it already does. Not to mention that most products (like most startups) fail. As a product-centric company, do you break teams apart after a product fails? If a product succeeds, do you take that successful team and move it to the other product? That may work if you're lucky enough to have a team with the proper skills balance (not to mention passion) to move across different products. What about career goals? People in a particular field tend to want to work with the best people in that field--a huge draw of working at Google is working with the best people in Field X. In a functional organization, the functions (almost) never go away--so the teams are stable. Of course, being on the outside I can't speak for the problems that are mentioned, such as in-fighting, lack of executive discipline ("You will not launch your product because I don't like your hair style, FUUUUUUUUU!"), etc., but those problems are bad regardless of organization type.
Joakal 16y ago
What can Google do to fix those underlying issues?
- danbmil99 16y ago
  Allow autonomy to product groups. They should be run like little startups: choose whatever tech stack they want, hire their own PM's, do their own hiring (Google recruiting is a complete wasteland), and live or die by their scucess/failure. It'll never happen, because it's diametrically opposed to the founding ideology, which is that every problem has a rational solution. Without getting too Ayn Rand on this, it's the classic mistake of central planning vs personal initiative.
  - Hominem 16y ago
    Are you saying they should be able to go off into the wilderness, decide to run on Heroku or something and come back with an app? Companies invest huge sums of money on infrastructure and the expertise to manage it. Google has too big an infrastructure to let each project roll it's own. Let's say google reengineered to provide something like ec2,each team can run it's own images. does each group hire security experts? Does each group have to solve scaling problems google already faced? I think maybe it could work like this.Google proper hires domain experts, security, engineering etc. Provides each mini startup with a budget, mini startups can spend the cash on the Google proper experts, or go outside for areas where Google is lacking. Mini startup pays hosting fees to run on Google's EC2 like infrastructure, or use whatever Google has in place now.
    - waterlesscloud 16y ago
      What if you let teams select from internal security teams, internal scaling teams, etc? You can run a wide range of things on EC2, but I'd be willing to bet certain sets of things dominate. If the solution google has internally really is the best solution available, let it compete. It'll win until it's no longer the best.
      - 16y ago
ojbyrne 16y ago
So this basically reads as someone who's taken a couple of Org Design courses in college arguing that it's actually significant whether the top level of an organization is "product" or "function" ("geography" is another alternative), and as a result he basically comes across as someone who could use a little "experience" so that he understands that scaling an organization isn't just about some analytical framework that he learned in school, but about people, dynamics, history, politics, egos, a lot of stuff that is really "hard." And that almost every fast growing company gets it, in some form or another.. "wrong." But they also find success despite their failure. Whereas the guy who took the couple of Org Design courses probably ends up working for a VC running innumerable other companies straight into the ground.
- babar 16y ago
  I would encourage you to talk to some people who you trust who have worked on a few different projects at google (especially anything not related to search or ads) and see if they think this answer is theoretical. If you don't have people there you can ask, take a close look at what has happened to the products and teams that google has acquired.
  - notauser 16y ago
    The problem is that people who say it's a "functional organization" problem are both correct and incorrect at the same time. The kind of problems described certainly seem to have been exacerbated by the way they are structured. But all organization structures have their own problems, and the issues caused by (say) a project based organization* might have been even more detrimental to success. The most professionally managed organizations I have worked for have generally been matrix orgs (sort of functional, with projects 'borrowing' resource for a year or two at a time) and that worked pretty well, but only because they burned a lot of time and money managing the process. In many cases it may be cheaper just to live with doing it imperfectly. *Lack of product integration, break up of technical pool of talent, even bigger resource fights, duplication of effort, lack of communication, and zombie projects amongst other things.
shadowmatter 16y ago
As an ex-Googler, I disagree with this second-hand account. Yeah, political infighting sometimes happens and things get shook up, but this person paints the picture of no project ever launching. Addressing some points: 1. See item 3. 2. VP changes rarely happen compared to a PM change. And a PM who comes may change some of the goals, but the focus area of your product doesn't change, e.g. your optimization tool for display advertising will continue to be an optimization tool for display advertising. It would be hard to find your project suddenly disagreeing with some "existing strategy." 3. New products that launch should have a UX designer contributing or at least advising. There should be mocks for the frontend engineers to follow, and the backend engineers are just told to make it work. 4. Yeah, there's a checklist, and it is long. SREs want to minimize the number of pages there, but pages happen because something's going bad, and they want you to have proper monitoring in place to address the pages quickly. The launch checklist also focuses a lot on addressing potential security vulnerabilities. Google is tough on security. 5. Google's all about the software compensating for commodity hardware. I don't think too many teams have specialized hardware. You can ask for what resources you need, but would have to fight for what form you get them in. 7. Can't speak for this. Why reorganize marketing? 8. While I agree that Google likes to fail fast, in my opinion some Google products probably didn't get the love or time they needed to take off. I wasn't really privy to how those decisions are made.
Steer 16y ago
Not sure if this is off-topic, but have any company ever grown as fast to be as big as Google have? Successfully?
yason 16y ago
In the light of this article, Google should have a play area with only limited network and processing power for publishing betas of new products. For example, anything that's Google Search or GMail or similar must meet some performance, stability, and realiability criteria but those applications would also get to run on the big clusters. Conversely, it wouldn't matter if a beta-toy is a bit buggy or hogs down the servers on the playground since it's just playground. Some users would bear with the unfinished tone and start using the apps there, and at times it would happen that one application would gain lots of momentum, and Google could consider fixing and lifting that one to the full service level.
adambyrtek 16y ago
It's always good to have an open discussion, but I don't find it convincing when an anonymous guy on Quora based on second-hand unverified information claims that he knows better what is good of bad for Google.

73 points by plinkplonk 16y ago

Honestly that sounds typical of a large organization, everyone wants to put their fingerprint on a project when it's way up. Once everyone gets their hands on it, every vp packs the team with their buddies, burdens you with no-show contractors from a firm they have a "relationship" with. Then everyone starts scuttling away to the next big thing. Of course someone gets bored one day, and decides to redefine the entire scope of the project, pushes it as the next big thing, then everyone come scuttling back.

cageface 16y ago
Exactly. It's a scramble for symbolic ownership that obscures the fact that everybody is increasingly reluctant to take real ownership. Accountability doesn't scale well.

ccarpenterg 16y ago

That is not a mistake. It's a corporation.

staunch 16y ago
That is not a mistake. It's a large corporation. But I repeat myself.

SoftwareMaven 16y ago

I've never seen success when an organization organizes around business function. Invariably, the function takes precedence over the product.

CaptainZapp 16y ago
I'd enter Oracle as a counter example. Personally, however, this is not something I would want to achieve, but to each his own I guess.
- jbooth 16y ago
  Maybe a corollary for businesses whose product is based on lock-in. Poor software design is sort of the point as long as you keep getting your hooks into people.

birken 16y ago

I think calling Google processes "mistakes" is a little strong. For example, I recall at one point at Google where they were trying to improve latency on the products, so you weren't allowed to launch a new feature if it hurt latency. I could imagine somebody who just made an awesome feature that made an application 5% slower complain about how stupid that policy is. However, they don't realize that however awesome their feature is, speed of applications is a hidden cost that everybody bears, and sometimes the priority for a company may differ with a specific developers vision. Google wants their products to be fast, wants them to be available, and wants them to be secure. I'm not saying that everything they do is great and their processes and checklists are optimal, but when they have processes designed to do a certain thing, it isn't fair to say their processes are a mistake because they don't do something else.

lemmsjid 16y ago
Very much agreed. When an organization becomes large it needs to make tough choices. In an organization built around product-centric teams, it may be easier to release products more quickly and the average employee may perceive a smoother course from concept to execution, but the organization will quickly discover that horizontal changes become extremely difficult. The frustrated product developer who can't get products out is replaced by the frustrated operations guy who can't patch unsecure systems fast enough. Which one is worse? For Google, I would say the latter. While it always seems like a tragedy to hear about "great" products getting killed, pre-birth, in the corporate mill, I much prefer my stable, high uptime, browser-performant Google to a Google that releases way more product than it already does. Not to mention that most products (like most startups) fail. As a product-centric company, do you break teams apart after a product fails? If a product succeeds, do you take that successful team and move it to the other product? That may work if you're lucky enough to have a team with the proper skills balance (not to mention passion) to move across different products. What about career goals? People in a particular field tend to want to work with the best people in that field--a huge draw of working at Google is working with the best people in Field X. In a functional organization, the functions (almost) never go away--so the teams are stable. Of course, being on the outside I can't speak for the problems that are mentioned, such as in-fighting, lack of executive discipline ("You will not launch your product because I don't like your hair style, FUUUUUUUUU!"), etc., but those problems are bad regardless of organization type.

Joakal 16y ago

What can Google do to fix those underlying issues?

danbmil99 16y ago
Allow autonomy to product groups. They should be run like little startups: choose whatever tech stack they want, hire their own PM's, do their own hiring (Google recruiting is a complete wasteland), and live or die by their scucess/failure. It'll never happen, because it's diametrically opposed to the founding ideology, which is that every problem has a rational solution. Without getting too Ayn Rand on this, it's the classic mistake of central planning vs personal initiative.
- Hominem 16y ago
  Are you saying they should be able to go off into the wilderness, decide to run on Heroku or something and come back with an app? Companies invest huge sums of money on infrastructure and the expertise to manage it. Google has too big an infrastructure to let each project roll it's own. Let's say google reengineered to provide something like ec2,each team can run it's own images. does each group hire security experts? Does each group have to solve scaling problems google already faced? I think maybe it could work like this.Google proper hires domain experts, security, engineering etc. Provides each mini startup with a budget, mini startups can spend the cash on the Google proper experts, or go outside for areas where Google is lacking. Mini startup pays hosting fees to run on Google's EC2 like infrastructure, or use whatever Google has in place now.
  - waterlesscloud 16y ago
    What if you let teams select from internal security teams, internal scaling teams, etc? You can run a wide range of things on EC2, but I'd be willing to bet certain sets of things dominate. If the solution google has internally really is the best solution available, let it compete. It'll win until it's no longer the best.
    - 16y ago

ojbyrne 16y ago

So this basically reads as someone who's taken a couple of Org Design courses in college arguing that it's actually significant whether the top level of an organization is "product" or "function" ("geography" is another alternative), and as a result he basically comes across as someone who could use a little "experience" so that he understands that scaling an organization isn't just about some analytical framework that he learned in school, but about people, dynamics, history, politics, egos, a lot of stuff that is really "hard." And that almost every fast growing company gets it, in some form or another.. "wrong." But they also find success despite their failure. Whereas the guy who took the couple of Org Design courses probably ends up working for a VC running innumerable other companies straight into the ground.

babar 16y ago
I would encourage you to talk to some people who you trust who have worked on a few different projects at google (especially anything not related to search or ads) and see if they think this answer is theoretical. If you don't have people there you can ask, take a close look at what has happened to the products and teams that google has acquired.
- notauser 16y ago
  The problem is that people who say it's a "functional organization" problem are both correct and incorrect at the same time. The kind of problems described certainly seem to have been exacerbated by the way they are structured. But all organization structures have their own problems, and the issues caused by (say) a project based organization* might have been even more detrimental to success. The most professionally managed organizations I have worked for have generally been matrix orgs (sort of functional, with projects 'borrowing' resource for a year or two at a time) and that worked pretty well, but only because they burned a lot of time and money managing the process. In many cases it may be cheaper just to live with doing it imperfectly. *Lack of product integration, break up of technical pool of talent, even bigger resource fights, duplication of effort, lack of communication, and zombie projects amongst other things.

shadowmatter 16y ago

As an ex-Googler, I disagree with this second-hand account. Yeah, political infighting sometimes happens and things get shook up, but this person paints the picture of no project ever launching. Addressing some points: 1. See item 3. 2. VP changes rarely happen compared to a PM change. And a PM who comes may change some of the goals, but the focus area of your product doesn't change, e.g. your optimization tool for display advertising will continue to be an optimization tool for display advertising. It would be hard to find your project suddenly disagreeing with some "existing strategy." 3. New products that launch should have a UX designer contributing or at least advising. There should be mocks for the frontend engineers to follow, and the backend engineers are just told to make it work. 4. Yeah, there's a checklist, and it is long. SREs want to minimize the number of pages there, but pages happen because something's going bad, and they want you to have proper monitoring in place to address the pages quickly. The launch checklist also focuses a lot on addressing potential security vulnerabilities. Google is tough on security. 5. Google's all about the software compensating for commodity hardware. I don't think too many teams have specialized hardware. You can ask for what resources you need, but would have to fight for what form you get them in. 7. Can't speak for this. Why reorganize marketing? 8. While I agree that Google likes to fail fast, in my opinion some Google products probably didn't get the love or time they needed to take off. I wasn't really privy to how those decisions are made.

Steer 16y ago

Not sure if this is off-topic, but have any company ever grown as fast to be as big as Google have? Successfully?

yason 16y ago

In the light of this article, Google should have a play area with only limited network and processing power for publishing betas of new products. For example, anything that's Google Search or GMail or similar must meet some performance, stability, and realiability criteria but those applications would also get to run on the big clusters. Conversely, it wouldn't matter if a beta-toy is a bit buggy or hogs down the servers on the playground since it's just playground. Some users would bear with the unfinished tone and start using the apps there, and at times it would happen that one application would gain lots of momentum, and Google could consider fixing and lifting that one to the full service level.

adambyrtek 16y ago

It's always good to have an open discussion, but I don't find it convincing when an anonymous guy on Quora based on second-hand unverified information claims that he knows better what is good of bad for Google.