5 ms·

What is the solution? I've worked at big companies that are mired in process because they would rather spend more time crossing i's and dotting t's than risk b

What is the solution? I've worked at big companies that are mired in process because they would rather spend more time crossing i's and dotting t's than risk breaking something. I can see why. And I've worked at smaller companies where the clients are small and it's easy to fix things that break. Move fast and break things at a small scale maybe. But how do you grow to be a big company and still operate like a small company? I can't seem to see an answer.

ignoramous 5y ago
Autonomy. Solution to such org woes, in part, is discussed by Clayton Christensen in his work, The Innovator's Solution http://web.mit.edu/6.933/www/Fall2000/teradyne/clay.html http://web.mit.edu/6.933/www/Fall2000/teradyne/clay.html: Even after correctly identifying potentially disruptive technologies, firms still must circumvent its hierarchy and bureaucracy that can stifle the free pursuit of creative ideas. Christensen suggests that firms need to provide experimental groups within the company a freer rein. "With a few exceptions, the only instances in which mainstream firms have successfully established a timely position in a disruptive technology were those in which the firms' managers set up an autonomous organization charged with building a new and independent business around the disruptive technology." This autonomous organization will then be able to choose the customers it answers to, choose how much profit it needs to make, and how to run its business. --- Amazon and Cloudflare are good examples of big-orgs trying their best to implement late Prof. Christensen's ideas. Andy Jassy on Amazon's approach to innovation: https://www.hbs.edu/forum-for-growth-and-innovation/podcasts/disruptive-voice/Pages/podcast-details.aspx?episode=15834284 https://www.hbs.edu/forum-for-growth-and-innovation/podcasts...: And then if we like the answers to those first four elements, then we ask, can we put a group of single threaded focused people on this initiative, even if it seems like they're overwhelming it with strong senior people, if you try to add really busy people do the existing business and the big new idea, they will always favor the existing business because it's surer bet. So we want to peel people away from the existing business and put them just on the new initiative. Pace of innovation at Cloudflare https://blog.cloudflare.com/the-secret-to-cloudflare-pace-of-innovation/ https://blog.cloudflare.com/the-secret-to-cloudflare-pace-of...: ...it is not unusual for an initial product idea to start with a team small enough to split a pack of Twinkies and for the initial proof of concept to go from whiteboard to rolled out in days. We intentionally staff and structure our teams and our backlogs so that we have flexibility to pivot and innovate. Our Emerging Technology and Incubation team is a small group of product managers and engineers solely dedicated to exploring new products for new markets. Our Research team is dedicated to thinking deeply and partnering with organizations across the globe to define new standards and new ways to tackle some of the hardest challenges. --- Also read: Clayton Christensen and Stephen Kaufman on "Resources, Process, and Priorities": https://personal.utdallas.edu/~chasteen/Christensen%20-%202nd%20article.htm https://personal.utdallas.edu/~chasteen/Christensen%20-%202n...
Ao7bei3s 5y ago
Self-service approvals. Instead of appointing a senior eng to be approver, task the same senior eng with writing down his decision criteria (as text or where it makes sense even as code). This has advantages for everyone: 1. It lets the engineers who need approval move at their own speed, and plan time for it as a predictable work item like any other, instead of depending on an approver for whom the approvals will usually be at a lower priority and mid-sprint. 2. For the approval policy writer, it turns this into a one time effort with a defined scope that can be planned and prioritized in his/her own backlog, instead of open ended toil that can come at any time, take any time, and not clearly relate to their own current priorities. 3. For the company, writing down the policy brings consistent decision making. Obviously this requires trust that employees can and will say "no, can't do" when they're tasked with something that is not approvable, which can be culturally difficult (business and otherwise). Checklists (literally a list of checkboxes to click on, "I confirm that...") can help with this. (As an example of writing down the policy as code: that's any CI/CD pipeline. But it's not limited to engineering decision making - for example, we're using a well-known open source license management tool that promises auto-approval for open source library use depending on policies configured by legal. This works moderately not so well because this particular tool is not great; the idea is sound. We still made it work: now legal wrote down their policies, trained a large number of engineers on them and those are now empowered to make decisions.)
native_samples 5y ago
There are many, but the problems are more subtle than this video really gives credit for. I worked at Google at the time this video was made, and empathized (in fact I had been an SRE for years by that point). Nonetheless, there are flip sides that the video maker obviously didn't consider. Firstly, why did everything at Google have to be replicated up the wazoo? Why was so much time spent talking about PCRs? The reason is, Google had consciously established a culture up front in which individual clusters were considered "unreliable" and everyone had to engineer around that. This was a move specifically intended to increase the velocity of the datacenter engineering groups, by ensuring they did not have to get a billion approvals to do changes. Consider how slow it'd be to get approval from every user of a Google cluster, let alone an entire datacenter, to take things offline for maintenance. These things had tens of thousands of machines per cluster and that was over a decade ago. They'd easily be running hundreds of thousands of processes, managed by dozens of different groups. Getting them all to synchronize and approve things would be impossible. So Google said - no approvals are necessary. If the SRE/NetOps/HWOPS teams want to take a cluster or even entire datacenter offline then they simply announce they're going to do it in advance, and, everyone else has to just handle it. This was fantastic for Google's datacenter tech velocity. They had incredibly advanced facilities years ahead of anyone else, partly due to the frenetic pace of upgrades this system allowed them to achieve. The downside: software engineers have to run their services in >1 cluster, unless they're willing to tolerate downtime. Secondly, why couldn't cat woman just run a single replica and accept some downtime? Mostly because Google had a brand to maintain. When she "just" wanted to serve 5TB, that wasn't really true. She "just" wanted to do it under the Google brand, advertised as a Google service, with all the benefits that brought her. One of the aspects of that brand that we take for granted is Google's insane levels of reliability. Nobody, and I mean nobody, spends serious time planning for "what if Google is down", even though massive companies routinely outsource all their corporate email and other critical infrastructure to them. Now imagine how hard it'd be to maintain that brand if random services kept going offline for long periods without Google employees even noticing? They could say, sure, this particular service just wasn't important enough for us to replicate or monitor and the DC is under maintenance, we think it'll be back in 3 days, sorry. But customers and users would freak out, and rightly so. How on earth could they guess what Google would or would not find worthy of proper production quality? That would be opaque to them, yet Google has thousands of services. It'd destroy the brand to have some parts that are reliable and others not according to basically random factors nobody outside the firm can understand. The only solution is to ensure every externally visible service is reliable to the same very high degree. Indeed, "don't trust that service because Google might kill it" is one of the worst problems the brand has, and that's partly due to efforts to avoid corporate slowdown and launch bureaucracy. Google signed off on a lot of dud uncompetitive services that had serious problems, specifically because they hated the idea of becoming a slow moving behemoth that couldn't innovate. Yet it trashed their brand in the end. A lot of corporate process engineering is like this. It often boils down to tradeoffs consciously made by executives that the individual employee may not care about or value or even know about, but which is good for the group as a whole. Was Google wrong to take an unreliable-DC-but-reliable-services approach? I don't know but I really doubt it. Most of the stuff that SWEs were super impatient to launch and got bitchy about bureaucracy wasn't actually world changing stuff, and a lot ended up not achieving any kind of escape velocity.
deleted 5y ago
[deleted]
plaguuuuuu 5y ago
Trust your people! Otherwise don't hire them in the first place. You want to deploy something to prod? Okay, either call an API or fill out a webform (or message a Slack bot idk) - but the contents will be a checklist of stuff you need to have done. 1. Did you [load test, integration test, whatever] 2. Did your local architect look over and approve the high-level design? (NB: notice how we aren't requiring an architect to sign off, we trust the developer. Because if they lie they're fired lol.) 3. Other stuff. Maybe some taxonomic stuff like tags associated with deployed infrastructure? Swagger endpoints? Go nuts as long as it's stuff actually needed by central planning - the documentation and paper trail aspect is covered here This is picked up to be ingested into databases, wikis, emails, wherever. Compare with my last large corp where we had change approval boards where 25+ people sat in a long meeting and essentially just asked if you'd done the above and you'd then be greenlit to go to prod (at the time deployments were pretty manual and required scheduling as well which is obviously suboptimal). I'm just about to move from a small consulting company to a startup/scaleup so it's going to be interesting to see how things move there..

ignoramous 5y ago

Autonomy. Solution to such org woes, in part, is discussed by Clayton Christensen in his work, The Innovator's Solution http://web.mit.edu/6.933/www/Fall2000/teradyne/clay.html http://web.mit.edu/6.933/www/Fall2000/teradyne/clay.html: Even after correctly identifying potentially disruptive technologies, firms still must circumvent its hierarchy and bureaucracy that can stifle the free pursuit of creative ideas. Christensen suggests that firms need to provide experimental groups within the company a freer rein. "With a few exceptions, the only instances in which mainstream firms have successfully established a timely position in a disruptive technology were those in which the firms' managers set up an autonomous organization charged with building a new and independent business around the disruptive technology." This autonomous organization will then be able to choose the customers it answers to, choose how much profit it needs to make, and how to run its business. --- Amazon and Cloudflare are good examples of big-orgs trying their best to implement late Prof. Christensen's ideas. Andy Jassy on Amazon's approach to innovation: https://www.hbs.edu/forum-for-growth-and-innovation/podcasts/disruptive-voice/Pages/podcast-details.aspx?episode=15834284 https://www.hbs.edu/forum-for-growth-and-innovation/podcasts...: And then if we like the answers to those first four elements, then we ask, can we put a group of single threaded focused people on this initiative, even if it seems like they're overwhelming it with strong senior people, if you try to add really busy people do the existing business and the big new idea, they will always favor the existing business because it's surer bet. So we want to peel people away from the existing business and put them just on the new initiative. Pace of innovation at Cloudflare https://blog.cloudflare.com/the-secret-to-cloudflare-pace-of-innovation/ https://blog.cloudflare.com/the-secret-to-cloudflare-pace-of...: ...it is not unusual for an initial product idea to start with a team small enough to split a pack of Twinkies and for the initial proof of concept to go from whiteboard to rolled out in days. We intentionally staff and structure our teams and our backlogs so that we have flexibility to pivot and innovate. Our Emerging Technology and Incubation team is a small group of product managers and engineers solely dedicated to exploring new products for new markets. Our Research team is dedicated to thinking deeply and partnering with organizations across the globe to define new standards and new ways to tackle some of the hardest challenges. --- Also read: Clayton Christensen and Stephen Kaufman on "Resources, Process, and Priorities": https://personal.utdallas.edu/~chasteen/Christensen%20-%202nd%20article.htm https://personal.utdallas.edu/~chasteen/Christensen%20-%202n...

Ao7bei3s 5y ago

Self-service approvals. Instead of appointing a senior eng to be approver, task the same senior eng with writing down his decision criteria (as text or where it makes sense even as code). This has advantages for everyone: 1. It lets the engineers who need approval move at their own speed, and plan time for it as a predictable work item like any other, instead of depending on an approver for whom the approvals will usually be at a lower priority and mid-sprint. 2. For the approval policy writer, it turns this into a one time effort with a defined scope that can be planned and prioritized in his/her own backlog, instead of open ended toil that can come at any time, take any time, and not clearly relate to their own current priorities. 3. For the company, writing down the policy brings consistent decision making. Obviously this requires trust that employees can and will say "no, can't do" when they're tasked with something that is not approvable, which can be culturally difficult (business and otherwise). Checklists (literally a list of checkboxes to click on, "I confirm that...") can help with this. (As an example of writing down the policy as code: that's any CI/CD pipeline. But it's not limited to engineering decision making - for example, we're using a well-known open source license management tool that promises auto-approval for open source library use depending on policies configured by legal. This works moderately not so well because this particular tool is not great; the idea is sound. We still made it work: now legal wrote down their policies, trained a large number of engineers on them and those are now empowered to make decisions.)

native_samples 5y ago

There are many, but the problems are more subtle than this video really gives credit for. I worked at Google at the time this video was made, and empathized (in fact I had been an SRE for years by that point). Nonetheless, there are flip sides that the video maker obviously didn't consider. Firstly, why did everything at Google have to be replicated up the wazoo? Why was so much time spent talking about PCRs? The reason is, Google had consciously established a culture up front in which individual clusters were considered "unreliable" and everyone had to engineer around that. This was a move specifically intended to increase the velocity of the datacenter engineering groups, by ensuring they did not have to get a billion approvals to do changes. Consider how slow it'd be to get approval from every user of a Google cluster, let alone an entire datacenter, to take things offline for maintenance. These things had tens of thousands of machines per cluster and that was over a decade ago. They'd easily be running hundreds of thousands of processes, managed by dozens of different groups. Getting them all to synchronize and approve things would be impossible. So Google said - no approvals are necessary. If the SRE/NetOps/HWOPS teams want to take a cluster or even entire datacenter offline then they simply announce they're going to do it in advance, and, everyone else has to just handle it. This was fantastic for Google's datacenter tech velocity. They had incredibly advanced facilities years ahead of anyone else, partly due to the frenetic pace of upgrades this system allowed them to achieve. The downside: software engineers have to run their services in >1 cluster, unless they're willing to tolerate downtime. Secondly, why couldn't cat woman just run a single replica and accept some downtime? Mostly because Google had a brand to maintain. When she "just" wanted to serve 5TB, that wasn't really true. She "just" wanted to do it under the Google brand, advertised as a Google service, with all the benefits that brought her. One of the aspects of that brand that we take for granted is Google's insane levels of reliability. Nobody, and I mean nobody, spends serious time planning for "what if Google is down", even though massive companies routinely outsource all their corporate email and other critical infrastructure to them. Now imagine how hard it'd be to maintain that brand if random services kept going offline for long periods without Google employees even noticing? They could say, sure, this particular service just wasn't important enough for us to replicate or monitor and the DC is under maintenance, we think it'll be back in 3 days, sorry. But customers and users would freak out, and rightly so. How on earth could they guess what Google would or would not find worthy of proper production quality? That would be opaque to them, yet Google has thousands of services. It'd destroy the brand to have some parts that are reliable and others not according to basically random factors nobody outside the firm can understand. The only solution is to ensure every externally visible service is reliable to the same very high degree. Indeed, "don't trust that service because Google might kill it" is one of the worst problems the brand has, and that's partly due to efforts to avoid corporate slowdown and launch bureaucracy. Google signed off on a lot of dud uncompetitive services that had serious problems, specifically because they hated the idea of becoming a slow moving behemoth that couldn't innovate. Yet it trashed their brand in the end. A lot of corporate process engineering is like this. It often boils down to tradeoffs consciously made by executives that the individual employee may not care about or value or even know about, but which is good for the group as a whole. Was Google wrong to take an unreliable-DC-but-reliable-services approach? I don't know but I really doubt it. Most of the stuff that SWEs were super impatient to launch and got bitchy about bureaucracy wasn't actually world changing stuff, and a lot ended up not achieving any kind of escape velocity.

deleted 5y ago

[deleted]

plaguuuuuu 5y ago

Trust your people! Otherwise don't hire them in the first place. You want to deploy something to prod? Okay, either call an API or fill out a webform (or message a Slack bot idk) - but the contents will be a checklist of stuff you need to have done. 1. Did you [load test, integration test, whatever] 2. Did your local architect look over and approve the high-level design? (NB: notice how we aren't requiring an architect to sign off, we trust the developer. Because if they lie they're fired lol.) 3. Other stuff. Maybe some taxonomic stuff like tags associated with deployed infrastructure? Swagger endpoints? Go nuts as long as it's stuff actually needed by central planning - the documentation and paper trail aspect is covered here This is picked up to be ingested into databases, wikis, emails, wherever. Compare with my last large corp where we had change approval boards where 25+ people sat in a long meeting and essentially just asked if you'd done the above and you'd then be greenlit to go to prod (at the time deployments were pretty manual and required scheduling as well which is obviously suboptimal). I'm just about to move from a small consulting company to a startup/scaleup so it's going to be interesting to see how things move there..