3 ms·

> Who exactly does this. And by what mechanism do you enforce: That's one of the key pieces of what makes SRE "SRE" as opposed to an ops team. They are given t

> Who exactly does this. And by what mechanism do you enforce: That's one of the key pieces of what makes SRE "SRE" as opposed to an ops team. They are given the power to stop deployment. If they can't, they're not SREs. > Are you literally proposing some reality where VP Alice nonconsensually tells peer VP Bob "you're missing your roadmap ship dates. Im taking away your orgs prod creds because your lines arent up and to the right enough?" Yes, and no. No because "roadmap ship dates" aren't reliability. I'm describing a reality where VP Alice nonconsentually tells VP Bob "you've had three outages in the past three weeks, we're out of SLO and so you no longer get to deploy new versions until we have remediations in place to prevent future outages". (and of course they both agreed to this) Of course, Bob, being a rational human probably starts taking actions after the first outage, and so it never gets to this point, but yes, that Alice can tell Bob "no more deployments" is literally the point.

aluminussoma 4y ago
> They are given the power to stop deployment. If they can't, they're not SREs Google is the only place where this is even discussed. I cannot name any other company that does this. Not Apple, Meta's Production Engineers, Netflix, Amazon's SysDevEngs, Stripe SRE, Uber, Pinterest, and more. This must mean that nobody else does real SRE!
- xorcist 4y ago
  Given the availability of Facebook, Netflix and the rest, one might think that Google is on to something there.
donavanm 4y ago
> They are given the power to stop deployment. Sweet. Given by whom > Alice can tell Bob "no more deployments" is literally the point. Fantastic. Why can she say & enforce this. You seem to be going a long ways to describe “what” while explicitly ignoring “why” and “how.” To be clear my thesis is that success in both operating models is built on a combination of top down incentive/buy in and broad consensus expressed as a cultural norm. Im not being intentionally obtuse, but Im yet to see any other hypothesis or method of action whicb would explain the differentiation of ability to execute.
- joshuamorton 4y ago
  > Sweet. Given by whom Directly, the answer is the permissions system. In my neck of the woods, devs are quite literally incapable of pushing things to production on their own. The automated tooling handles it 99% of the time, but in the case that they turn it off, only the SREs have production access necessary to actually push new things. But I think you're speaking organizationally, and the answer there is that that agreement was made by SVPs and is enforced by the PRR process. When you adopt SRE support, you hand over the keys, so to speak. > Fantastic. Why can she say & enforce this. Because there's nothing Bob can do except escalate, and SRE leadership will support Alice all the way up to the SVP level. > To be clear my thesis is that success in both operating models is built on a combination of top down incentive/buy in and broad consensus expressed as a cultural norm. My point is that the process creates the cultural norm, and enforces everywhere. That is, your org might have these good processes, and that's laudable. But if Andy turned around tomorrow and said "hey, this year your first priority needs to be shipping new features, because we're falling behind", would your org maintain that commitment? Would every exec? The same question goes for every exec speaking to their subordinates. Its much more difficult to turn to the guy whose job is reliability and reports directly to the CEO and say "hey, your focus this year is shipping features". Like maybe you could say "your focus is shipping features reliably this year", and that should take the focus off of some other things that the SRE org is involved in, but I'm not convinced. Or briefly, your approach is betting on there being a cohesive company culture of reliability that permeates every team, org, and manager. We both know that's not true, at Amazon or Google or any other company. They're all big enough to have different cultures in different parts. A distinct org and reporting structure, and split responsibilities, ensures that leadership has to be really explicit about such things ("lower your SLOs"), instead of just quietly deprioritizing enforcing or reporting on them. (And I'm generally a fan of multiple individuals with well scoped and distinct responsibility who are charged with reaching consensus over a single individual who makes a decision).

aluminussoma 4y ago

> They are given the power to stop deployment. If they can't, they're not SREs Google is the only place where this is even discussed. I cannot name any other company that does this. Not Apple, Meta's Production Engineers, Netflix, Amazon's SysDevEngs, Stripe SRE, Uber, Pinterest, and more. This must mean that nobody else does real SRE!

xorcist 4y ago
Given the availability of Facebook, Netflix and the rest, one might think that Google is on to something there.

donavanm 4y ago

> They are given the power to stop deployment. Sweet. Given by whom > Alice can tell Bob "no more deployments" is literally the point. Fantastic. Why can she say & enforce this. You seem to be going a long ways to describe “what” while explicitly ignoring “why” and “how.” To be clear my thesis is that success in both operating models is built on a combination of top down incentive/buy in and broad consensus expressed as a cultural norm. Im not being intentionally obtuse, but Im yet to see any other hypothesis or method of action whicb would explain the differentiation of ability to execute.

joshuamorton 4y ago
> Sweet. Given by whom Directly, the answer is the permissions system. In my neck of the woods, devs are quite literally incapable of pushing things to production on their own. The automated tooling handles it 99% of the time, but in the case that they turn it off, only the SREs have production access necessary to actually push new things. But I think you're speaking organizationally, and the answer there is that that agreement was made by SVPs and is enforced by the PRR process. When you adopt SRE support, you hand over the keys, so to speak. > Fantastic. Why can she say & enforce this. Because there's nothing Bob can do except escalate, and SRE leadership will support Alice all the way up to the SVP level. > To be clear my thesis is that success in both operating models is built on a combination of top down incentive/buy in and broad consensus expressed as a cultural norm. My point is that the process creates the cultural norm, and enforces everywhere. That is, your org might have these good processes, and that's laudable. But if Andy turned around tomorrow and said "hey, this year your first priority needs to be shipping new features, because we're falling behind", would your org maintain that commitment? Would every exec? The same question goes for every exec speaking to their subordinates. Its much more difficult to turn to the guy whose job is reliability and reports directly to the CEO and say "hey, your focus this year is shipping features". Like maybe you could say "your focus is shipping features reliably this year", and that should take the focus off of some other things that the SRE org is involved in, but I'm not convinced. Or briefly, your approach is betting on there being a cohesive company culture of reliability that permeates every team, org, and manager. We both know that's not true, at Amazon or Google or any other company. They're all big enough to have different cultures in different parts. A distinct org and reporting structure, and split responsibilities, ensures that leadership has to be really explicit about such things ("lower your SLOs"), instead of just quietly deprioritizing enforcing or reporting on them. (And I'm generally a fan of multiple individuals with well scoped and distinct responsibility who are charged with reaching consensus over a single individual who makes a decision).