3 ms·

Isnt this a bit reductive? The other side of the fence is “How would an independent SRE org be able to fund/influence/prioritize inside the product org.” And Im

Isnt this a bit reductive? The other side of the fence is “How would an independent SRE org be able to fund/influence/prioritize inside the product org.” And Im pretty sure the answer there is > A dedicated model can't work unless the company leadership wants to make it work. Product reliability and performance is not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately. So you have the same answer, incentives and top down buy in. And if you have incentives at management layers, and top down buy in, then youve solved the root people + business problem. And now how/why do you choose between options?

joshuamorton 4y ago
> Isnt this a bit reductive? The other side of the fence is “How would an independent SRE org be able to fund/influence/prioritize inside the product org.” And Im pretty sure the answer there is No, the answer is that the SRE org is unilaterally able to stop supporting the product/implement a production freeze if the product org is not doing enough reliability work (or falls out of SLO, or whatever). That's literally the trick, you take the ability to deploy out of the product org's hands and put it into the hands of an organization whose job is reliability, and whose management up to the VP level will back them up on that! It is far easier to have an organization with the incentive "reliability" and an organization with the incentive "features" and have senior management deal with conflicts when they arise, than to have every middle manager try to correctly split their prioritization in the correct way.
- donavanm 4y ago
  > the SRE org is unilaterally able to stop supporting the product/implement a production freeze Who exactly does this. And by what mechanism do you enforce: > take the ability to deploy out of the product org's hands If it is not "incentives and top down buy in" from above/at both orgs, what is it? Are you literally proposing some reality where VP Alice nonconsensually tells peer VP Bob "you're missing your roadmap ship dates. Im taking away your orgs prod creds because your lines arent up and to the right enough?" Because when there's a conflict between the two it's either going to get resolved by negotiation and common understanding, at and above Alice & Bob, or Bob is going to tell Alice to stay in her lane and get bent. Edit: For the above, Im interested in the medium/long term. Yes, if you have hived off deployment gates to a different team I believe the could tactically say "no" once. I do not believe they could maintain that position. Unilateral actions only work when the power dynamics support the person taking action. And those power dynamics are roughly more senior management, broad cultural consensus, and revenue, in my experience. So I'm really curious what enables the dedicated SRE orgs unilateral force of action if its not incentives & top down buy in. This isn't hypothetical to me. Im a principal systems engineer at AWS who spent years reporting directly to a VP with a billion dollar business, and now work on operations & incident management in the same VPs now larger org running multiple billion dollar businesses. I was there when a billion dollar business took a year off the feature roadmap to focus on availability & performance. No one "told" us to do that, but it was supported as the best thing for the business. I've also stopped working on teams when they weren't making progress with operational investments.
  - joshuamorton 4y ago
    > Who exactly does this. And by what mechanism do you enforce: That's one of the key pieces of what makes SRE "SRE" as opposed to an ops team. They are given the power to stop deployment. If they can't, they're not SREs. > Are you literally proposing some reality where VP Alice nonconsensually tells peer VP Bob "you're missing your roadmap ship dates. Im taking away your orgs prod creds because your lines arent up and to the right enough?" Yes, and no. No because "roadmap ship dates" aren't reliability. I'm describing a reality where VP Alice nonconsentually tells VP Bob "you've had three outages in the past three weeks, we're out of SLO and so you no longer get to deploy new versions until we have remediations in place to prevent future outages". (and of course they both agreed to this) Of course, Bob, being a rational human probably starts taking actions after the first outage, and so it never gets to this point, but yes, that Alice can tell Bob "no more deployments" is literally the point.
    - aluminussoma 4y ago
      > They are given the power to stop deployment. If they can't, they're not SREs Google is the only place where this is even discussed. I cannot name any other company that does this. Not Apple, Meta's Production Engineers, Netflix, Amazon's SysDevEngs, Stripe SRE, Uber, Pinterest, and more. This must mean that nobody else does real SRE!
      - xorcist 4y ago
        Given the availability of Facebook, Netflix and the rest, one might think that Google is on to something there.
    - donavanm 4y ago
      > They are given the power to stop deployment. Sweet. Given by whom > Alice can tell Bob "no more deployments" is literally the point. Fantastic. Why can she say & enforce this. You seem to be going a long ways to describe “what” while explicitly ignoring “why” and “how.” To be clear my thesis is that success in both operating models is built on a combination of top down incentive/buy in and broad consensus expressed as a cultural norm. Im not being intentionally obtuse, but Im yet to see any other hypothesis or method of action whicb would explain the differentiation of ability to execute.

joshuamorton 4y ago

> Isnt this a bit reductive? The other side of the fence is “How would an independent SRE org be able to fund/influence/prioritize inside the product org.” And Im pretty sure the answer there is No, the answer is that the SRE org is unilaterally able to stop supporting the product/implement a production freeze if the product org is not doing enough reliability work (or falls out of SLO, or whatever). That's literally the trick, you take the ability to deploy out of the product org's hands and put it into the hands of an organization whose job is reliability, and whose management up to the VP level will back them up on that! It is far easier to have an organization with the incentive "reliability" and an organization with the incentive "features" and have senior management deal with conflicts when they arise, than to have every middle manager try to correctly split their prioritization in the correct way.

donavanm 4y ago
> the SRE org is unilaterally able to stop supporting the product/implement a production freeze Who exactly does this. And by what mechanism do you enforce: > take the ability to deploy out of the product org's hands If it is not "incentives and top down buy in" from above/at both orgs, what is it? Are you literally proposing some reality where VP Alice nonconsensually tells peer VP Bob "you're missing your roadmap ship dates. Im taking away your orgs prod creds because your lines arent up and to the right enough?" Because when there's a conflict between the two it's either going to get resolved by negotiation and common understanding, at and above Alice & Bob, or Bob is going to tell Alice to stay in her lane and get bent. Edit: For the above, Im interested in the medium/long term. Yes, if you have hived off deployment gates to a different team I believe the could tactically say "no" once. I do not believe they could maintain that position. Unilateral actions only work when the power dynamics support the person taking action. And those power dynamics are roughly more senior management, broad cultural consensus, and revenue, in my experience. So I'm really curious what enables the dedicated SRE orgs unilateral force of action if its not incentives & top down buy in. This isn't hypothetical to me. Im a principal systems engineer at AWS who spent years reporting directly to a VP with a billion dollar business, and now work on operations & incident management in the same VPs now larger org running multiple billion dollar businesses. I was there when a billion dollar business took a year off the feature roadmap to focus on availability & performance. No one "told" us to do that, but it was supported as the best thing for the business. I've also stopped working on teams when they weren't making progress with operational investments.
- joshuamorton 4y ago
  > Who exactly does this. And by what mechanism do you enforce: That's one of the key pieces of what makes SRE "SRE" as opposed to an ops team. They are given the power to stop deployment. If they can't, they're not SREs. > Are you literally proposing some reality where VP Alice nonconsensually tells peer VP Bob "you're missing your roadmap ship dates. Im taking away your orgs prod creds because your lines arent up and to the right enough?" Yes, and no. No because "roadmap ship dates" aren't reliability. I'm describing a reality where VP Alice nonconsentually tells VP Bob "you've had three outages in the past three weeks, we're out of SLO and so you no longer get to deploy new versions until we have remediations in place to prevent future outages". (and of course they both agreed to this) Of course, Bob, being a rational human probably starts taking actions after the first outage, and so it never gets to this point, but yes, that Alice can tell Bob "no more deployments" is literally the point.
  - aluminussoma 4y ago
    > They are given the power to stop deployment. If they can't, they're not SREs Google is the only place where this is even discussed. I cannot name any other company that does this. Not Apple, Meta's Production Engineers, Netflix, Amazon's SysDevEngs, Stripe SRE, Uber, Pinterest, and more. This must mean that nobody else does real SRE!
    - xorcist 4y ago
      Given the availability of Facebook, Netflix and the rest, one might think that Google is on to something there.
  - donavanm 4y ago
    > They are given the power to stop deployment. Sweet. Given by whom > Alice can tell Bob "no more deployments" is literally the point. Fantastic. Why can she say & enforce this. You seem to be going a long ways to describe “what” while explicitly ignoring “why” and “how.” To be clear my thesis is that success in both operating models is built on a combination of top down incentive/buy in and broad consensus expressed as a cultural norm. Im not being intentionally obtuse, but Im yet to see any other hypothesis or method of action whicb would explain the differentiation of ability to execute.