5 ms·

In the several examples of separate SRE orgs that I've seen, SRE builds their own little empires with no alignment to the product. A cynic would guess that the

In the several examples of separate SRE orgs that I've seen, SRE builds their own little empires with no alignment to the product. A cynic would guess that the goal of SRE management is to hire enough SREs so there can be senior managers and directors and eventually a VP with pay packages to boot. They come to Product from the mountain-top with their holy scriptures (the Google SRE book), prescribing surface solutions to problems with deep roots (pray it away with SLOs). SRE 2.0 needs to make the embedded model work.

joshuamorton 4y ago
The entire point of the unembedded model is to create the management structure that isn't strictly beholden to the whims of product. An embedded model can't work unless the company leadership wants to make it work, and product leadership won't care enough (as demonstrated by: everything pre-sre). An embedded model is designed to fail, because the incentives are misaligned. Like, what your asking is "hey, random engineer or engineering manager who is an 'SRE', make the VP of product prioritize your work and staff your team correctly!". That's not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately.
- arminiusreturns 4y ago
  Greybeard sysadmin turned SRE-mask ops dude here, and this is correct. I've seen inside literally hundreds of orgs in my career, and almost always the real disconnect exists at the c-suite and board level. For a long time I blamed them for this dysfunction, but it tooks some years and maturity to start thinking more about how I can speak their language to reach their lofty, double-scheduled minds. Its a failure on both ends. This is where a good CTO/CIO with actual leadership qualities will make or break an org, as they should usually be the SRE-esque visionaries, leaders, defenders. When that doesnt happen, and lets face it, it doesnt happen very often, you get cycles of "try this other ops/admin/swe/sre structure and management method" until it falls apart and new management tries something else or they start rolling heads right under the CTO/CIO until it becomes an 1000 and 1 nights. One of the biggest warning signs of this is when tech competent middle managers (worth their weight in gold) start jumping ship enmasse.
- donavanm 4y ago
  Isnt this a bit reductive? The other side of the fence is “How would an independent SRE org be able to fund/influence/prioritize inside the product org.” And Im pretty sure the answer there is > A dedicated model can't work unless the company leadership wants to make it work. Product reliability and performance is not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately. So you have the same answer, incentives and top down buy in. And if you have incentives at management layers, and top down buy in, then youve solved the root people + business problem. And now how/why do you choose between options?
  - joshuamorton 4y ago
    > Isnt this a bit reductive? The other side of the fence is “How would an independent SRE org be able to fund/influence/prioritize inside the product org.” And Im pretty sure the answer there is No, the answer is that the SRE org is unilaterally able to stop supporting the product/implement a production freeze if the product org is not doing enough reliability work (or falls out of SLO, or whatever). That's literally the trick, you take the ability to deploy out of the product org's hands and put it into the hands of an organization whose job is reliability, and whose management up to the VP level will back them up on that! It is far easier to have an organization with the incentive "reliability" and an organization with the incentive "features" and have senior management deal with conflicts when they arise, than to have every middle manager try to correctly split their prioritization in the correct way.
    - donavanm 4y ago
      > the SRE org is unilaterally able to stop supporting the product/implement a production freeze Who exactly does this. And by what mechanism do you enforce: > take the ability to deploy out of the product org's hands If it is not "incentives and top down buy in" from above/at both orgs, what is it? Are you literally proposing some reality where VP Alice nonconsensually tells peer VP Bob "you're missing your roadmap ship dates. Im taking away your orgs prod creds because your lines arent up and to the right enough?" Because when there's a conflict between the two it's either going to get resolved by negotiation and common understanding, at and above Alice & Bob, or Bob is going to tell Alice to stay in her lane and get bent. Edit: For the above, Im interested in the medium/long term. Yes, if you have hived off deployment gates to a different team I believe the could tactically say "no" once. I do not believe they could maintain that position. Unilateral actions only work when the power dynamics support the person taking action. And those power dynamics are roughly more senior management, broad cultural consensus, and revenue, in my experience. So I'm really curious what enables the dedicated SRE orgs unilateral force of action if its not incentives & top down buy in. This isn't hypothetical to me. Im a principal systems engineer at AWS who spent years reporting directly to a VP with a billion dollar business, and now work on operations & incident management in the same VPs now larger org running multiple billion dollar businesses. I was there when a billion dollar business took a year off the feature roadmap to focus on availability & performance. No one "told" us to do that, but it was supported as the best thing for the business. I've also stopped working on teams when they weren't making progress with operational investments.
- __turbobrew__ 4y ago
  Isn’t reliability a feature of the product? I am currently building out a team for a new product and reliability is quite high on our feature list. We have a embedded SREs who are working on improving our product reliability full time. We share the operational workload between the SWEs and SREs instead of just throwing all the operational problems at the SREs. This gives developers an appreciation for reliability and stability and makes sure that the SREs actually have the time and resources to improve the product instead of being a help desk. It sounds like you needed a separate chain of command for SREs because your product leadership was incompetent.
  - dilyevsky 4y ago
    If your stack is complex enough and/or team is large enough the reliability work can be smeared so thin or be so intricate that it becomes nobody’s responsibility. Hence separate command chain that Google and some others use
- solatic 4y ago
  > incentives are misaligned If an organization decides that reliability is important (which honestly isn't always the case), then the question becomes whether they can trust Product to include reliability work or not. If not, if reliability is so essential to the business that the business is willing to sacrifice velocity for it, then a separate SRE reporting structure can exist to slow down Product when reliability objectives aren't being met. If reliability is negotiable, then an embedded structure means Product can make its own prioritization decisions and move quickly when it needs to. The embedded structure doesn't mean that SRE always, necessarily gets overruled. It means that SRE leadership needs observability into when SRE is getting deprioritized by Product, so that consistent failure by Product to respect a core business goal (reliability) can reach executives who will, ahem, clarify for Product that they need to spend some time prioritizing reliability.
  - dilyevsky 4y ago
    My experience suggests that you can’t bolt reliability on the side when you suddenly decide to prioritize it. Not after certain scale at least.
    - solatic 4y ago
      Even with an embedded model, SRE leadership is responsible, first and foremost, for setting up tooling like observability and auto-healing (i.e. Kubernetes) for the company's workloads, which gets you about 90% of the way there. If Product can't keep their service online because they never wrote tests etc., then the way you "bolt reliability on the side" is, SRE leadership gives the exec the raw lack-of-availability numbers (see: observability tooling), then the exec tells Product to stop fudging around and prioritize making the availability numbers better. Then Product management freezes feature development and takes a vacation for a sprint (or two) while Product developers focus on bolstering their QA.
      - dilyevsky 4y ago
        This scenario seems overly naive unless there exists totally separate sre chain like I described up the thread.
dilyevsky 4y ago
Yes SLOs are kind of like OKRs of sre management - they only ever as good as the objectives and objectives are often not set very well by senior leadership. Ultimately that stems from the fact that reliability is not valued by the market. You see a lot of the same things in security (outside of the boneheaded compliance cya).

joshuamorton 4y ago

The entire point of the unembedded model is to create the management structure that isn't strictly beholden to the whims of product. An embedded model can't work unless the company leadership wants to make it work, and product leadership won't care enough (as demonstrated by: everything pre-sre). An embedded model is designed to fail, because the incentives are misaligned. Like, what your asking is "hey, random engineer or engineering manager who is an 'SRE', make the VP of product prioritize your work and staff your team correctly!". That's not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately.

arminiusreturns 4y ago
Greybeard sysadmin turned SRE-mask ops dude here, and this is correct. I've seen inside literally hundreds of orgs in my career, and almost always the real disconnect exists at the c-suite and board level. For a long time I blamed them for this dysfunction, but it tooks some years and maturity to start thinking more about how I can speak their language to reach their lofty, double-scheduled minds. Its a failure on both ends. This is where a good CTO/CIO with actual leadership qualities will make or break an org, as they should usually be the SRE-esque visionaries, leaders, defenders. When that doesnt happen, and lets face it, it doesnt happen very often, you get cycles of "try this other ops/admin/swe/sre structure and management method" until it falls apart and new management tries something else or they start rolling heads right under the CTO/CIO until it becomes an 1000 and 1 nights. One of the biggest warning signs of this is when tech competent middle managers (worth their weight in gold) start jumping ship enmasse.
donavanm 4y ago
Isnt this a bit reductive? The other side of the fence is “How would an independent SRE org be able to fund/influence/prioritize inside the product org.” And Im pretty sure the answer there is > A dedicated model can't work unless the company leadership wants to make it work. Product reliability and performance is not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately. So you have the same answer, incentives and top down buy in. And if you have incentives at management layers, and top down buy in, then youve solved the root people + business problem. And now how/why do you choose between options?
- joshuamorton 4y ago
  > Isnt this a bit reductive? The other side of the fence is “How would an independent SRE org be able to fund/influence/prioritize inside the product org.” And Im pretty sure the answer there is No, the answer is that the SRE org is unilaterally able to stop supporting the product/implement a production freeze if the product org is not doing enough reliability work (or falls out of SLO, or whatever). That's literally the trick, you take the ability to deploy out of the product org's hands and put it into the hands of an organization whose job is reliability, and whose management up to the VP level will back them up on that! It is far easier to have an organization with the incentive "reliability" and an organization with the incentive "features" and have senior management deal with conflicts when they arise, than to have every middle manager try to correctly split their prioritization in the correct way.
  - donavanm 4y ago
    > the SRE org is unilaterally able to stop supporting the product/implement a production freeze Who exactly does this. And by what mechanism do you enforce: > take the ability to deploy out of the product org's hands If it is not "incentives and top down buy in" from above/at both orgs, what is it? Are you literally proposing some reality where VP Alice nonconsensually tells peer VP Bob "you're missing your roadmap ship dates. Im taking away your orgs prod creds because your lines arent up and to the right enough?" Because when there's a conflict between the two it's either going to get resolved by negotiation and common understanding, at and above Alice & Bob, or Bob is going to tell Alice to stay in her lane and get bent. Edit: For the above, Im interested in the medium/long term. Yes, if you have hived off deployment gates to a different team I believe the could tactically say "no" once. I do not believe they could maintain that position. Unilateral actions only work when the power dynamics support the person taking action. And those power dynamics are roughly more senior management, broad cultural consensus, and revenue, in my experience. So I'm really curious what enables the dedicated SRE orgs unilateral force of action if its not incentives & top down buy in. This isn't hypothetical to me. Im a principal systems engineer at AWS who spent years reporting directly to a VP with a billion dollar business, and now work on operations & incident management in the same VPs now larger org running multiple billion dollar businesses. I was there when a billion dollar business took a year off the feature roadmap to focus on availability & performance. No one "told" us to do that, but it was supported as the best thing for the business. I've also stopped working on teams when they weren't making progress with operational investments.
__turbobrew__ 4y ago
Isn’t reliability a feature of the product? I am currently building out a team for a new product and reliability is quite high on our feature list. We have a embedded SREs who are working on improving our product reliability full time. We share the operational workload between the SWEs and SREs instead of just throwing all the operational problems at the SREs. This gives developers an appreciation for reliability and stability and makes sure that the SREs actually have the time and resources to improve the product instead of being a help desk. It sounds like you needed a separate chain of command for SREs because your product leadership was incompetent.
- dilyevsky 4y ago
  If your stack is complex enough and/or team is large enough the reliability work can be smeared so thin or be so intricate that it becomes nobody’s responsibility. Hence separate command chain that Google and some others use
solatic 4y ago
> incentives are misaligned If an organization decides that reliability is important (which honestly isn't always the case), then the question becomes whether they can trust Product to include reliability work or not. If not, if reliability is so essential to the business that the business is willing to sacrifice velocity for it, then a separate SRE reporting structure can exist to slow down Product when reliability objectives aren't being met. If reliability is negotiable, then an embedded structure means Product can make its own prioritization decisions and move quickly when it needs to. The embedded structure doesn't mean that SRE always, necessarily gets overruled. It means that SRE leadership needs observability into when SRE is getting deprioritized by Product, so that consistent failure by Product to respect a core business goal (reliability) can reach executives who will, ahem, clarify for Product that they need to spend some time prioritizing reliability.
- dilyevsky 4y ago
  My experience suggests that you can’t bolt reliability on the side when you suddenly decide to prioritize it. Not after certain scale at least.
  - solatic 4y ago
    Even with an embedded model, SRE leadership is responsible, first and foremost, for setting up tooling like observability and auto-healing (i.e. Kubernetes) for the company's workloads, which gets you about 90% of the way there. If Product can't keep their service online because they never wrote tests etc., then the way you "bolt reliability on the side" is, SRE leadership gives the exec the raw lack-of-availability numbers (see: observability tooling), then the exec tells Product to stop fudging around and prioritize making the availability numbers better. Then Product management freezes feature development and takes a vacation for a sprint (or two) while Product developers focus on bolstering their QA.
    - dilyevsky 4y ago
      This scenario seems overly naive unless there exists totally separate sre chain like I described up the thread.

dilyevsky 4y ago

Yes SLOs are kind of like OKRs of sre management - they only ever as good as the objectives and objectives are often not set very well by senior leadership. Ultimately that stems from the fact that reliability is not valued by the market. You see a lot of the same things in security (outside of the boneheaded compliance cya).