3 ms·

> incentives are misaligned If an organization decides that reliability is important (which honestly isn't always the case), then the question becomes whether

> incentives are misaligned If an organization decides that reliability is important (which honestly isn't always the case), then the question becomes whether they can trust Product to include reliability work or not. If not, if reliability is so essential to the business that the business is willing to sacrifice velocity for it, then a separate SRE reporting structure can exist to slow down Product when reliability objectives aren't being met. If reliability is negotiable, then an embedded structure means Product can make its own prioritization decisions and move quickly when it needs to. The embedded structure doesn't mean that SRE always, necessarily gets overruled. It means that SRE leadership needs observability into when SRE is getting deprioritized by Product, so that consistent failure by Product to respect a core business goal (reliability) can reach executives who will, ahem, clarify for Product that they need to spend some time prioritizing reliability.

dilyevsky 4y ago
My experience suggests that you can’t bolt reliability on the side when you suddenly decide to prioritize it. Not after certain scale at least.
- solatic 4y ago
  Even with an embedded model, SRE leadership is responsible, first and foremost, for setting up tooling like observability and auto-healing (i.e. Kubernetes) for the company's workloads, which gets you about 90% of the way there. If Product can't keep their service online because they never wrote tests etc., then the way you "bolt reliability on the side" is, SRE leadership gives the exec the raw lack-of-availability numbers (see: observability tooling), then the exec tells Product to stop fudging around and prioritize making the availability numbers better. Then Product management freezes feature development and takes a vacation for a sprint (or two) while Product developers focus on bolstering their QA.
  - dilyevsky 4y ago
    This scenario seems overly naive unless there exists totally separate sre chain like I described up the thread.

dilyevsky 4y ago

My experience suggests that you can’t bolt reliability on the side when you suddenly decide to prioritize it. Not after certain scale at least.

solatic 4y ago
Even with an embedded model, SRE leadership is responsible, first and foremost, for setting up tooling like observability and auto-healing (i.e. Kubernetes) for the company's workloads, which gets you about 90% of the way there. If Product can't keep their service online because they never wrote tests etc., then the way you "bolt reliability on the side" is, SRE leadership gives the exec the raw lack-of-availability numbers (see: observability tooling), then the exec tells Product to stop fudging around and prioritize making the availability numbers better. Then Product management freezes feature development and takes a vacation for a sprint (or two) while Product developers focus on bolstering their QA.
- dilyevsky 4y ago
  This scenario seems overly naive unless there exists totally separate sre chain like I described up the thread.