4 ms·

I think this is the point where we're getting too serious about the discipline. We need to be a little more grounded in reality. SRE is basically a highly polis

I think this is the point where we're getting too serious about the discipline. We need to be a little more grounded in reality. SRE is basically a highly polished, best-practice, telemetry-and-automation-focused sysadmin. And a sysadmin doesn't really have a say in the business. They can make suggestions, like error budgets. But what about sales contracts? The contracts define what kind of guarantees you have to give and on what timeline. And what about customer experience? How is SRE supposed to quantify if the users are happy if they can't tell the UX and customer support how to build the product to better collect user happiness? Or tell the devs how to design their apps to fail less often? If you keep optimizing and focusing on all the ways to improve satisfaction, you literally end up an executive director of product. If we really want to make products that users are happy with, we need to teach executives how to achieve it, not SREs. SREs shouldn't be driving the bus from the back row. They should be rotating the tires and checking the fuel and oil gauges and telling the driver when to stop for gas. Let's teach executives how to set everyone up to succeed, and then SRE doesn't have to get all existential just to keep the bus running.

dilyevsky 4y ago
> How is SRE supposed to quantify if the users are happy They can easily tell if the users aren’t happy which is usually same time their pagers go off
spondyl 4y ago
> If we really want to make products that users are happy with, we need to teach executives how to achieve it, not SREs. SREs shouldn't be driving the bus from the back row. They should be rotating the tires and checking the fuel and oil gauges and telling the driver when to stop for gas. To extend on this analogy a bit, the theory is that you might have business OKRs that are driven by SLOs. For example, you might have a wishy washy goal of having happy customers. If you're in the shipping business, one aspect of being happy with the service would be to receive deliveries on time (or earlier!) so you might set an OKR where the objective is "Deliveries are made in a timely manner" where key results might be the number of deliveries made on time and inversely, the number of deliveries that are not returned. By pitting those two metrics (speed and state of the delivery) together, you can ideally capture that an increase in speedy deliveries is not just achieved by people throwing packages out of moving vehicles as one or the other will decline. Of course, simply setting these metrics doesn't mean very much but this is where we can drop down to a lower level and make use of SLOs. An SLO is just some number with a target after all, it doesn't have to be technical by any means. So, how can we help drive our goal of timely deliveries? Well, we might set an OKR that says something like "99% of vehicles should have their oil checked every 3 months". On the face of it, that has nothing at all to do with vehicle deliveries but inspecting historical data might determine that failed deliveries were often correlated with poorly serviced vehicles so by ensuring they're serviced, we can indirectly improve customer happiness by something that seems entirely unrelated. This is all in theory of course and assumes your business is able to communicate across layers and share some sort of overall vision which is very hard! It also assumes someone (in the middle?) is able to bridge the gap to co-locate SLOs with OKRs and what not. Anyway, that's how I understand the "dream state" but have I ever seen it actually accomplished? I can't say I have!
mihaigalos 4y ago
> I think this is the point where we're getting too serious about the discipline. In my opinion, we are not serious enough. The role of the cited sysadmin is dying or dead in the modern company. As an SRE, you are expected to directly contribute to the product to improve SLOs as well as several other dimensions you quoted (like user happiness). I've personally performed code refactoring to reduce build job duration slowing down pipelines for _everybody_ in a component I was unfamiliar with. The code review was then performed by a code owner and the PR merged. > The contracts define what kind of guarantees you have to give and on what timeline. That's why you commit to ship product increments, not hard deadlines. See the failed launch of healthcare.gov, for example. > How is SRE supposed to quantify if the users are happy? The same way a dev would: review with stakeholders, "show-and-tell" where you collect feedback and improve your domain knowledge by focusing on the customer problem instead of your own vision of the solution. > Or tell the devs how to design their apps to fail less often? This is precisely why SREs would provide value by direct contribution. > you literally end up an executive director of product. We are all ambassadors of the product and can contribute past our designated "boundaries". That's what cross-functional is all about. No, you cannot make business decisions but if it is code, you have very high leverage. > If we really want to make products that users are happy with, we need to teach executives how to achieve it, not SREs. This has not happened yet and I suspect it never will. > SREs shouldn't be driving the bus from the back row [..] Let's teach executives how to set everyone up to succeed [..]. Agreed. Do you have a concrete proposal? Without fundamental organizational change, this is impossible. Conway's law still applies, so does its inverse. Meaning you can break down silos (including at executive level) by migrating from component to cross-functional feature teams.
- throwaway892238 4y ago
  My proposal would be to form an industry panel made up of experts in each of the discrete roles that make up a typical software product value chain. Ask that panel to produce the following: 1. Adapt industry best practices into a training program to teach executives what is necessary for a software product with happy customers. Very simple and to the point: here are the components of the value chain, here is their common problems, here is what each job needs to do to prevent them, here is how they have to work together. The purpose isn't to micro-manage each department, but to know how to identify when they aren't operating correctly and how to put people in place who know #2 below. 2. Take the above and break it down for each part of the value chain. Have the managers/directors of each part learn what their own department's best practice requires, its pitfalls, strategies. Teach them how inter-departmental overlap is required for happy customers, how departments should set each other up for success and collaborate, how "ownership" is tenuous and demands collaboration and compromise. They're all tributaries of a river. Charge money for people to go through that training and get a certificate. The money goes back into the whole thing. This becomes a baseline that shows you understand how to work on software products with best practice geared towards customer satisfaction.

dilyevsky 4y ago

> How is SRE supposed to quantify if the users are happy They can easily tell if the users aren’t happy which is usually same time their pagers go off

spondyl 4y ago

> If we really want to make products that users are happy with, we need to teach executives how to achieve it, not SREs. SREs shouldn't be driving the bus from the back row. They should be rotating the tires and checking the fuel and oil gauges and telling the driver when to stop for gas. To extend on this analogy a bit, the theory is that you might have business OKRs that are driven by SLOs. For example, you might have a wishy washy goal of having happy customers. If you're in the shipping business, one aspect of being happy with the service would be to receive deliveries on time (or earlier!) so you might set an OKR where the objective is "Deliveries are made in a timely manner" where key results might be the number of deliveries made on time and inversely, the number of deliveries that are not returned. By pitting those two metrics (speed and state of the delivery) together, you can ideally capture that an increase in speedy deliveries is not just achieved by people throwing packages out of moving vehicles as one or the other will decline. Of course, simply setting these metrics doesn't mean very much but this is where we can drop down to a lower level and make use of SLOs. An SLO is just some number with a target after all, it doesn't have to be technical by any means. So, how can we help drive our goal of timely deliveries? Well, we might set an OKR that says something like "99% of vehicles should have their oil checked every 3 months". On the face of it, that has nothing at all to do with vehicle deliveries but inspecting historical data might determine that failed deliveries were often correlated with poorly serviced vehicles so by ensuring they're serviced, we can indirectly improve customer happiness by something that seems entirely unrelated. This is all in theory of course and assumes your business is able to communicate across layers and share some sort of overall vision which is very hard! It also assumes someone (in the middle?) is able to bridge the gap to co-locate SLOs with OKRs and what not. Anyway, that's how I understand the "dream state" but have I ever seen it actually accomplished? I can't say I have!

mihaigalos 4y ago

> I think this is the point where we're getting too serious about the discipline. In my opinion, we are not serious enough. The role of the cited sysadmin is dying or dead in the modern company. As an SRE, you are expected to directly contribute to the product to improve SLOs as well as several other dimensions you quoted (like user happiness). I've personally performed code refactoring to reduce build job duration slowing down pipelines for _everybody_ in a component I was unfamiliar with. The code review was then performed by a code owner and the PR merged. > The contracts define what kind of guarantees you have to give and on what timeline. That's why you commit to ship product increments, not hard deadlines. See the failed launch of healthcare.gov, for example. > How is SRE supposed to quantify if the users are happy? The same way a dev would: review with stakeholders, "show-and-tell" where you collect feedback and improve your domain knowledge by focusing on the customer problem instead of your own vision of the solution. > Or tell the devs how to design their apps to fail less often? This is precisely why SREs would provide value by direct contribution. > you literally end up an executive director of product. We are all ambassadors of the product and can contribute past our designated "boundaries". That's what cross-functional is all about. No, you cannot make business decisions but if it is code, you have very high leverage. > If we really want to make products that users are happy with, we need to teach executives how to achieve it, not SREs. This has not happened yet and I suspect it never will. > SREs shouldn't be driving the bus from the back row [..] Let's teach executives how to set everyone up to succeed [..]. Agreed. Do you have a concrete proposal? Without fundamental organizational change, this is impossible. Conway's law still applies, so does its inverse. Meaning you can break down silos (including at executive level) by migrating from component to cross-functional feature teams.

throwaway892238 4y ago
My proposal would be to form an industry panel made up of experts in each of the discrete roles that make up a typical software product value chain. Ask that panel to produce the following: 1. Adapt industry best practices into a training program to teach executives what is necessary for a software product with happy customers. Very simple and to the point: here are the components of the value chain, here is their common problems, here is what each job needs to do to prevent them, here is how they have to work together. The purpose isn't to micro-manage each department, but to know how to identify when they aren't operating correctly and how to put people in place who know #2 below. 2. Take the above and break it down for each part of the value chain. Have the managers/directors of each part learn what their own department's best practice requires, its pitfalls, strategies. Teach them how inter-departmental overlap is required for happy customers, how departments should set each other up for success and collaborate, how "ownership" is tenuous and demands collaboration and compromise. They're all tributaries of a river. Charge money for people to go through that training and get a certificate. The money goes back into the whole thing. This becomes a baseline that shows you understand how to work on software products with best practice geared towards customer satisfaction.