8 ms·

What SRE Could Be

96 points by zdw 4y ago

citizenpaul 4y ago
SRE has gotten so good that people don't even think about it. It is default expectation that any online service is basically five nine availability or better. If it is worse than five nine is the only time it gets talked about. SRE is really one of the only well solidified parts of tech as far as career and expectations go. The author seems to look at the same info and come to a different conclusion.
- 4by4by4 4y ago
  I’m curious what services you interact with that have five nines availability. Most services that I interact with, AWS APIs as an example, top out at an SLA of four nines for availability.
  - dijit 4y ago
    Even Google's own load balancer only offers 4 nines. https://cloud.google.com/compute/sla https://cloud.google.com/compute/sla
    - citizenpaul 4y ago
      OMG you guys are missing the point. THe article is not talking about what is a good SLA level. Its that SRE "could be so much more" My point is no it cannot. SRE is now a commodity, the expectation is that the system is up pretty much always. Even two nine is only an hour or so a year downtime. Jeeze
  - rwiggins 4y ago
    There is some nuance there. Because SLA violations typically have real-money impact, they're often backed by internal objectives (SLOs) that are quite a bit tighter. So if an API promises four nines of availability in its SLA, there is probably an internal target that's at least four-and-a-half nines, if not five. Which is to say, I would not be surprised if many APIs with a four-nines SLA were actually closer to five nines. From a planning perspective, you should probably stick to the public guarantees (i.e. the SLA), though. Although things can always go horribly wrong, like in the big Atlassian deletion snafu recently.
    - citizenpaul 4y ago
      I'm saying SRE/SLA has become a commodity. No one thinks about it. They just expect the systems to be up. The article is about how SRE can be more than "downtime" I propose it cannot and they are wrong.
  - citizenpaul 4y ago
    I guess I made a poor point. I am saying that SRE has basically solidified into the expectation of the high 99 percentile as the bottom level of expected downtime. Even two nine is like 1hr per year. There really isn't much to discuss in the industry of SRE. You basically have to be flawless. Its like the janitor. Miss the garbage pickup one day and heads roll. If you have SRE's and the service is down you are done. There is no longer any nuance which is why the CEO guy in the OP doesn't think about SRE anymore than the garbage collection.
- kodah 4y ago
  In my experience as a SRE, most teams/companies claiming above three nines are not measuring correctly or lying.
  - harpratap 4y ago
    Depends a lot on scale too. 3 nines while serving 1M RPS is quite a different challenge from 3 nines at 1 RPS
  - pojzon 4y ago
    My previous company SLA was for 99.999% uptime which was built on top of a service with 99.9% uptime without anything additional in between. SLAs in most cases are bullshit and if anything really bad happens (Atlassian) they are just prepared to pay out class lawsuit.
benlivengood 4y ago
A lot of the metrics for outage impact are available, and if you really want to Do Science you can use whatever A/B Canary/experiment framework you have to simulate outages (which, arguably, are not perfect because of down-detectors) on users to understand their behavior. I tend to agree that most people don't care about transient downtime beyond 2 or 3 nines for reputation. Covid has proven how bad people are at reasoning about 0.5% probabilities so there's essentially no chance that they are noticing (en mass) the difference between 2.5 or 3 nines of reliability. Know your customer, as the article points out; more complex customers will know and measure actual availability because they need it for their own reliability calculations. This brings me to my final guess, that SLOs are primarily useful because they provide a risk model that upstream and downstream consumers can reason about. SLOs are a way of building even larger distributed and interdependent systems that can still be somewhat reliable. Precisely what SLIs are chosen and the limits are a little less important than the fact that they are published publicly. There's an economy of reliability in large systems and like any market information about the market is key to its efficiency. If everyone just trusted that everyone was doing their best at maintaining dependencies most of those beliefs would be very unanchored. A side benefit of SLOs is postmortems and communication; people working on codependent products and systems need to talk to each other from time to time and understand the true nature of the interfaces between their systems. Postmortems and SLO negotiations give a concrete framework to do that within, metrics drive quantitative understanding of the systems instead of handwaving.
LadyCailin 4y ago
I’m still trying to figure out what SRE stands for…
- deathanatos 4y ago
  Site Reliability Engineer
- comprev 4y ago
  Something Really Elusive
- dijit 4y ago
  Sysadmins: Really Expensive
  - LilBytes 4y ago
    I'm keeping this too. - also a Sysadmin, really expensive.
- andrewxdiamond 4y ago
  Google somewhat confusingly uses it for both “site reliability engineer” and “site reliability engineering”. Both the practice and the practitioner
  - OJFord 4y ago
    That seems fairly normal? SE & PhD for example.
- deleted 4y ago
  [deleted]
- chrismarlow9 4y ago
  Set Real Expectations
- chucky_z 4y ago
  Principle CJ
mr-ron 4y ago
SRE is a great toolkit of good things that any team can use. Some super amazing chapters I think everyone should read: Eliminating toil: https://sre.google/sre-book/eliminating-toil/ https://sre.google/sre-book/eliminating-toil/ Every team tech and non tech everywhere should read this Incident Management: https://sre.google/workbook/incident-response/ https://sre.google/workbook/incident-response/ Post Mortems: https://sre.google/sre-book/postmortem-culture/ https://sre.google/sre-book/postmortem-culture/ These 2 chapters are necessary reading. SLOS: https://sre.google/sre-book/service-level-objectives/ https://sre.google/sre-book/service-level-objectives/ Disclaimer: I think the SLO chapter is good to know the theory around, but SLOs specifically Ive used very few times.
- sharadov 4y ago
  A good read but a lot of topics in that book apply to google scale companies only. 95% of companies will never reach that scale or deal with those issues.
- citruscomputing 4y ago
  Thank you for posting these, I learned a lot reading them.
throwaway892238 4y ago
I think this is the point where we're getting too serious about the discipline. We need to be a little more grounded in reality. SRE is basically a highly polished, best-practice, telemetry-and-automation-focused sysadmin. And a sysadmin doesn't really have a say in the business. They can make suggestions, like error budgets. But what about sales contracts? The contracts define what kind of guarantees you have to give and on what timeline. And what about customer experience? How is SRE supposed to quantify if the users are happy if they can't tell the UX and customer support how to build the product to better collect user happiness? Or tell the devs how to design their apps to fail less often? If you keep optimizing and focusing on all the ways to improve satisfaction, you literally end up an executive director of product. If we really want to make products that users are happy with, we need to teach executives how to achieve it, not SREs. SREs shouldn't be driving the bus from the back row. They should be rotating the tires and checking the fuel and oil gauges and telling the driver when to stop for gas. Let's teach executives how to set everyone up to succeed, and then SRE doesn't have to get all existential just to keep the bus running.
- dilyevsky 4y ago
  > How is SRE supposed to quantify if the users are happy They can easily tell if the users aren’t happy which is usually same time their pagers go off
- spondyl 4y ago
  > If we really want to make products that users are happy with, we need to teach executives how to achieve it, not SREs. SREs shouldn't be driving the bus from the back row. They should be rotating the tires and checking the fuel and oil gauges and telling the driver when to stop for gas. To extend on this analogy a bit, the theory is that you might have business OKRs that are driven by SLOs. For example, you might have a wishy washy goal of having happy customers. If you're in the shipping business, one aspect of being happy with the service would be to receive deliveries on time (or earlier!) so you might set an OKR where the objective is "Deliveries are made in a timely manner" where key results might be the number of deliveries made on time and inversely, the number of deliveries that are not returned. By pitting those two metrics (speed and state of the delivery) together, you can ideally capture that an increase in speedy deliveries is not just achieved by people throwing packages out of moving vehicles as one or the other will decline. Of course, simply setting these metrics doesn't mean very much but this is where we can drop down to a lower level and make use of SLOs. An SLO is just some number with a target after all, it doesn't have to be technical by any means. So, how can we help drive our goal of timely deliveries? Well, we might set an OKR that says something like "99% of vehicles should have their oil checked every 3 months". On the face of it, that has nothing at all to do with vehicle deliveries but inspecting historical data might determine that failed deliveries were often correlated with poorly serviced vehicles so by ensuring they're serviced, we can indirectly improve customer happiness by something that seems entirely unrelated. This is all in theory of course and assumes your business is able to communicate across layers and share some sort of overall vision which is very hard! It also assumes someone (in the middle?) is able to bridge the gap to co-locate SLOs with OKRs and what not. Anyway, that's how I understand the "dream state" but have I ever seen it actually accomplished? I can't say I have!
aluminussoma 4y ago
SRE has an identity crisis. It means too many things to too many people. In some companies, it is an operational help desk. In a few companies, SRE is rumored to be a different aspect of software engineering. Pay is a rough signal on what a company expects from SRE. If SREs get paid as much or more than Software Engineers, it is a good sign. Yet even that signal is not accurate. I know highly paid SREs stuck in Ops Hell. You could not pay me enough money to do a job like that. It is also not sustainable because eventually the company will fire you and replace you with a lower paid employee who can do that job just as well. I strongly oppose organizations that have SRE separate from Software Engineers. I strongly believe that SREs must be fully integrated into Software Engineering teams (aka the embedded model), with the same reporting lines as their product colleagues. Keeping SRE organizationally separate has done a great disservice to SRE. SRE 2.0 needs to tear down the mini empires that SRE 1.0 built and get back to software engineering.
- dilyevsky 4y ago
  Imo embedded mode is no better - usually just a token ops person that everyone on the team throws over the fence to. I’d say better signal is if sre has its own separate org structure which roots at the cto/vp of eng level. Soon as you have sres reporting to product eng managers it’s beginning of an end.
  - aluminussoma 4y ago
    In the several examples of separate SRE orgs that I've seen, SRE builds their own little empires with no alignment to the product. A cynic would guess that the goal of SRE management is to hire enough SREs so there can be senior managers and directors and eventually a VP with pay packages to boot. They come to Product from the mountain-top with their holy scriptures (the Google SRE book), prescribing surface solutions to problems with deep roots (pray it away with SLOs). SRE 2.0 needs to make the embedded model work.
    - joshuamorton 4y ago
      The entire point of the unembedded model is to create the management structure that isn't strictly beholden to the whims of product. An embedded model can't work unless the company leadership wants to make it work, and product leadership won't care enough (as demonstrated by: everything pre-sre). An embedded model is designed to fail, because the incentives are misaligned. Like, what your asking is "hey, random engineer or engineering manager who is an 'SRE', make the VP of product prioritize your work and staff your team correctly!". That's not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately.
flurie 4y ago
I remember a common, puerile early adulthood "game" in which one would add the words "in bed" to the text of a fortune cookie. I feel like proclamations about the industry by people who were and are at Google need a similar party trick: add "at Google" to the end of each sentence. SRE is Google's answer to Google's problem; it says so right in the first revelatory header of the introductory chapter of the book: "Google’s Approach to Service Management: Site Reliability Engineering". Inherent in the text of technical books that reveal what people did is an attempt to persuade the reader why they should do it, too. In some ways this reveals the central, myopic conceit at the heart of Google: if you don't have Google's problems, you should. And I'm not sure why anyone who isn't Google should just accept this premise. That's not to say that the premise is unacceptable. I'm sure that there are people who want to participate in a thing that looks like Google, except that they'd rather start it from scratch than join the one that already exists. Perhaps those people want to have Google's problems. Now we have a piece, expanded from a talk, by one of the editors of the books that persuaded organizations that, if only they would squint their eyes a bit and turn their head, they could make the problems of their organization orders of magnitude smaller by so many metrics look like the problems of Google. And now, perhaps, Google's answer to Google's problem wasn't quite right, because it didn't involve...something? It could be philosophy or human-centered design, and I wouldn't be surprised to find Google bereft of either, but it certainly isn't going to be found in a paper by Google (ostensibly, a Google answer to a Google question) that predates the first book by nearly a decade. It isn't going to be found in people for whom these things are afterthoughts rather than guiding principles. Instead, shouldn't we be asking whether we let SRE as a Google answer to a Google problem be a solution outside of Google in the first place?
- coward123 4y ago
  Thank you for this - you put into words very well all of the problems I had with that book. In fact, I found myself very frustrated with the authors, wondering if they had any familiarity with all the rest of the decades of work that had been done in related fields outside Google.
luxuryballs 4y ago
After reading these comments I have even less of a clue what SRE actually is…
- 1970-01-01 4y ago
  It's a telltale sign of a naive blogger. If you think anyone outside your profession will read your writing (or code!), always expand acronyms on first use. I'll fix this one for everyone: It’s natural, in the swirling chaos of the past few years, to take a step back and wonder just where everything is going. Though I’ve certainly been doing my fair share of that, I’ve been also thinking about defining precisely where things are. Answering that question for SITE RELIABILITY ENGINEERING (SRE) is not easy. But your question was probably about the job, not the acronym. Carry on.
chazu 4y ago
90% of SREs and SRE managers haven't read the SRE book(s). 99.9% of folks hiring SREs or starting SRE teams haven't read the SRE book. The SRE book (and its sequels) say quite plainly what SRE is and isn't. They also say that not every org is going to be exactly like google so no, "we're not google" isn't an excuse. the E in SRE is for engineering. As in software engineering. SREs are software engineers. Or should be. If your SREs don't know basic SWE principles, they're not SREs. If your org isn't applying software engineering principles to minimizing operational complexity at scale, your org isn't doing SRE. I'm constantly shocked by how hard these things are to grasp, even for most SREs. If the problems I (occasionally) get to solve weren't more interesting than most regular product work, I'd get out of "SRE" entirely.
- throwaway892238 4y ago
  I think this myth exists because Google was (is?) famously obsessed with SWE. But if you actually read the SRE books and look at the actual discipline of SRE ("what's the difference between SWE and SRE?"), SRE is quite blatantly just operations management. The website is a power plant, and the SRE runs the power plant. You don't build parts to run a power plant, you use software (as in manipulate/control/operate) to run it. You act quickly when the numbers go out of line, you write reports and control how much power is going in and out, respond to surges and dips, etc. For whatever reason, Google decided to tell people that the same person who's building the klaxon and the concrete wall and the pipes for the power plant, and the person who's operating the power plant, are one in the same. But that's clearly bunk. Building a part and running a system are completely different disciplines, and anyone who does both will only be half good at both. Humans are shit at multitasking and there's few true polymaths out there. Show me a master programmer and I'll show you an amateur woodworker. I also don't believe software engineering principles will help you reduce operational complexity. If anything, software engineering tends to either make things either inefficient or subtly complicated. Reducing operational complexity comes from the discipline of operations, which isn't engineering. Non-tech companies have known about these distinctions for like a hundred years. Deming applied scientific rigor and analysis to come up with better practices, but he didn't have to design any widgets to do it.
  - 4y ago
krishnadaskp23 4y ago
In a purist approach there are clear guidelines for DevOps and SRE but these are seldom followed. In all the cases I have seen personally, for a developer both DevOps and SRE are support teams. Major cultural changes in an organisation especially at the development end is very hard to make. What has worked for me is having Ops people with some development experience work as DevOps Engineers who work with Dev team for doing whatever is required for making development and release easier. And the SRE team work with development teams for doing whatever is required for making production support easier. Once we have a fairly seamless and well defined pipeline between committing code to supporting in production, the DevOps and SRE Engineers can merge with Dev or Production Support teams depending on their skill set
coward123 4y ago
From 2008-2018, I went all over the world handling incidents and attempting to teach companies how to do the kinds of things the author is talking about. It was multi-diciplinary and very demanding, both in terms of technical and organizational (political and communication) skills. One of the core observations I made was that few companies were willing to actually get better. Rather than figure out what skills they lacked organizationally, or how to improve their processes, or how to align their goals and teams, or any of the lessons they could have learned, it was way easier and cheaper to just keep on status quo. SRE as a practice, a position, a skill - whatever it could have been, it instead failed to really exist apart from maybe a hero here and there who chose to care about it. SLOs (or whatever name might have been used) did not exist to actually measure or inform anything. They were a kind of placebo - come up with something that looks measurable, find a way to make sure we always score well, and make sure there is a wall between these metrics and anyone's compensation / job performance. An example of this, at a very prominent health insurer. I identified issues that could have been solved for, let's say well under $10MM. Rather than do any of that work, the company threw about $50MM in new hardware at the problem. In another case, at a national name Pharma company, about every two years I went back out to do the same basic consulting engagement, because they lacked the leadership to develop the skills in house or to figure out how to align their teams with the work they actually did. Still another case (prominent bank), they had this serious problem that blocked their commercial lenders from doing their job... they finally called me in after about a year of fooling around, I got the problem fixed, rolled out a bunch of tooling and training, but I guarantee they never used any of it... again - problem outright blocked key bank personnel from doing their job and yet it took them a year to do anything about it. So... I guess what I'm saying is that yes SRE work can be fun and rewarding, but it's just not something most companies have the maturity to really do. I read the Google SRE book way back when, but I didn't find it particularly insightful. Likewise, I'm not really sure what to take away from this blog - for me, it doesn't answer any questions about a clear path forward and leads me to believe not much has changed since I left that kind of work years ago.
mycentstoo 4y ago
Somewhat relatedly, I converted from SWE to SRE for three reasons. 1. SWE interviews are torture and no matter what I accomplish at my job, the next job I have to re-prove everything all over again. SRE interviews do not typically have this style of interview (I've heard Meta and Alphabet do however). 2. SRE focuses on lower level abstractions and therefore seems less prone to product oversight which is tiresome. SRE work is typically longer lived and deadlines less tight. 3. SRE jobs seem even more remote friendly than traditional SWE roles because of reason 2. I'm much better as a SWE but I'm paid more as an SRE and have much more freedom. I don't actually like ops work that much but I'll take that trade off to guarantee the other three advantages above. All that said, in my short time as an SRE I've noticed that it's very much in an identity crises because there's not really a standard job. Infrastructure is very different company to company and some companies expect SREs to be able to code as well as SWEs whereas others don't expect you know how to code at all. In my experience, an SRE is someone who's principal goal is to maintain infrastructure and ensure code is able to be developed, built and shipped. In the "better" SRE roles, SREs are able to build platform tooling rather than just ops.
- pojzon 4y ago
  > principal goal is to maintain infrastructure and ensure code is able to be developed, built and shipped From the SRE vs DevOps definition I just read from the top of google -> this description sounds more like a regular DevOps role instead of SRE. But IMHO, SRE is just a Google way of naming regular SystemAdmin DevOps role. I see no difference. I did exactly the same tasks as an SRE and as a Senior DevOps. Could anyone with 20+ years of experience explain why exactly we still distinguish those ? SRE is supposed to be a superset of DevOps but how?
  - AlchemistCamp 4y ago
    > SRE is supposed to be a superset of DevOps but how? Steve Yegge has said in his tech talks on YT that DevOps is often "all ops" and not necessarily involving any dev.
nunez 4y ago
> The Parable of the Sticky Users This was the most unfortunate section of this blog for me. Some of the companies that need SRE the most, and have the money to actually fund this practice, care the least because they are the only game in town. My take on SRE: 1. It is an extremely important and extremely technical practice in desperate need of good non-technical PR (example: most SLOs I've helped craft are guesses from engineering because they couldn't get data from product to support real numbers) 2. It is damn hard to hire for on skills alone (good SREs are good sysadmins _and_ good SWEs, i.e. two hard types of candidates rolled into one) 3. The hires are damned expensive (see [2]), and 4. Ops-aligned BUs that are traditionally given SRE headcount are (still) considered cost centers, which, combined with [1], makes [3] difficult to justify and [2] difficult to obtain as a result.
apple4ever 4y ago
I have never been enamored with SRE culture, and definitely not the name. Reliability isn't the only thing to worry about - as the article sales, latency is just as if not more important. I also say that systems should be responsive, functional, reliable, and secure, and not necessarily in that order.

96 points by zdw 4y ago

SRE has gotten so good that people don't even think about it. It is default expectation that any online service is basically five nine availability or better. If it is worse than five nine is the only time it gets talked about. SRE is really one of the only well solidified parts of tech as far as career and expectations go. The author seems to look at the same info and come to a different conclusion.

4by4by4 4y ago
I’m curious what services you interact with that have five nines availability. Most services that I interact with, AWS APIs as an example, top out at an SLA of four nines for availability.
- dijit 4y ago
  Even Google's own load balancer only offers 4 nines. https://cloud.google.com/compute/sla https://cloud.google.com/compute/sla
  - citizenpaul 4y ago
    OMG you guys are missing the point. THe article is not talking about what is a good SLA level. Its that SRE "could be so much more" My point is no it cannot. SRE is now a commodity, the expectation is that the system is up pretty much always. Even two nine is only an hour or so a year downtime. Jeeze
- rwiggins 4y ago
  There is some nuance there. Because SLA violations typically have real-money impact, they're often backed by internal objectives (SLOs) that are quite a bit tighter. So if an API promises four nines of availability in its SLA, there is probably an internal target that's at least four-and-a-half nines, if not five. Which is to say, I would not be surprised if many APIs with a four-nines SLA were actually closer to five nines. From a planning perspective, you should probably stick to the public guarantees (i.e. the SLA), though. Although things can always go horribly wrong, like in the big Atlassian deletion snafu recently.
  - citizenpaul 4y ago
    I'm saying SRE/SLA has become a commodity. No one thinks about it. They just expect the systems to be up. The article is about how SRE can be more than "downtime" I propose it cannot and they are wrong.
- citizenpaul 4y ago
  I guess I made a poor point. I am saying that SRE has basically solidified into the expectation of the high 99 percentile as the bottom level of expected downtime. Even two nine is like 1hr per year. There really isn't much to discuss in the industry of SRE. You basically have to be flawless. Its like the janitor. Miss the garbage pickup one day and heads roll. If you have SRE's and the service is down you are done. There is no longer any nuance which is why the CEO guy in the OP doesn't think about SRE anymore than the garbage collection.
kodah 4y ago
In my experience as a SRE, most teams/companies claiming above three nines are not measuring correctly or lying.
- harpratap 4y ago
  Depends a lot on scale too. 3 nines while serving 1M RPS is quite a different challenge from 3 nines at 1 RPS
- pojzon 4y ago
  My previous company SLA was for 99.999% uptime which was built on top of a service with 99.9% uptime without anything additional in between. SLAs in most cases are bullshit and if anything really bad happens (Atlassian) they are just prepared to pay out class lawsuit.

benlivengood 4y ago

A lot of the metrics for outage impact are available, and if you really want to Do Science you can use whatever A/B Canary/experiment framework you have to simulate outages (which, arguably, are not perfect because of down-detectors) on users to understand their behavior. I tend to agree that most people don't care about transient downtime beyond 2 or 3 nines for reputation. Covid has proven how bad people are at reasoning about 0.5% probabilities so there's essentially no chance that they are noticing (en mass) the difference between 2.5 or 3 nines of reliability. Know your customer, as the article points out; more complex customers will know and measure actual availability because they need it for their own reliability calculations. This brings me to my final guess, that SLOs are primarily useful because they provide a risk model that upstream and downstream consumers can reason about. SLOs are a way of building even larger distributed and interdependent systems that can still be somewhat reliable. Precisely what SLIs are chosen and the limits are a little less important than the fact that they are published publicly. There's an economy of reliability in large systems and like any market information about the market is key to its efficiency. If everyone just trusted that everyone was doing their best at maintaining dependencies most of those beliefs would be very unanchored. A side benefit of SLOs is postmortems and communication; people working on codependent products and systems need to talk to each other from time to time and understand the true nature of the interfaces between their systems. Postmortems and SLO negotiations give a concrete framework to do that within, metrics drive quantitative understanding of the systems instead of handwaving.

LadyCailin 4y ago

I’m still trying to figure out what SRE stands for…

deathanatos 4y ago
Site Reliability Engineer
comprev 4y ago
Something Really Elusive
dijit 4y ago
Sysadmins: Really Expensive
- LilBytes 4y ago
  I'm keeping this too. - also a Sysadmin, really expensive.
andrewxdiamond 4y ago
Google somewhat confusingly uses it for both “site reliability engineer” and “site reliability engineering”. Both the practice and the practitioner
- OJFord 4y ago
  That seems fairly normal? SE & PhD for example.
deleted 4y ago
[deleted]
chrismarlow9 4y ago
Set Real Expectations
chucky_z 4y ago
Principle CJ

mr-ron 4y ago

SRE is a great toolkit of good things that any team can use. Some super amazing chapters I think everyone should read: Eliminating toil: https://sre.google/sre-book/eliminating-toil/ https://sre.google/sre-book/eliminating-toil/ Every team tech and non tech everywhere should read this Incident Management: https://sre.google/workbook/incident-response/ https://sre.google/workbook/incident-response/ Post Mortems: https://sre.google/sre-book/postmortem-culture/ https://sre.google/sre-book/postmortem-culture/ These 2 chapters are necessary reading. SLOS: https://sre.google/sre-book/service-level-objectives/ https://sre.google/sre-book/service-level-objectives/ Disclaimer: I think the SLO chapter is good to know the theory around, but SLOs specifically Ive used very few times.

sharadov 4y ago
A good read but a lot of topics in that book apply to google scale companies only. 95% of companies will never reach that scale or deal with those issues.
citruscomputing 4y ago
Thank you for posting these, I learned a lot reading them.

throwaway892238 4y ago

I think this is the point where we're getting too serious about the discipline. We need to be a little more grounded in reality. SRE is basically a highly polished, best-practice, telemetry-and-automation-focused sysadmin. And a sysadmin doesn't really have a say in the business. They can make suggestions, like error budgets. But what about sales contracts? The contracts define what kind of guarantees you have to give and on what timeline. And what about customer experience? How is SRE supposed to quantify if the users are happy if they can't tell the UX and customer support how to build the product to better collect user happiness? Or tell the devs how to design their apps to fail less often? If you keep optimizing and focusing on all the ways to improve satisfaction, you literally end up an executive director of product. If we really want to make products that users are happy with, we need to teach executives how to achieve it, not SREs. SREs shouldn't be driving the bus from the back row. They should be rotating the tires and checking the fuel and oil gauges and telling the driver when to stop for gas. Let's teach executives how to set everyone up to succeed, and then SRE doesn't have to get all existential just to keep the bus running.

dilyevsky 4y ago
> How is SRE supposed to quantify if the users are happy They can easily tell if the users aren’t happy which is usually same time their pagers go off
spondyl 4y ago
> If we really want to make products that users are happy with, we need to teach executives how to achieve it, not SREs. SREs shouldn't be driving the bus from the back row. They should be rotating the tires and checking the fuel and oil gauges and telling the driver when to stop for gas. To extend on this analogy a bit, the theory is that you might have business OKRs that are driven by SLOs. For example, you might have a wishy washy goal of having happy customers. If you're in the shipping business, one aspect of being happy with the service would be to receive deliveries on time (or earlier!) so you might set an OKR where the objective is "Deliveries are made in a timely manner" where key results might be the number of deliveries made on time and inversely, the number of deliveries that are not returned. By pitting those two metrics (speed and state of the delivery) together, you can ideally capture that an increase in speedy deliveries is not just achieved by people throwing packages out of moving vehicles as one or the other will decline. Of course, simply setting these metrics doesn't mean very much but this is where we can drop down to a lower level and make use of SLOs. An SLO is just some number with a target after all, it doesn't have to be technical by any means. So, how can we help drive our goal of timely deliveries? Well, we might set an OKR that says something like "99% of vehicles should have their oil checked every 3 months". On the face of it, that has nothing at all to do with vehicle deliveries but inspecting historical data might determine that failed deliveries were often correlated with poorly serviced vehicles so by ensuring they're serviced, we can indirectly improve customer happiness by something that seems entirely unrelated. This is all in theory of course and assumes your business is able to communicate across layers and share some sort of overall vision which is very hard! It also assumes someone (in the middle?) is able to bridge the gap to co-locate SLOs with OKRs and what not. Anyway, that's how I understand the "dream state" but have I ever seen it actually accomplished? I can't say I have!

aluminussoma 4y ago

SRE has an identity crisis. It means too many things to too many people. In some companies, it is an operational help desk. In a few companies, SRE is rumored to be a different aspect of software engineering. Pay is a rough signal on what a company expects from SRE. If SREs get paid as much or more than Software Engineers, it is a good sign. Yet even that signal is not accurate. I know highly paid SREs stuck in Ops Hell. You could not pay me enough money to do a job like that. It is also not sustainable because eventually the company will fire you and replace you with a lower paid employee who can do that job just as well. I strongly oppose organizations that have SRE separate from Software Engineers. I strongly believe that SREs must be fully integrated into Software Engineering teams (aka the embedded model), with the same reporting lines as their product colleagues. Keeping SRE organizationally separate has done a great disservice to SRE. SRE 2.0 needs to tear down the mini empires that SRE 1.0 built and get back to software engineering.

dilyevsky 4y ago
Imo embedded mode is no better - usually just a token ops person that everyone on the team throws over the fence to. I’d say better signal is if sre has its own separate org structure which roots at the cto/vp of eng level. Soon as you have sres reporting to product eng managers it’s beginning of an end.
- aluminussoma 4y ago
  In the several examples of separate SRE orgs that I've seen, SRE builds their own little empires with no alignment to the product. A cynic would guess that the goal of SRE management is to hire enough SREs so there can be senior managers and directors and eventually a VP with pay packages to boot. They come to Product from the mountain-top with their holy scriptures (the Google SRE book), prescribing surface solutions to problems with deep roots (pray it away with SLOs). SRE 2.0 needs to make the embedded model work.
  - joshuamorton 4y ago
    The entire point of the unembedded model is to create the management structure that isn't strictly beholden to the whims of product. An embedded model can't work unless the company leadership wants to make it work, and product leadership won't care enough (as demonstrated by: everything pre-sre). An embedded model is designed to fail, because the incentives are misaligned. Like, what your asking is "hey, random engineer or engineering manager who is an 'SRE', make the VP of product prioritize your work and staff your team correctly!". That's not something anyone can fix, no matter how good, unless the management is willing to listen, and continue to listen and prioritize appropriately.

flurie 4y ago

I remember a common, puerile early adulthood "game" in which one would add the words "in bed" to the text of a fortune cookie. I feel like proclamations about the industry by people who were and are at Google need a similar party trick: add "at Google" to the end of each sentence. SRE is Google's answer to Google's problem; it says so right in the first revelatory header of the introductory chapter of the book: "Google’s Approach to Service Management: Site Reliability Engineering". Inherent in the text of technical books that reveal what people did is an attempt to persuade the reader why they should do it, too. In some ways this reveals the central, myopic conceit at the heart of Google: if you don't have Google's problems, you should. And I'm not sure why anyone who isn't Google should just accept this premise. That's not to say that the premise is unacceptable. I'm sure that there are people who want to participate in a thing that looks like Google, except that they'd rather start it from scratch than join the one that already exists. Perhaps those people want to have Google's problems. Now we have a piece, expanded from a talk, by one of the editors of the books that persuaded organizations that, if only they would squint their eyes a bit and turn their head, they could make the problems of their organization orders of magnitude smaller by so many metrics look like the problems of Google. And now, perhaps, Google's answer to Google's problem wasn't quite right, because it didn't involve...something? It could be philosophy or human-centered design, and I wouldn't be surprised to find Google bereft of either, but it certainly isn't going to be found in a paper by Google (ostensibly, a Google answer to a Google question) that predates the first book by nearly a decade. It isn't going to be found in people for whom these things are afterthoughts rather than guiding principles. Instead, shouldn't we be asking whether we let SRE as a Google answer to a Google problem be a solution outside of Google in the first place?

coward123 4y ago
Thank you for this - you put into words very well all of the problems I had with that book. In fact, I found myself very frustrated with the authors, wondering if they had any familiarity with all the rest of the decades of work that had been done in related fields outside Google.

luxuryballs 4y ago

After reading these comments I have even less of a clue what SRE actually is…

1970-01-01 4y ago
It's a telltale sign of a naive blogger. If you think anyone outside your profession will read your writing (or code!), always expand acronyms on first use. I'll fix this one for everyone: It’s natural, in the swirling chaos of the past few years, to take a step back and wonder just where everything is going. Though I’ve certainly been doing my fair share of that, I’ve been also thinking about defining precisely where things are. Answering that question for SITE RELIABILITY ENGINEERING (SRE) is not easy. But your question was probably about the job, not the acronym. Carry on.

chazu 4y ago

90% of SREs and SRE managers haven't read the SRE book(s). 99.9% of folks hiring SREs or starting SRE teams haven't read the SRE book. The SRE book (and its sequels) say quite plainly what SRE is and isn't. They also say that not every org is going to be exactly like google so no, "we're not google" isn't an excuse. the E in SRE is for engineering. As in software engineering. SREs are software engineers. Or should be. If your SREs don't know basic SWE principles, they're not SREs. If your org isn't applying software engineering principles to minimizing operational complexity at scale, your org isn't doing SRE. I'm constantly shocked by how hard these things are to grasp, even for most SREs. If the problems I (occasionally) get to solve weren't more interesting than most regular product work, I'd get out of "SRE" entirely.

throwaway892238 4y ago
I think this myth exists because Google was (is?) famously obsessed with SWE. But if you actually read the SRE books and look at the actual discipline of SRE ("what's the difference between SWE and SRE?"), SRE is quite blatantly just operations management. The website is a power plant, and the SRE runs the power plant. You don't build parts to run a power plant, you use software (as in manipulate/control/operate) to run it. You act quickly when the numbers go out of line, you write reports and control how much power is going in and out, respond to surges and dips, etc. For whatever reason, Google decided to tell people that the same person who's building the klaxon and the concrete wall and the pipes for the power plant, and the person who's operating the power plant, are one in the same. But that's clearly bunk. Building a part and running a system are completely different disciplines, and anyone who does both will only be half good at both. Humans are shit at multitasking and there's few true polymaths out there. Show me a master programmer and I'll show you an amateur woodworker. I also don't believe software engineering principles will help you reduce operational complexity. If anything, software engineering tends to either make things either inefficient or subtly complicated. Reducing operational complexity comes from the discipline of operations, which isn't engineering. Non-tech companies have known about these distinctions for like a hundred years. Deming applied scientific rigor and analysis to come up with better practices, but he didn't have to design any widgets to do it.
- 4y ago

krishnadaskp23 4y ago

In a purist approach there are clear guidelines for DevOps and SRE but these are seldom followed. In all the cases I have seen personally, for a developer both DevOps and SRE are support teams. Major cultural changes in an organisation especially at the development end is very hard to make. What has worked for me is having Ops people with some development experience work as DevOps Engineers who work with Dev team for doing whatever is required for making development and release easier. And the SRE team work with development teams for doing whatever is required for making production support easier. Once we have a fairly seamless and well defined pipeline between committing code to supporting in production, the DevOps and SRE Engineers can merge with Dev or Production Support teams depending on their skill set

coward123 4y ago

From 2008-2018, I went all over the world handling incidents and attempting to teach companies how to do the kinds of things the author is talking about. It was multi-diciplinary and very demanding, both in terms of technical and organizational (political and communication) skills. One of the core observations I made was that few companies were willing to actually get better. Rather than figure out what skills they lacked organizationally, or how to improve their processes, or how to align their goals and teams, or any of the lessons they could have learned, it was way easier and cheaper to just keep on status quo. SRE as a practice, a position, a skill - whatever it could have been, it instead failed to really exist apart from maybe a hero here and there who chose to care about it. SLOs (or whatever name might have been used) did not exist to actually measure or inform anything. They were a kind of placebo - come up with something that looks measurable, find a way to make sure we always score well, and make sure there is a wall between these metrics and anyone's compensation / job performance. An example of this, at a very prominent health insurer. I identified issues that could have been solved for, let's say well under $10MM. Rather than do any of that work, the company threw about $50MM in new hardware at the problem. In another case, at a national name Pharma company, about every two years I went back out to do the same basic consulting engagement, because they lacked the leadership to develop the skills in house or to figure out how to align their teams with the work they actually did. Still another case (prominent bank), they had this serious problem that blocked their commercial lenders from doing their job... they finally called me in after about a year of fooling around, I got the problem fixed, rolled out a bunch of tooling and training, but I guarantee they never used any of it... again - problem outright blocked key bank personnel from doing their job and yet it took them a year to do anything about it. So... I guess what I'm saying is that yes SRE work can be fun and rewarding, but it's just not something most companies have the maturity to really do. I read the Google SRE book way back when, but I didn't find it particularly insightful. Likewise, I'm not really sure what to take away from this blog - for me, it doesn't answer any questions about a clear path forward and leads me to believe not much has changed since I left that kind of work years ago.

mycentstoo 4y ago

Somewhat relatedly, I converted from SWE to SRE for three reasons. 1. SWE interviews are torture and no matter what I accomplish at my job, the next job I have to re-prove everything all over again. SRE interviews do not typically have this style of interview (I've heard Meta and Alphabet do however). 2. SRE focuses on lower level abstractions and therefore seems less prone to product oversight which is tiresome. SRE work is typically longer lived and deadlines less tight. 3. SRE jobs seem even more remote friendly than traditional SWE roles because of reason 2. I'm much better as a SWE but I'm paid more as an SRE and have much more freedom. I don't actually like ops work that much but I'll take that trade off to guarantee the other three advantages above. All that said, in my short time as an SRE I've noticed that it's very much in an identity crises because there's not really a standard job. Infrastructure is very different company to company and some companies expect SREs to be able to code as well as SWEs whereas others don't expect you know how to code at all. In my experience, an SRE is someone who's principal goal is to maintain infrastructure and ensure code is able to be developed, built and shipped. In the "better" SRE roles, SREs are able to build platform tooling rather than just ops.

pojzon 4y ago
> principal goal is to maintain infrastructure and ensure code is able to be developed, built and shipped From the SRE vs DevOps definition I just read from the top of google -> this description sounds more like a regular DevOps role instead of SRE. But IMHO, SRE is just a Google way of naming regular SystemAdmin DevOps role. I see no difference. I did exactly the same tasks as an SRE and as a Senior DevOps. Could anyone with 20+ years of experience explain why exactly we still distinguish those ? SRE is supposed to be a superset of DevOps but how?
- AlchemistCamp 4y ago
  > SRE is supposed to be a superset of DevOps but how? Steve Yegge has said in his tech talks on YT that DevOps is often "all ops" and not necessarily involving any dev.

nunez 4y ago

> The Parable of the Sticky Users This was the most unfortunate section of this blog for me. Some of the companies that need SRE the most, and have the money to actually fund this practice, care the least because they are the only game in town. My take on SRE: 1. It is an extremely important and extremely technical practice in desperate need of good non-technical PR (example: most SLOs I've helped craft are guesses from engineering because they couldn't get data from product to support real numbers) 2. It is damn hard to hire for on skills alone (good SREs are good sysadmins _and_ good SWEs, i.e. two hard types of candidates rolled into one) 3. The hires are damned expensive (see [2]), and 4. Ops-aligned BUs that are traditionally given SRE headcount are (still) considered cost centers, which, combined with [1], makes [3] difficult to justify and [2] difficult to obtain as a result.

apple4ever 4y ago

I have never been enamored with SRE culture, and definitely not the name. Reliability isn't the only thing to worry about - as the article sales, latency is just as if not more important. I also say that systems should be responsive, functional, reliable, and secure, and not necessarily in that order.