5 ms·

Is it just me or do I keep seeing this 'SRE' stuff being hyped up as the next big thing everywhere on Twitter/HN/Reddit etc, just because Google invented it and

Is it just me or do I keep seeing this 'SRE' stuff being hyped up as the next big thing everywhere on Twitter/HN/Reddit etc, just because Google invented it and now everyone wants to behave as if they have Google's problems. I am genuinely curious why this is the case, am I missing something here? Even my local grocer's website somehow needs Kubernetes skills all of a sudden, not kidding at all.

WrtCdEvrydy 6y ago
It's cargo culting of the highest order. The truth is, noone knows we deployed the thing using caprover by just wrapping the app in a docker container.
yters 6y ago
SRE is orthogonal to kubernetes. It is a quantitative approach to balancing customer demands with operational optimization, the latter which may be addressed with kubernetes, though not solely or even necessarily in the general case.
vergessenmir 6y ago
I'm fairly sure the SRE books by Google don't mention kubernetes once, since it's a technology agnostic set of practices [1]. I would also say that the practices are independent of scale. Google didn't necessarily invent it inasmuch as they coined a term for keeping systems running. [1] https://sre.google/books/ https://sre.google/books/
- dijit 6y ago
  They do mention Borg though.
- vergessenmir 6y ago
  I see a lot of comments mentioning scale, and dedicated SRE teams and IaaC and thought this might be helpful. I believe distinguishing the engineering from the engineer in SRE is important. The engineering side is essential but a dedicated full-time engineer may not be crucial. The amount of engineering required depends primarily on the delta between your required availability (captured in SLOs) and your current availability. The higher your availability needs are the more engineering effort you'll need as the code travels from commit to production because availability doesn't start and end in production systems. The framework exposes or makes explicit considerations to achieve that goal. Oftentimes it comes down to a choice in balancing the cost between human time (or manual effort) and automation to replace it,and out of that the technology to achieve it. You may very well find that your availability requirements are so low that you can get by with a human pressing a button every few hours, and that this very manual approach is acceptable. Going through the process will answer most of the questions on scale, whether a dedicated Site Reliability Engineer is required and so on. The SRE framework scales even if the system themselves don't require sizable scale.
rachelbythebay 6y ago
Those damn books. They keep showing up and making life miserable for people. "How to Cargo Cult".
- herodoturtle 6y ago
  Side note - for those HN readers like me who had never heard this term "Cargo Cult" before, here is a quick Wikipedia definition: > A cargo cult is a millenarian belief system in which adherents perform rituals which they believe will cause a more technologically advanced society to deliver goods. These cults were first described in Melanesia in the wake of contact with allied military forces during the Second World War. Personally, I'm still trying to wrap my mind around the implied context here. Sorry it's slightly off topic - but perhaps someone can clarify? Thanks :)
  - cytzol 6y ago
    It's a bit of a weird one! I think the most famous example of the term comes from Richard Feynman, in his essay "Cargo Cult Science" (http://calteches.library.caltech.edu/51/2/CargoCult.htm http://calteches.library.caltech.edu/51/2/CargoCult.htm): > In the South Seas there is a Cargo Cult of people. During the war they saw airplanes land with lots of good materials, and they want the same thing to happen now. So they’ve arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas—he’s the controller—and they wait for the airplanes to land. They’re doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn’t work. No airplanes land. It's about copying others to achieve a goal — but because you aren't copying the motives, only the visible methods, you don't succeed. In this case, it's about copying the engineering practices of big organisations without being a big organisation yourself, in the hope that you become one. Hope this helps.
    - herodoturtle 6y ago
      That definitely helps - I understand the reference perfectly now - thanks :)
lordnacho 6y ago
The main thing I take from it isn't actually the scaling side. If you need to scale it's probably when you're getting a bunch of money to do it, so you'll find a way. The thing that caught my eye is infrastructure-as-code. It's massively useful to have your setup in version control, regardless of how big you are. A few years ago I was managing code separately to eg network configs, typing commands into routers, that kind of thing. It's a natural step to have all the infra in a form where you can see changes as well as reliably deploy.
- twic 6y ago
  > The thing that caught my eye is infrastructure-as-code I thought that's what DevOps was? My understanding - from one conversation with an SRE one time over a beer! - was that an SRE's job is not to keep things running and scaling, but to work with development teams to help them keep their things running and scaling. So the key idea is that SREs get leverage by having all other developers work on operability.
  - lordnacho 6y ago
    Yeah, but everyone seems to be talking about the scale side more than the changes-being-accountable side.
thewarrior 6y ago
Your local grocers website doesn’t need Kubernetes. However what’s happening is that the way we deploy software is being automated and standardized. The dream scenario we’re trying to achieve is to tell the system here’s my application here are are some servers now go deploy it and keep it up and running. This actually does work even in the face of failures. It’s just complex because it’s new and solving a very hard problem. I would considers these systems as a form of AI. When you manage tens of thousands of servers these investments make sense. And once these systems stabilize they will be usable without much difficulty. Which means in five years you’ll have an in demand skill if you learn it now.
Juliate 6y ago
It's as much "the next big thing" as "Cloud" has been since 2006 or so: as basic a valid option to organize for substantial scale operations as for the past 15 years. "SRE" may be a Google-initiated label, the practices behind predate it (first sight I've seen was Flickr in 2008).
- zeckalpha 6y ago
  It predates Google, though in a manufacturing or janitorial sense.
joshb888 6y ago
SRE hasn't really got anything to do with scale. Just because Google literally wrote the book on it, doesn't mean you need Google sized problems, to adopt the philosophy. My take away is the switch to focussing on reliability from a customer perspective, infrastructure failure is inevitable, impact to the end user doesn't have to be. To understand how any failures do impact your users, it's usually very important to pick some good metrics and make informed decisions on what level of reliability you want and how you are doing. That then feeds decisions on where you may want to improve your own tooling/processes to improve these metrics/end user reliability. The dev focus is because you need the skills to automate and fill gaps in your tooling and you need to be able to understand the software you want to measure.
- dagss 6y ago
  Scale still has something to do with it. If you are small scale, spending an engineer month to think about and make informed decisions and implement it may be more expensive than just accepting whatever downtime happens along the road. I think premature optimization in uptime is common. Yes, being down a bit is costly, but may still be cheaper than a development team being obsessed about adding 9-s to the service level %. OTOH, obviously for Google, downtime must be minimized at almost any cost. It is all about scale; but scale of budget, not traffic.
  - remus 6y ago
    > Scale still has something to do with it. If you are small scale, spending an engineer month to think about and make informed decisions and implement it may be more expensive than just accepting whatever downtime happens along the road. This is discussed pretty explicitly in the SRE book, in particular the idea of an error budget. Obviously they use some google services as examples, but the key message I got from it was "think about how much downtime is acceptable to you and work around that". > OTOH, obviously for Google, downtime must be minimized at almost any cost. Interestingly the SRE book pretty explicitly says that this is a poor goal for pretty much everyone. The cost of chasing more 9s goes up exponentially, while for most users whether your service is 99.999% available or 99.9999% available makes no difference, because 0.1% of the time their shitty router crashes and they have to restart it. Better to pick a level of availability that strikes a balance between cost and user experience then work towards that.
deepstack 6y ago
Yes technology is like fashion. All these SRE thing are just common sense that sysadmin/DevOP people know how to do. Many things are not needed. Do we really need docker and the cloud provider can just take a snapshot? It really comes down to basically systemadmin skills.
lrem 6y ago
As an actual full-time senior SRE: telling the grocer's website dev team they don't need Kubernetes is pretty much what the point of SRE is.
100011_100001 6y ago
It's human nature. People want to do what the "pros" are doing disregarding skill level, needs, experience. You can see it in the gym where people emulate their favorite's bodybuilders routine even though they weigh 110lbs soaking wet and have less testosterone than a mouse. SRE is the same. DevOps, microservices fall in the same bucket. The irony of course is big companies, the ones that really need DevOps, microservices architecture etc sometimes have the hardest time to adopt new ideas to leverage them. Case in point, I work in a large FinTech company, we are talking multi-billions. 2015 small group wanted DevOps, I became that, only now, 5 years later the greater part of the company is starting to adopt it. I'm still giving presentations on VM vs Docker differences. While at the same time the team that wanted DevOps deploys to production Openshift clusters multiple times a day.
dvtrn 6y ago
Even my local grocer's website somehow needs Kubernetes skills all of a sudden, not kidding at all. Maybe I’m hurting myself, maybe I’m helping, who knows, but it’s become a recent tactic of mine in job interviews when a company demands Kubernetes skills (which isn’t to say I don’t have them, I’m just not as effective as I’d like to be...YET) to look dead in the camera and ask: “How many nodes are in your production cluster right now, how many are control plane nodes and when was the last time two of them failed?” Just to see if they’re putting in Kubernetes because they actually need it, or if they’re just faffing about with new tools because new tools and are struggling to even keep the lights on for their 3 microservices. If there’s a better question to reveal this, I’ve got an open mind about deploying it to production.
- tpxl 6y ago
  > “How many nodes are in your production cluster right now, how many are control plane nodes and when was the last time two of them failed?” What is a good answer and a bad answer for this, and why do two control plane nodes have to fail?
  - dvtrn 6y ago
    I don't know if the number needs to necessarily be two, my intention is to fish for an answer that isn't so much prescriptive but descriptive of their infrastructure needs and goals. But a good answer would probably involve articulating some kind of genuine need that can't be solved with something else, I'd hope the answer involves some candor that the company looked at their options, vetted out that K8s was what they needed, why K8s is going to/has already solve(d) them over something (anything) else (nomad, or others but not necessarily limited to nomad), and if the interviewer is so inclined, disclosures of what they've learned trying to bring k8s into the house. This last bit may be a follow up question from me "what have you learned so far implementing k8s? What would you have done differently if you knew better at the time?" etc. What I'm looking for is a determination if the team thinks through the problem deliberately or if they're simply throwing their stuff at kubernetes because someone who no longer works at the company sold the leaders on kubernetes and no one asked any questions as the org jumped into those waters. Mind you, this part is important: I am NOT anti-kubernetes, I am anti-foisting-kubernetes-on-platforms-just-because-its-hot-and-exciting. If a team wants to R&D it somewhere, prove out it's merit, create an MVP internally and can show the effort is worth undertaking to redeploy services on k8s, awesome. My lived experience has been more teams than not are doing it because a thought leader somewhere wanted to pad their resumes, and then left the company, now the rest of Ops is paying down technical debt they never took out loans for.
Cthulhu_ 6y ago
I think the mindset is important (think the chapters from the SRE book), but as you imply, it's a problem you and your company WISH you had, it's the kind of focus on quality and uptime you WISH you had. Therefore, while I think knowledge of SRE is important, at the same time you need to fear cargo cult thinking. A (imo) worse trend I'm seeing is the focus on microservices. I'm in the Golang slack channel, and on a fairly regular basis a fairly inexperienced dev comes in and asks about microservices - implying they're solo developers that need to set up a microservices architecture. The cargo cult is strong, and people (in general) underestimate the amount of work involved in setting up a proper distributed system (microservices on kubernetes with SRE or otherwise). And people underestimate just how much software and engineers a company like Google has and maintains. I mean my biggest software project(s) had at best three dozen engineers. Those + additional staff already filled a building. Someone decided that we should do microservices, but even then I thought it was overkill, it's amounts of traffic we WISH we had. And of course the split was wrong, it was a rough split on domains instead of something more solid (e.g. we had a separate 'address book' service, and an 'order history' service that was separate from the actual 'order' service. Which was a gateway to an older Java application, which itself was a gateway to an even older mainframe. I mean those two needed the attention, the microservices (which were written in Scala because otherwise they would be too boring) were just self-gratification. I have no clue if they still have them. Part of me is thinking of going back there (I was working there as a contractor at the time).
sdevonoes 6y ago
Google et al. invent new problems (e.g., Kubernetes) and solve them accordingly (SREs). Quite clever, if you think about it but depressing nonetheless (and a waste of time and brains).

WrtCdEvrydy 6y ago

It's cargo culting of the highest order. The truth is, noone knows we deployed the thing using caprover by just wrapping the app in a docker container.

yters 6y ago

SRE is orthogonal to kubernetes. It is a quantitative approach to balancing customer demands with operational optimization, the latter which may be addressed with kubernetes, though not solely or even necessarily in the general case.

vergessenmir 6y ago

I'm fairly sure the SRE books by Google don't mention kubernetes once, since it's a technology agnostic set of practices [1]. I would also say that the practices are independent of scale. Google didn't necessarily invent it inasmuch as they coined a term for keeping systems running. [1] https://sre.google/books/ https://sre.google/books/

dijit 6y ago
They do mention Borg though.
vergessenmir 6y ago
I see a lot of comments mentioning scale, and dedicated SRE teams and IaaC and thought this might be helpful. I believe distinguishing the engineering from the engineer in SRE is important. The engineering side is essential but a dedicated full-time engineer may not be crucial. The amount of engineering required depends primarily on the delta between your required availability (captured in SLOs) and your current availability. The higher your availability needs are the more engineering effort you'll need as the code travels from commit to production because availability doesn't start and end in production systems. The framework exposes or makes explicit considerations to achieve that goal. Oftentimes it comes down to a choice in balancing the cost between human time (or manual effort) and automation to replace it,and out of that the technology to achieve it. You may very well find that your availability requirements are so low that you can get by with a human pressing a button every few hours, and that this very manual approach is acceptable. Going through the process will answer most of the questions on scale, whether a dedicated Site Reliability Engineer is required and so on. The SRE framework scales even if the system themselves don't require sizable scale.

rachelbythebay 6y ago

Those damn books. They keep showing up and making life miserable for people. "How to Cargo Cult".

herodoturtle 6y ago
Side note - for those HN readers like me who had never heard this term "Cargo Cult" before, here is a quick Wikipedia definition: > A cargo cult is a millenarian belief system in which adherents perform rituals which they believe will cause a more technologically advanced society to deliver goods. These cults were first described in Melanesia in the wake of contact with allied military forces during the Second World War. Personally, I'm still trying to wrap my mind around the implied context here. Sorry it's slightly off topic - but perhaps someone can clarify? Thanks :)
- cytzol 6y ago
  It's a bit of a weird one! I think the most famous example of the term comes from Richard Feynman, in his essay "Cargo Cult Science" (http://calteches.library.caltech.edu/51/2/CargoCult.htm http://calteches.library.caltech.edu/51/2/CargoCult.htm): > In the South Seas there is a Cargo Cult of people. During the war they saw airplanes land with lots of good materials, and they want the same thing to happen now. So they’ve arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas—he’s the controller—and they wait for the airplanes to land. They’re doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn’t work. No airplanes land. It's about copying others to achieve a goal — but because you aren't copying the motives, only the visible methods, you don't succeed. In this case, it's about copying the engineering practices of big organisations without being a big organisation yourself, in the hope that you become one. Hope this helps.
  - herodoturtle 6y ago
    That definitely helps - I understand the reference perfectly now - thanks :)

lordnacho 6y ago

The main thing I take from it isn't actually the scaling side. If you need to scale it's probably when you're getting a bunch of money to do it, so you'll find a way. The thing that caught my eye is infrastructure-as-code. It's massively useful to have your setup in version control, regardless of how big you are. A few years ago I was managing code separately to eg network configs, typing commands into routers, that kind of thing. It's a natural step to have all the infra in a form where you can see changes as well as reliably deploy.

twic 6y ago
> The thing that caught my eye is infrastructure-as-code I thought that's what DevOps was? My understanding - from one conversation with an SRE one time over a beer! - was that an SRE's job is not to keep things running and scaling, but to work with development teams to help them keep their things running and scaling. So the key idea is that SREs get leverage by having all other developers work on operability.
- lordnacho 6y ago
  Yeah, but everyone seems to be talking about the scale side more than the changes-being-accountable side.

thewarrior 6y ago

Your local grocers website doesn’t need Kubernetes. However what’s happening is that the way we deploy software is being automated and standardized. The dream scenario we’re trying to achieve is to tell the system here’s my application here are are some servers now go deploy it and keep it up and running. This actually does work even in the face of failures. It’s just complex because it’s new and solving a very hard problem. I would considers these systems as a form of AI. When you manage tens of thousands of servers these investments make sense. And once these systems stabilize they will be usable without much difficulty. Which means in five years you’ll have an in demand skill if you learn it now.

Juliate 6y ago

It's as much "the next big thing" as "Cloud" has been since 2006 or so: as basic a valid option to organize for substantial scale operations as for the past 15 years. "SRE" may be a Google-initiated label, the practices behind predate it (first sight I've seen was Flickr in 2008).

zeckalpha 6y ago
It predates Google, though in a manufacturing or janitorial sense.

joshb888 6y ago

SRE hasn't really got anything to do with scale. Just because Google literally wrote the book on it, doesn't mean you need Google sized problems, to adopt the philosophy. My take away is the switch to focussing on reliability from a customer perspective, infrastructure failure is inevitable, impact to the end user doesn't have to be. To understand how any failures do impact your users, it's usually very important to pick some good metrics and make informed decisions on what level of reliability you want and how you are doing. That then feeds decisions on where you may want to improve your own tooling/processes to improve these metrics/end user reliability. The dev focus is because you need the skills to automate and fill gaps in your tooling and you need to be able to understand the software you want to measure.

dagss 6y ago
Scale still has something to do with it. If you are small scale, spending an engineer month to think about and make informed decisions and implement it may be more expensive than just accepting whatever downtime happens along the road. I think premature optimization in uptime is common. Yes, being down a bit is costly, but may still be cheaper than a development team being obsessed about adding 9-s to the service level %. OTOH, obviously for Google, downtime must be minimized at almost any cost. It is all about scale; but scale of budget, not traffic.
- remus 6y ago
  > Scale still has something to do with it. If you are small scale, spending an engineer month to think about and make informed decisions and implement it may be more expensive than just accepting whatever downtime happens along the road. This is discussed pretty explicitly in the SRE book, in particular the idea of an error budget. Obviously they use some google services as examples, but the key message I got from it was "think about how much downtime is acceptable to you and work around that". > OTOH, obviously for Google, downtime must be minimized at almost any cost. Interestingly the SRE book pretty explicitly says that this is a poor goal for pretty much everyone. The cost of chasing more 9s goes up exponentially, while for most users whether your service is 99.999% available or 99.9999% available makes no difference, because 0.1% of the time their shitty router crashes and they have to restart it. Better to pick a level of availability that strikes a balance between cost and user experience then work towards that.

deepstack 6y ago

Yes technology is like fashion. All these SRE thing are just common sense that sysadmin/DevOP people know how to do. Many things are not needed. Do we really need docker and the cloud provider can just take a snapshot? It really comes down to basically systemadmin skills.

lrem 6y ago

As an actual full-time senior SRE: telling the grocer's website dev team they don't need Kubernetes is pretty much what the point of SRE is.

100011_100001 6y ago

It's human nature. People want to do what the "pros" are doing disregarding skill level, needs, experience. You can see it in the gym where people emulate their favorite's bodybuilders routine even though they weigh 110lbs soaking wet and have less testosterone than a mouse. SRE is the same. DevOps, microservices fall in the same bucket. The irony of course is big companies, the ones that really need DevOps, microservices architecture etc sometimes have the hardest time to adopt new ideas to leverage them. Case in point, I work in a large FinTech company, we are talking multi-billions. 2015 small group wanted DevOps, I became that, only now, 5 years later the greater part of the company is starting to adopt it. I'm still giving presentations on VM vs Docker differences. While at the same time the team that wanted DevOps deploys to production Openshift clusters multiple times a day.

dvtrn 6y ago

Even my local grocer's website somehow needs Kubernetes skills all of a sudden, not kidding at all. Maybe I’m hurting myself, maybe I’m helping, who knows, but it’s become a recent tactic of mine in job interviews when a company demands Kubernetes skills (which isn’t to say I don’t have them, I’m just not as effective as I’d like to be...YET) to look dead in the camera and ask: “How many nodes are in your production cluster right now, how many are control plane nodes and when was the last time two of them failed?” Just to see if they’re putting in Kubernetes because they actually need it, or if they’re just faffing about with new tools because new tools and are struggling to even keep the lights on for their 3 microservices. If there’s a better question to reveal this, I’ve got an open mind about deploying it to production.

tpxl 6y ago
> “How many nodes are in your production cluster right now, how many are control plane nodes and when was the last time two of them failed?” What is a good answer and a bad answer for this, and why do two control plane nodes have to fail?
- dvtrn 6y ago
  I don't know if the number needs to necessarily be two, my intention is to fish for an answer that isn't so much prescriptive but descriptive of their infrastructure needs and goals. But a good answer would probably involve articulating some kind of genuine need that can't be solved with something else, I'd hope the answer involves some candor that the company looked at their options, vetted out that K8s was what they needed, why K8s is going to/has already solve(d) them over something (anything) else (nomad, or others but not necessarily limited to nomad), and if the interviewer is so inclined, disclosures of what they've learned trying to bring k8s into the house. This last bit may be a follow up question from me "what have you learned so far implementing k8s? What would you have done differently if you knew better at the time?" etc. What I'm looking for is a determination if the team thinks through the problem deliberately or if they're simply throwing their stuff at kubernetes because someone who no longer works at the company sold the leaders on kubernetes and no one asked any questions as the org jumped into those waters. Mind you, this part is important: I am NOT anti-kubernetes, I am anti-foisting-kubernetes-on-platforms-just-because-its-hot-and-exciting. If a team wants to R&D it somewhere, prove out it's merit, create an MVP internally and can show the effort is worth undertaking to redeploy services on k8s, awesome. My lived experience has been more teams than not are doing it because a thought leader somewhere wanted to pad their resumes, and then left the company, now the rest of Ops is paying down technical debt they never took out loans for.

Cthulhu_ 6y ago

I think the mindset is important (think the chapters from the SRE book), but as you imply, it's a problem you and your company WISH you had, it's the kind of focus on quality and uptime you WISH you had. Therefore, while I think knowledge of SRE is important, at the same time you need to fear cargo cult thinking. A (imo) worse trend I'm seeing is the focus on microservices. I'm in the Golang slack channel, and on a fairly regular basis a fairly inexperienced dev comes in and asks about microservices - implying they're solo developers that need to set up a microservices architecture. The cargo cult is strong, and people (in general) underestimate the amount of work involved in setting up a proper distributed system (microservices on kubernetes with SRE or otherwise). And people underestimate just how much software and engineers a company like Google has and maintains. I mean my biggest software project(s) had at best three dozen engineers. Those + additional staff already filled a building. Someone decided that we should do microservices, but even then I thought it was overkill, it's amounts of traffic we WISH we had. And of course the split was wrong, it was a rough split on domains instead of something more solid (e.g. we had a separate 'address book' service, and an 'order history' service that was separate from the actual 'order' service. Which was a gateway to an older Java application, which itself was a gateway to an even older mainframe. I mean those two needed the attention, the microservices (which were written in Scala because otherwise they would be too boring) were just self-gratification. I have no clue if they still have them. Part of me is thinking of going back there (I was working there as a contractor at the time).

sdevonoes 6y ago

Google et al. invent new problems (e.g., Kubernetes) and solve them accordingly (SREs). Quite clever, if you think about it but depressing nonetheless (and a waste of time and brains).