7 ms·

"SRE" doesn't seem to mean anything useful any more

91 points by CuriousIndian 2y ago

dijit 2y ago
People need operations staff, people don't like operations staff and keep trying to treat them like developers. But, operations staff do and have always developed software, just internal software for glue or orchestration, and they work differently to regular software developers in that their customers are usually themselves to meet an internal objective of reliability, stability or ease-of-use for developers. It's interesting to me, because I'm a bit longer lived it seems and have been tied to industry for a long time, and I see the treadmill grinding along continuously; Sysadmins (different from system operators) were usually among the most senior developers who ended up knowing how operating systems and compute worked fundamentally. Over time this eroded and eventually you had helpdesk people being labelled as sysadmins. Then "DevOps" emerged as a job title, which meant a dozen things to a dozen people, and the same issue happened, it was the same operational needs and the same operational solutions, just with better tools as the passage of time allowed better tools to exist. Then SRE, which ironically was devised before DevOps was, which did exactly the same thing of trying to turn operations problems into the easier to reason about software development space. But still, it's operations folks, people who are more responsible about an outcome and a continuance than they are about delivering a feature. But Managers genuinely can't reason about anything other than features, so a cost center it becomes and eradication is desired, and the treadmill begins again. So, SRE's, who embody essentially the same characteristics as early sysadmins (but with large budgets and better tools) will eventually become system operators, who will eventually become helpdesk and eventually replaced by some new title that insists that "SREs never wrote code, and this next generation will!". The author is experiencing the exact same thing I did over a decade ago when "sysadmin" became unfashionable and everyone told me that "sysadmins can't code", despite working in teams of sysadmins who wrote precursors to kubernetes/nomad, on bare-metal, in perl on Solaris. Will be interesting to see what the next iteration will be called, the author will just need to alter her vernacular.
- diggan 2y ago
  Such a overused quote, but still, seems to apply to "position names" as well: > There are only two hard problems in computer science...
  - devsda 2y ago
    That's because it's difficult to really define a job in IT with clear description along with set roles and responsibilities. In some orgs, it's the title that dictates what you can/can't do. In some orgs, eventually your title is decided based on your role/responsibilities. In many orgs, your title is just an HR/accounting construct and has no relation to what you do and you do whatever your boss asks you to regardless of what your role on payslip says.
    - theshrike79 2y ago
      I was in a company where we had separate external and internal titles. The external ones were all fancy like "Principal Architect" and all that. The internal ones told people inside the company what you _actually_ did.
      - htrp 2y ago
        was the internal one something like "developer V"?
- benfortuna 2y ago
  >Then "DevOps" emerged as a job title, which meant a dozen things to a dozen people, and the same issue happened, it was the same operational needs and the same operational solutions, just with better tools as the passage of time allowed better tools to exist. DevOps was/is not just better tools, it's the same team that builds the software, operates the software (and underlying infrastructure). Maybe when it became a job title (later) then it just became a rebranding of the ops team.
  - dijit 2y ago
bravetraveler 2y ago
I tire of being the internet janitor, good read
candiddevmike 2y ago
It still means mandatory, potentially grueling, on-call rotations.
- dijit 2y ago
  If you can affect the quality of the on-call, then it's not so bad. If you can't, then that's your ammunition to remove the feature from being "blessed". This is the gift that the SRE book actually gave people. That, and error budgets. It loosened peoples idea of what ops was, before this it was 100% uptime of all services at all times, and ops people being responsible for things that they could not reasonably affect nor push back on without extreme resistance from developers and management.
  - xena 2y ago
    In my experience, being "SRE" means being pager bitch. I was almost completely burned out by being pager bitch for a company that had two warring tribes of PHP and Haskell developers that just sort of threw things over the wall for SRE to keep up. We had no meaningful way to push back on developers because that impacted product release timelines and we can't have that, can we? IDK, part of the reason I got into things like DevRel is that I am good at SRE style work, but I'm just not cut out to be pager bitch like other people are. I now have a 500% premium for any job that involves oncall work. Nobody's taken me up on it, surprisingly.
  - nunez 2y ago
    Do you know of any company that actually _does_ error budgets? Like, they did the statistics on their previous downtime, calculated them and wrote SLOs with them baked in?
dimitar 2y ago
you're going to be the "ops bitch" for the "real" programmers Rachel is spot on about what is often wrong with IT culture; "typecasting" people for someone's convenience or to get a fancy title leads to learned helplessness and dissmissing other people's expertise and interests. I rather we all try to keep things simple and encourage people to be well-rounded engineers.
- buttocks 2y ago
  Yet she keeps calling ops people “monkeys.”
  - magicalhippo 2y ago
    To be fair, many of them do seem to operate on the "op sees, op does" level. Just last week, we had some mails escalating, client had "issues" with their on-prem install of our software, which ran on a dedicated VM. I read the mail thread and turned out the database service "used a lot of memory", and they'd tried rebooting the server several times but it just kept using a lot of memory. So now they had escalated because they couldn't figure out how to "fix" this issue. Of course, this was not an issue. The database is designed to use all available memory by default, and this was a dedicated VM so it didn't affect other services. I've seen many such and other instances over the years. While there are certainly awesome ops people out there, which is always a pleasure to interact with, a significant amount are at a much more basic level.
  - spc476 2y ago
    As a developer, I've had to tell "ops" how syslog() works. Also, as a developer, I've had "ops" bitch because we had different config files for the service. Yes, because the service is geographically redundant, and each site had different IP address! And I, the developer, have maintained, and continue to maintain, the configuration file! Checked into your ops repo. You (ops) have never had to maintain it at all! I've also been on the other side of the wall, as ops, having to tell the developers why the keep seeing zombie processes on the server. Because the developers had no experience with Unix signals and having to wait on child processes. And again, about how syslog() works. It goes both ways.
formerly_proven 2y ago
OP seems to think SRE and DevOps are seen as lowly by management, yet those jobs still exist here, while the "real programmers" were offshored long ago. Good luck getting a job Actually Building Stuff, almost no one does it any more.
- dimitar 2y ago
  They are seen as unfortunate cost centers. They don't add new features that are sold to clients. They don't even fix the bugs the clients care about. Making the others more efficient or preventing catastrophe is invisible work.
PaulHoule 2y ago
It’s a predictable problem. Make up a new and fashionable term and it will bite you in the end. See: euphemism treadmill. It has everything to do with the specifics of ‘site’, ‘reliability’, ‘engineer’ and all the technical and social stuff Rachel talks about but on another level it is about the style of discourse, see ‘non’, ‘fungible’, ‘token’.
joshstrange 2y ago
Honest question, what would you call a role that: * Is on call * Manages internal software (grafana, Prometheus, salt stack, etc) * is the first line of defense for issues in the field, works with support and the engineering team to handle problems * Manages a distributed fleet of servers (uses off the self and/or custom code to do so) * Builds internal tools/automations to improve the reliability of our platform and/or address problems automatically * Pushes us to make software/OS updates I ask because that’s what I’d love to hire. Someone who owns the production environment and has the time/space to focus on improving/maintaining it. “Ops” or “Production engineer” is what I’ve referred to it as in the past but this seems like a good place to ask what others would call it.
- liveoneggs 2y ago
  I've settled on "systems engineer" -- I just try to keep "systems" in the title to imply that my job is largely -- "making many things work together to run the business" I think my current title has the word "cloud" in it or something and my boss rejects (with prejudice) "SRE" or "devops" explicitly to avoid being labeled as operations.
- cedws 2y ago
  >what would you call a role that … Overworked?
- deleted 2y ago
  [deleted]
- Chilinot 2y ago
  "Platform Engineer" has started to grow as a name for pretty much this. An engineer that builds the platform that all things run on. For example, the k8s clusters, the observability stack, has on-call, and builds the internal tools and automations (called the IDP, or internal developer platform).
  - liveoneggs 2y ago
    IDP already has a very important meaning as a part of any platform :)
minkles 2y ago
It never meant anything outside Google or huge orgs. Everyone else is running cargo cults.
- mike_hearn 2y ago
  Moreover, they must be cargo cults. An SRE role was my first job at Google in 2006. It was a product of its time and place. Google could hire the sort of full stack programmer/sysadmin types that the article talks about because they had enough clout and desirability at that time that they could get very skilled programmers to give up programming and do sucky on-call work instead. Even then, just two years after IPO and receiving glowing press everywhere, they struggled with this proposition. A typical strategy (which worked on me) was to promise that whilst they wouldn't offer a job as a developer right away you'd be able to transfer out of SRE after a few years. Another strategy was to build teams that consisted of mixes of sysadmins and developers. In other words, even back then in the most optimal hiring environment possible, there were very few people who were genuinely both developers and sysadmins in one neat package. There was always a clear bias towards one or the other. The hiring process was nightmarish for both sides. SRE had a higher rejection rate than SWE with only about 1% of candidates getting an offer; the accept rate was lower. Hundreds of interviews to get one person through the door. I had to do eight interviews that covered everything from TCP to obscure bash puzzlers, simultaneous equations, C, performance debugging, Python and CS algorithms. All of us had to spend enormous amounts of time running interviews as a consequence, and there was a constant shortage of SREs too, meaning many services were being administered and on-call covered by SWE teams instead. Who then grumbled a lot because that wasn't what they'd signed up for, of course. I think if you're not in a similar situation as Google in 2006 then the SRE role isn't going to make sense for you to try and hire. It's a unicorn role. There just aren't enough people out there who have skills that span such a wide gamut of topics. The fact that she put up an ad for SRE and got "a bunch" of "ops monkeys" is the least surprising thing in the world. Of course that happens. Unless you're willing to interview hundreds of candidates to get one hire, whilst simultaneously running a dragnet recruiting operation, then you'll have to compromise. Even Google had to compromise.
  - mwcampbell 2y ago
deleted 2y ago
[deleted]
ChrisMarshallNY 2y ago
Boy, this sounds familiar. Welcome to "Olds-land," Rachel. Sorry about that. It's almost impossible to be a developer/engineer/opsmonkey, with any varied experience, without running into this. People will always find something in your résumé, that makes them uncomfortable. > they didn't have the usual lists of godawful clown software that most places rattle off that you'd be expected to work with. That could be a summary of all that is amiss with the tech industry, these days.
austin-cheney 2y ago
And that’s why I don’t mention my military background during job interviews. In software world it would likely be lost on them anyways, but really certain words are triggers for unfounded assumptions in an industry dominated by unfounded assumptions.
sgarland 2y ago
It doesn’t mean anything for two reasons: companies have treated it as a catch—all, and there is a glut of people calling themselves SREs who have never operated a server that wasn’t in a cloud. You can learn enough about Linux to be decent at your job on only VMs if you’re dedicated, but I’d argue that until you’ve also dealt with hypervisors, bare metal, and hardware issues, you’re missing some of the picture. “That’s no longer applicable, so why should I care?” Because it pops up everywhere I’ve been. Random build server that everyone forgot about but is critical suddenly shits itself, it’s running some ancient version of Ubuntu, and it’s all hand-rolled. Someone decided to provision a bunch of EC2s with bash, but they made critical errors like not knowing to make a new initramfs after configuring mdadm, so now the RAID disappears on reboots. Understanding the fundamentals has always and will always matter. Anyone telling you differently is selling you something.
- playingalong 2y ago
  > people calling themselves SREs who have never operated a server that wasn’t in a cloud Wait. Since when this is a requirement?
  - sgarland 2y ago
    Because if you’ve never touched hardware, you haven’t seen the full gamut of what can go wrong. I don’t think everyone needs a rack in their house, but you can buy old tower servers for dirt cheap, and learn stuff. In my career so far, the best Ops-adjacent people all had homelabs. I view it as a huge green flag if you have one. Not necessarily a red flag if you don’t, but it warrants some more questions. That’s not to say “I have a server” means you’re great; you could just be running Plex with nothing in IaC, no config management, no monitoring, etc.
    - timacles 2y ago
      Listen man, you’re not wrong, but you’re also Living in the past. It’s no different from saying a real programmer does low level C coding. The field has changed, everything is in the cloud. Yeah most engineers can’t troubleshoot low level stuff, but they rarely Need to. There’s no need to be grumpy about it and just accept their jobs are mainly configuring abstractions rather than being some bare metal guru who knows kernel intricacies. I work with like 15 ops/system engineers. None of them know anything about cpu schedulers and raid arrays. We’re just old man
lr4444lr 2y ago
Ops people are cost centers. They can display their wizardry in blog posts until they are blue in the face, but except for those few companies with an incredibly large moat whose main profit is just raw traffic or high uptime (and how many companies like that can maintain that moat indefinitely anyway), engineers not actually building or directly improving product will always be cost centers. It brings me no joy to say this, as ops people tend to be very smart and cool under pressure, but I never see myself becoming one. High quality software and airtight system integrity seems decreasingly important to people paying salaries and investing money. And the world moves on. I refuse to be seen as a cost center if I can avoid it, and I don't have much sympathy left for otherwise very smart tech people who haven't figured this out yet. If you love the job, do it. If not, transition and don't complain. The people in charge are not going to be persuaded by blog rhetoric. I said what I said.
- loa_in_ 2y ago
  That's a heap of tough love. Ultimately indeed, those who pay, decide, and they're taught to systematically cut costs to improve short term profit above all else.
- sgarland 2y ago
  I’m not scared. We’ve seen the rise of DBRE because all of the ZIRP startups (the ones who survived, anyway) realized they couldn’t just YOLO their way through indefinitely, and that RDBMS is hard, actually. Of course, they couldn’t admit that they needed DBAs, so they invented a title and mixed in SRE so it would seem cool. I might be in a cost center, but companies will cut this cost center to their own demise.
- nunez 2y ago
  Sucks, but true. Which is extremely unfortunate. Bad software is expensive to maintain, bad ops can take down companies, and both can slow the sales cycle and threaten renewals. Engineering as a whole should be treated as a profit center.
- telotortium 2y ago
  Even as a Google SRE who came in with a Systems Engineering focus (i.e., sysadmin), I found it useful to qualify for, and become, a SRE-SWE so that I could also add Software Engineer to my résumé (I wrote enough code that I should get that title). Big and scaling companies still find the combination of sysadmin and software engineering skills extremely valuable, but you do need to emphasize that you are a skilled software engineer as well.
znpy 2y ago
I got the SRE book by google, and read the whole thing, cover to cover. Companies want SRE people but aren't willing to give SRE empowerment and authority. So companies do what companies to: take a regular team of operations people and slap the SRE term on it, and call it a day. And it doesn't work, of course. --- Regarding the empowerment & authority: according to "the book", SREs often play the role of "launch coordination engineers" as in vetting (read: roasting) a service before it goes live and have authority to say "this won't go live, fix this first" and to do so no matter what deadline is going to be missed. Also SRE team have the extreme prerogative to "give back the pager" as in take a service back to the development team and say: it's not stable enough, YOU will be on-call for it until you fix the shit you wrote. These are two emblematic examples, but there are many more in the book. Can you imagine any of the non-google (and non-faang) companies actually doing something like that?
- 28304283409234 2y ago
  As companies did with DevOps.
znpy 2y ago
To me, a SRE is *both* a sysadmin AND a programmer, developer, whatever you want to call it. It's a logical-and, not an XOR. By sysadmin, I mean "runs a mean Unix box, including fixing things and diving deeply when they break", and by the programmer/whatever part of it, I mean "makes stuff come into existence that wasn't there before". The main issue i see with that is the companies usually aren't willing to advertise and pay SRE salaries for actual SRE skills. The skillset described in the above quotes are essentially the skills of an SWE and of a Sysadmin. So essentially you're doing two jobs for one salary. There are people capable of doing two jobs, but you won't find them until you start advertising and paying actual-SRE salaries.
nunez 2y ago
Well, yeah, in a world where corporates don't want to pay for sysadmins that can actually code or giving their devs a pager, you get what you saw: sysadmins and the people that do Jenkins being renamed as the "devops" team in 2014, then the "SRE" team in 2016 or so, then, "platform teams" after the age of Kubernetes in 2018-ish. There are so many companies that have SRE teams despite those teams not maintaining a website!
kkfx 2y ago
Management hate classic operation because it's too powerful, it's the operation part of the nervous system of a company, operation can't be managed nor segmented by the management, as a result many try to do their best to delete sysadmins from the scene. Results are that most end up on someone else system with some form of operation. Meaning the cloud. Now that costs skyrocket, issues piling up, no one seems to been able to create a damn full infra well, the push deflate. We have experienced many similar trends: - full stack virtualization on x86, sold as the future, a super-duper simplification, actually a way to allow third party selling pre-made images to those who have no operation or do not own the bare metal (VPS etc); - when people realize how big the overhead is paravirtualization became an old-new trend, mostly with k8s, now this model start to creak and the push toward owning back the infra and the iron start to be noisy I expect in a decade a mainstream NixOS/Guix System move as we had with Ansible/Salt before, the old CFEngine much before and so on, in 20 years probably companies will own back their machine room with a Plan 9 -alike model to just get redundancies and extra temporary resources.
muppetman 2y ago
Everyone lost respect for SREs when Elon took over Twitter and culled lots of staff. Every SRE in the land lined up to shout from the rooftops "There is no way Twitter can keep working it's going to flame out and fall over and keep crashing" and it didn't, not even once.

91 points by CuriousIndian 2y ago

People need operations staff, people don't like operations staff and keep trying to treat them like developers. But, operations staff do and have always developed software, just internal software for glue or orchestration, and they work differently to regular software developers in that their customers are usually themselves to meet an internal objective of reliability, stability or ease-of-use for developers. It's interesting to me, because I'm a bit longer lived it seems and have been tied to industry for a long time, and I see the treadmill grinding along continuously; Sysadmins (different from system operators) were usually among the most senior developers who ended up knowing how operating systems and compute worked fundamentally. Over time this eroded and eventually you had helpdesk people being labelled as sysadmins. Then "DevOps" emerged as a job title, which meant a dozen things to a dozen people, and the same issue happened, it was the same operational needs and the same operational solutions, just with better tools as the passage of time allowed better tools to exist. Then SRE, which ironically was devised before DevOps was, which did exactly the same thing of trying to turn operations problems into the easier to reason about software development space. But still, it's operations folks, people who are more responsible about an outcome and a continuance than they are about delivering a feature. But Managers genuinely can't reason about anything other than features, so a cost center it becomes and eradication is desired, and the treadmill begins again. So, SRE's, who embody essentially the same characteristics as early sysadmins (but with large budgets and better tools) will eventually become system operators, who will eventually become helpdesk and eventually replaced by some new title that insists that "SREs never wrote code, and this next generation will!". The author is experiencing the exact same thing I did over a decade ago when "sysadmin" became unfashionable and everyone told me that "sysadmins can't code", despite working in teams of sysadmins who wrote precursors to kubernetes/nomad, on bare-metal, in perl on Solaris. Will be interesting to see what the next iteration will be called, the author will just need to alter her vernacular.

diggan 2y ago
Such a overused quote, but still, seems to apply to "position names" as well: > There are only two hard problems in computer science...
- devsda 2y ago
  That's because it's difficult to really define a job in IT with clear description along with set roles and responsibilities. In some orgs, it's the title that dictates what you can/can't do. In some orgs, eventually your title is decided based on your role/responsibilities. In many orgs, your title is just an HR/accounting construct and has no relation to what you do and you do whatever your boss asks you to regardless of what your role on payslip says.
  - theshrike79 2y ago
    I was in a company where we had separate external and internal titles. The external ones were all fancy like "Principal Architect" and all that. The internal ones told people inside the company what you _actually_ did.
    - htrp 2y ago
      was the internal one something like "developer V"?
benfortuna 2y ago
>Then "DevOps" emerged as a job title, which meant a dozen things to a dozen people, and the same issue happened, it was the same operational needs and the same operational solutions, just with better tools as the passage of time allowed better tools to exist. DevOps was/is not just better tools, it's the same team that builds the software, operates the software (and underlying infrastructure). Maybe when it became a job title (later) then it just became a rebranding of the ops team.
- dijit 2y ago

bravetraveler 2y ago

I tire of being the internet janitor, good read

candiddevmike 2y ago

It still means mandatory, potentially grueling, on-call rotations.

dijit 2y ago
If you can affect the quality of the on-call, then it's not so bad. If you can't, then that's your ammunition to remove the feature from being "blessed". This is the gift that the SRE book actually gave people. That, and error budgets. It loosened peoples idea of what ops was, before this it was 100% uptime of all services at all times, and ops people being responsible for things that they could not reasonably affect nor push back on without extreme resistance from developers and management.
- xena 2y ago
  In my experience, being "SRE" means being pager bitch. I was almost completely burned out by being pager bitch for a company that had two warring tribes of PHP and Haskell developers that just sort of threw things over the wall for SRE to keep up. We had no meaningful way to push back on developers because that impacted product release timelines and we can't have that, can we? IDK, part of the reason I got into things like DevRel is that I am good at SRE style work, but I'm just not cut out to be pager bitch like other people are. I now have a 500% premium for any job that involves oncall work. Nobody's taken me up on it, surprisingly.
- nunez 2y ago
  Do you know of any company that actually _does_ error budgets? Like, they did the statistics on their previous downtime, calculated them and wrote SLOs with them baked in?

dimitar 2y ago

you're going to be the "ops bitch" for the "real" programmers Rachel is spot on about what is often wrong with IT culture; "typecasting" people for someone's convenience or to get a fancy title leads to learned helplessness and dissmissing other people's expertise and interests. I rather we all try to keep things simple and encourage people to be well-rounded engineers.

buttocks 2y ago
Yet she keeps calling ops people “monkeys.”
- magicalhippo 2y ago
  To be fair, many of them do seem to operate on the "op sees, op does" level. Just last week, we had some mails escalating, client had "issues" with their on-prem install of our software, which ran on a dedicated VM. I read the mail thread and turned out the database service "used a lot of memory", and they'd tried rebooting the server several times but it just kept using a lot of memory. So now they had escalated because they couldn't figure out how to "fix" this issue. Of course, this was not an issue. The database is designed to use all available memory by default, and this was a dedicated VM so it didn't affect other services. I've seen many such and other instances over the years. While there are certainly awesome ops people out there, which is always a pleasure to interact with, a significant amount are at a much more basic level.
- spc476 2y ago
  As a developer, I've had to tell "ops" how syslog() works. Also, as a developer, I've had "ops" bitch because we had different config files for the service. Yes, because the service is geographically redundant, and each site had different IP address! And I, the developer, have maintained, and continue to maintain, the configuration file! Checked into your ops repo. You (ops) have never had to maintain it at all! I've also been on the other side of the wall, as ops, having to tell the developers why the keep seeing zombie processes on the server. Because the developers had no experience with Unix signals and having to wait on child processes. And again, about how syslog() works. It goes both ways.

formerly_proven 2y ago

OP seems to think SRE and DevOps are seen as lowly by management, yet those jobs still exist here, while the "real programmers" were offshored long ago. Good luck getting a job Actually Building Stuff, almost no one does it any more.

dimitar 2y ago
They are seen as unfortunate cost centers. They don't add new features that are sold to clients. They don't even fix the bugs the clients care about. Making the others more efficient or preventing catastrophe is invisible work.

PaulHoule 2y ago

It’s a predictable problem. Make up a new and fashionable term and it will bite you in the end. See: euphemism treadmill. It has everything to do with the specifics of ‘site’, ‘reliability’, ‘engineer’ and all the technical and social stuff Rachel talks about but on another level it is about the style of discourse, see ‘non’, ‘fungible’, ‘token’.

joshstrange 2y ago

Honest question, what would you call a role that: * Is on call * Manages internal software (grafana, Prometheus, salt stack, etc) * is the first line of defense for issues in the field, works with support and the engineering team to handle problems * Manages a distributed fleet of servers (uses off the self and/or custom code to do so) * Builds internal tools/automations to improve the reliability of our platform and/or address problems automatically * Pushes us to make software/OS updates I ask because that’s what I’d love to hire. Someone who owns the production environment and has the time/space to focus on improving/maintaining it. “Ops” or “Production engineer” is what I’ve referred to it as in the past but this seems like a good place to ask what others would call it.

liveoneggs 2y ago
I've settled on "systems engineer" -- I just try to keep "systems" in the title to imply that my job is largely -- "making many things work together to run the business" I think my current title has the word "cloud" in it or something and my boss rejects (with prejudice) "SRE" or "devops" explicitly to avoid being labeled as operations.
cedws 2y ago
>what would you call a role that … Overworked?
deleted 2y ago
[deleted]
Chilinot 2y ago
"Platform Engineer" has started to grow as a name for pretty much this. An engineer that builds the platform that all things run on. For example, the k8s clusters, the observability stack, has on-call, and builds the internal tools and automations (called the IDP, or internal developer platform).
- liveoneggs 2y ago
  IDP already has a very important meaning as a part of any platform :)

minkles 2y ago

It never meant anything outside Google or huge orgs. Everyone else is running cargo cults.

mike_hearn 2y ago
Moreover, they must be cargo cults. An SRE role was my first job at Google in 2006. It was a product of its time and place. Google could hire the sort of full stack programmer/sysadmin types that the article talks about because they had enough clout and desirability at that time that they could get very skilled programmers to give up programming and do sucky on-call work instead. Even then, just two years after IPO and receiving glowing press everywhere, they struggled with this proposition. A typical strategy (which worked on me) was to promise that whilst they wouldn't offer a job as a developer right away you'd be able to transfer out of SRE after a few years. Another strategy was to build teams that consisted of mixes of sysadmins and developers. In other words, even back then in the most optimal hiring environment possible, there were very few people who were genuinely both developers and sysadmins in one neat package. There was always a clear bias towards one or the other. The hiring process was nightmarish for both sides. SRE had a higher rejection rate than SWE with only about 1% of candidates getting an offer; the accept rate was lower. Hundreds of interviews to get one person through the door. I had to do eight interviews that covered everything from TCP to obscure bash puzzlers, simultaneous equations, C, performance debugging, Python and CS algorithms. All of us had to spend enormous amounts of time running interviews as a consequence, and there was a constant shortage of SREs too, meaning many services were being administered and on-call covered by SWE teams instead. Who then grumbled a lot because that wasn't what they'd signed up for, of course. I think if you're not in a similar situation as Google in 2006 then the SRE role isn't going to make sense for you to try and hire. It's a unicorn role. There just aren't enough people out there who have skills that span such a wide gamut of topics. The fact that she put up an ad for SRE and got "a bunch" of "ops monkeys" is the least surprising thing in the world. Of course that happens. Unless you're willing to interview hundreds of candidates to get one hire, whilst simultaneously running a dragnet recruiting operation, then you'll have to compromise. Even Google had to compromise.
- mwcampbell 2y ago

deleted 2y ago

[deleted]

ChrisMarshallNY 2y ago

Boy, this sounds familiar. Welcome to "Olds-land," Rachel. Sorry about that. It's almost impossible to be a developer/engineer/opsmonkey, with any varied experience, without running into this. People will always find something in your résumé, that makes them uncomfortable. > they didn't have the usual lists of godawful clown software that most places rattle off that you'd be expected to work with. That could be a summary of all that is amiss with the tech industry, these days.

austin-cheney 2y ago

And that’s why I don’t mention my military background during job interviews. In software world it would likely be lost on them anyways, but really certain words are triggers for unfounded assumptions in an industry dominated by unfounded assumptions.

sgarland 2y ago

It doesn’t mean anything for two reasons: companies have treated it as a catch—all, and there is a glut of people calling themselves SREs who have never operated a server that wasn’t in a cloud. You can learn enough about Linux to be decent at your job on only VMs if you’re dedicated, but I’d argue that until you’ve also dealt with hypervisors, bare metal, and hardware issues, you’re missing some of the picture. “That’s no longer applicable, so why should I care?” Because it pops up everywhere I’ve been. Random build server that everyone forgot about but is critical suddenly shits itself, it’s running some ancient version of Ubuntu, and it’s all hand-rolled. Someone decided to provision a bunch of EC2s with bash, but they made critical errors like not knowing to make a new initramfs after configuring mdadm, so now the RAID disappears on reboots. Understanding the fundamentals has always and will always matter. Anyone telling you differently is selling you something.

playingalong 2y ago
> people calling themselves SREs who have never operated a server that wasn’t in a cloud Wait. Since when this is a requirement?
- sgarland 2y ago
  Because if you’ve never touched hardware, you haven’t seen the full gamut of what can go wrong. I don’t think everyone needs a rack in their house, but you can buy old tower servers for dirt cheap, and learn stuff. In my career so far, the best Ops-adjacent people all had homelabs. I view it as a huge green flag if you have one. Not necessarily a red flag if you don’t, but it warrants some more questions. That’s not to say “I have a server” means you’re great; you could just be running Plex with nothing in IaC, no config management, no monitoring, etc.
  - timacles 2y ago
    Listen man, you’re not wrong, but you’re also Living in the past. It’s no different from saying a real programmer does low level C coding. The field has changed, everything is in the cloud. Yeah most engineers can’t troubleshoot low level stuff, but they rarely Need to. There’s no need to be grumpy about it and just accept their jobs are mainly configuring abstractions rather than being some bare metal guru who knows kernel intricacies. I work with like 15 ops/system engineers. None of them know anything about cpu schedulers and raid arrays. We’re just old man

lr4444lr 2y ago

Ops people are cost centers. They can display their wizardry in blog posts until they are blue in the face, but except for those few companies with an incredibly large moat whose main profit is just raw traffic or high uptime (and how many companies like that can maintain that moat indefinitely anyway), engineers not actually building or directly improving product will always be cost centers. It brings me no joy to say this, as ops people tend to be very smart and cool under pressure, but I never see myself becoming one. High quality software and airtight system integrity seems decreasingly important to people paying salaries and investing money. And the world moves on. I refuse to be seen as a cost center if I can avoid it, and I don't have much sympathy left for otherwise very smart tech people who haven't figured this out yet. If you love the job, do it. If not, transition and don't complain. The people in charge are not going to be persuaded by blog rhetoric. I said what I said.

loa_in_ 2y ago
That's a heap of tough love. Ultimately indeed, those who pay, decide, and they're taught to systematically cut costs to improve short term profit above all else.
sgarland 2y ago
I’m not scared. We’ve seen the rise of DBRE because all of the ZIRP startups (the ones who survived, anyway) realized they couldn’t just YOLO their way through indefinitely, and that RDBMS is hard, actually. Of course, they couldn’t admit that they needed DBAs, so they invented a title and mixed in SRE so it would seem cool. I might be in a cost center, but companies will cut this cost center to their own demise.
nunez 2y ago
Sucks, but true. Which is extremely unfortunate. Bad software is expensive to maintain, bad ops can take down companies, and both can slow the sales cycle and threaten renewals. Engineering as a whole should be treated as a profit center.
telotortium 2y ago
Even as a Google SRE who came in with a Systems Engineering focus (i.e., sysadmin), I found it useful to qualify for, and become, a SRE-SWE so that I could also add Software Engineer to my résumé (I wrote enough code that I should get that title). Big and scaling companies still find the combination of sysadmin and software engineering skills extremely valuable, but you do need to emphasize that you are a skilled software engineer as well.

znpy 2y ago

I got the SRE book by google, and read the whole thing, cover to cover. Companies want SRE people but aren't willing to give SRE empowerment and authority. So companies do what companies to: take a regular team of operations people and slap the SRE term on it, and call it a day. And it doesn't work, of course. --- Regarding the empowerment & authority: according to "the book", SREs often play the role of "launch coordination engineers" as in vetting (read: roasting) a service before it goes live and have authority to say "this won't go live, fix this first" and to do so no matter what deadline is going to be missed. Also SRE team have the extreme prerogative to "give back the pager" as in take a service back to the development team and say: it's not stable enough, YOU will be on-call for it until you fix the shit you wrote. These are two emblematic examples, but there are many more in the book. Can you imagine any of the non-google (and non-faang) companies actually doing something like that?

28304283409234 2y ago
As companies did with DevOps.

znpy 2y ago

To me, a SRE is *both* a sysadmin AND a programmer, developer, whatever you want to call it. It's a logical-and, not an XOR. By sysadmin, I mean "runs a mean Unix box, including fixing things and diving deeply when they break", and by the programmer/whatever part of it, I mean "makes stuff come into existence that wasn't there before". The main issue i see with that is the companies usually aren't willing to advertise and pay SRE salaries for actual SRE skills. The skillset described in the above quotes are essentially the skills of an SWE and of a Sysadmin. So essentially you're doing two jobs for one salary. There are people capable of doing two jobs, but you won't find them until you start advertising and paying actual-SRE salaries.

nunez 2y ago

Well, yeah, in a world where corporates don't want to pay for sysadmins that can actually code or giving their devs a pager, you get what you saw: sysadmins and the people that do Jenkins being renamed as the "devops" team in 2014, then the "SRE" team in 2016 or so, then, "platform teams" after the age of Kubernetes in 2018-ish. There are so many companies that have SRE teams despite those teams not maintaining a website!

kkfx 2y ago

Management hate classic operation because it's too powerful, it's the operation part of the nervous system of a company, operation can't be managed nor segmented by the management, as a result many try to do their best to delete sysadmins from the scene. Results are that most end up on someone else system with some form of operation. Meaning the cloud. Now that costs skyrocket, issues piling up, no one seems to been able to create a damn full infra well, the push deflate. We have experienced many similar trends: - full stack virtualization on x86, sold as the future, a super-duper simplification, actually a way to allow third party selling pre-made images to those who have no operation or do not own the bare metal (VPS etc); - when people realize how big the overhead is paravirtualization became an old-new trend, mostly with k8s, now this model start to creak and the push toward owning back the infra and the iron start to be noisy I expect in a decade a mainstream NixOS/Guix System move as we had with Ansible/Salt before, the old CFEngine much before and so on, in 20 years probably companies will own back their machine room with a Plan 9 -alike model to just get redundancies and extra temporary resources.

muppetman 2y ago

Everyone lost respect for SREs when Elon took over Twitter and culled lots of staff. Every SRE in the land lined up to shout from the rooftops "There is no way Twitter can keep working it's going to flame out and fall over and keep crashing" and it didn't, not even once.