7 ms·

I have a theory that Google invented the SRE because they didn't know how to hire system administrators, but they knew how to hire software engineers. So, they

I have a theory that Google invented the SRE because they didn't know how to hire system administrators, but they knew how to hire software engineers. So, they just hired software engineers and told them to figure out the systems. I say this only partially in jest.

lclarkmichalek 10y ago
Eh, I think they say something similar in the book. The other side of it was that they wanted a software engineering approach to systems administration (which completely depends on your opinions of the respective professions)
- dj_jorjinho 10y ago
  The opposite also occurs: a SysAdmin gets hired to do DevOps, but they're not qualified to be Software Engineers. The results is that, if you're not looking, your ops code ends up a series of "scripts" instead of a series of well structured tools. But I guess this depends on the company culture to begin with.
  - deleted 10y ago
    [deleted]
- SwellJoe 10y ago
  It seems to be mostly working for them, though I think companies that haven't gone so far down that rabbit hole are also doing fine. One of the premises of SRE that I occasionally take issue with is "build your own tools". That's a fine idea, sometimes, but it also results in NIH syndrome, which has a real cost. Obviously, Google is operating at a scale that no one ever has before (and almost no one else does or ever will), and so, sometimes they have to invent the technologies for doing it. But, the somewhat dismissive tone of the SRE literature I've read that implies system administrators scale linearly while SREs scale drastically higher is kind of off-putting, and misrepresentative, I think. Sysadmins have been scaling systems for as long as there have been systems; sometimes just one person or a small team in the data center, running the whole show for quite large companies.
hashhar 10y ago
You may be saying it in jest but I've seen it to be true (sample size of 1). I am a recently graduating (this summer) CS undergrad from India and was offered the chance to interview for a customer service offering company (think Zendesk) for the role of a Site Reliability Engineer. The take home tests they gave me heavily reflected that they were looking for an engineer instead of a sysadmin as nowhere during the interview did they discuss about anything other than Chef, Terraform, Nginx and other things that mainly deal with deployment and devops and have very little to do with "reliability". They didn't have anything about networks or fault-tolerance. PS: I really liked the interview process though in that they had take home tests and there was a good depth of discussion relating to those tests the next day. I eventually won't be joining them because of some information about their office culture in their India office. Sad.
- vertex-four 10y ago
  I would suggest that what other companies call "SREs" is not likely to be what Google calls SREs, given the difficulty of implementing it correctly.
- markdown 10y ago
  Freshdesk?
  - hashhar 10y ago
    No. Kayako.
- StreamBright 10y ago
  When you are applying for SRE outside Google and Amazon or maybe Facebook it mean something entirely different. Usually they just use SRE for describing sysadmins, or a little bit more, maybe adding some automation knowledge. Getting hired as an SRE at Google and Amazon is extremely challenging task, lots of areas where you need to be proficient.
acchow 10y ago
My impression is many sysadmin at the time didn't really understand the systems they were dealing with - that kind of problem solving, hacking, and mapping the territory comes with software engineering. So why not just hire software engineers?
- tyingq 10y ago
  I don't see either path as having an overwhelming advantage. What happened here is that the scale of sysadmin tasks escalated to the point where software engineering skills brought some advantage. It did not, though, negate the importance of traditional sysadmin skills. The best people in this space are those that can bridge both worlds. Not just "SRE" either. Devops is also a hybrid role, as is performance engineering. Look at Adrian Cockroft's background as an example.
  - inopinatus 10y ago
    DevOps is a concept, not a role. It means breaking down the silos and allowing processes, ideas, capabilities to run across both SWE & operations teams. I never knew a really first-class *nix sysadmin who wasn't also a kernel hacker, so this isn't anything new, it's just got a label. Probably the most devops-y thing you can do in an applications environment is operate a continuous delivery pipeline. The most un-devops-y thing you can do is start a devops team.
    - tyingq 10y ago
      Whether it's correct or not, it's a named title people are hiring for. https://www.indeed.com/q-DevOps-jobs.html https://www.indeed.com/q-DevOps-jobs.html So I thought it worth mentioning as one of those hybrid roles.
      - inopinatus 10y ago
        That's a great example. There's nothing in those job listings that wasn't, conceptually speaking, in the job description of a system engineer or sysadmin thirty years ago. In this usage of "DevOps" it's a buzzword, and I recommend you should avoid anyone who puts buzzwords in your job title, because it speaks volumes up front about their culture and values and clarity of thinking. I really do like the descriptor Site Reliability Engineer, though, for exactly the same reasons.
jasode 10y ago
>they didn't know how to hire system administrators, but they knew how to hire software engineers Setting aside your partial jest, I think Google's approach of hiring "programmers" to fill the role of SRE makes perfect sense for their organization. The "sysadmin" can have a wide range of meanings and responsibilities but many will have experience managing COTS[1] software like Oracle RDBMS, Peoplesoft HR, MS Exchange email server, etc. A typical system admin can then use complimentary COTS tools like Microsoft SCOM System Center Operations Manager, HP OpenView, IBM Tivoli, CA Unicenter/Spectrum. On Linux, it would be similar tools like Nagios, Zabbix to monitor Apache webservers, Mysql, etc. In Google's case, they didn't cluster a bunch of COTS Oracle dbs together to serve web surfers. Instead, Google wrote a proprietary platform like BigTable or Spanner[2]. As a result, they also want additional software that monitors the health of Spanner. Since none of the monitoring tools from enterprise vendors like HP/MS/CA are adequate (because they have no out-of-box software-agents for Spanner), they need another layer of programmers to write custom management/monitoring tools. Those programmers are the SREs. All the above enterprise tools from HP/MS/CA/etc including the mother db server, the agents, the plugins are written in C/C++. If your hiring job description is "system admin", most candidates will not have the skills to develop those enterprise monitoring tools from scratch. Yes, many sysadmins have programming skills to write Bash/Powershell scripts. E.g. a bash script might have a polling loop that checks for an error file existence and then has if/else/fi to send an email. However, that's not the same programming skill as developing a proprietary version of HP OpenView and MS SCOM. Most traditional sys admins do not write low-level C code like Zabbix.[3] Basically, if you're a company that writes complex proprietary softare and none of the existing enterprise monitoring tools can manage it, you'll have to hire programmers instead of sysadmins as the baseline skillset for SREs. [1] https://en.wikipedia.org/wiki/Commercial_off-the-shelf https://en.wikipedia.org/wiki/Commercial_off-the-shelf [2] https://en.wikipedia.org/wiki/Spanner_(database) https://en.wikipedia.org/wiki/Spanner_(database) [3] e.g. https://github.com/zabbix/zabbix/blob/trunk/src/zabbix_server/events.c https://github.com/zabbix/zabbix/blob/trunk/src/zabbix_serve...
- technofiend 10y ago
  Makes me wonder what they pay and how they get over the usual SV prejudice against hiring people over 30. I read their SRE description a while ago and thought it sounded perfect for me, but if you end up there and for whatever reason it doesn't work out you'll end up losing a lot of money leaving SV again. It's not cheap to uproot yourself and move across country and it's not like anyone else in SV will hire folks over a certain age; at least that's what I'm left to conclude after lurking HN for a few years.
StreamBright 10y ago
This is not true at all. Most of the great first SRE were systems engineers (this is how Amazon used to call SREs) who understood the entire stack from the UI down the the network packet and OS kernel level. Most software engineers do not have serious networking or operating system knowledge. On the top of that, there is a reason why performance engineers are so rare, understanding the complex computing systems we have is not an easy task. System administrators are able to configure your services but not necessarily know all the details of the operating system and file systems that is required to be an SRE. I think you are trying to oversimplify this subject quite a bit.
- jacquesm 10y ago
  > Most of the great first SRE were systems engineers (this is how Amazon used to call SREs) who understood the entire stack from the UI down the the network packet and OS kernel level. Aka the 'full stack developer' as opposed to the 'full stack developer' that knows both JavaScript, PHP and how to install the base distribution of CentOS.
  - metaphorm 10y ago
    you're commenting on sloppy use of language. you're right, we do tend to abbreviate and exclude relevant descriptive words. the "full stack developer" that knows both JavasScript and PHP (or whatever), and probably some database stuff, and how to config a HTTP server, and a few other bits of web odds and ends is really a "full stack web application developer". usually there is enough context to know if "web application" has been abbreviated out of the term though. as for the other kind of "full stack developer", the legendary creature that is equally comfortable doing UI scripting and Linux Kernel hacking...well personally I've never met one. mythical beast in my experience.
- throwaway2016a 10y ago
  > Most software engineers do not have serious networking or operating system knowledge. Anecdotally, this seems wrong but may actually be sadly right. I don't know how you can get through a decent Computer Science program without knowing these things. Although, I acknowledge quality in CS programs vary hugely. We had to write an Operating System (with basic memory and process management), know the OSI model up and down, and be able to deep dive into TCP/IP and UDP packets. In addition to coding and math. I'm actually pro software "engineer" but perhaps we should draw the line somewhere on a bare minimum of skills required. Probably about the line of an ABET accredited undergraduate BS in CS degree?
skywhopper 10y ago
Given the tone of the first answer, I would say this is correct, as they clearly didn't hire any good ones, anyway.
dsr_ 10y ago
[quoting myself from quite a while ago] Once upon a time there were programmers. Then there were systems programmers and application programmers. Systems programmers wrote operating systems and utilities for them. App programmers wrote apps. There was a lot of crossover. Then there were operators, systems programmers and application programmers. Operator was a junior position who did physical things (mount tapes, plug in cables) and ran commands to do things on the systems. They usually moved up to being… Systems administrators, who did some programming in service to the systems, but not too much. The more senior a sysadmin was, the more time they spent programming and the less time they spent doing physical things… unless they wanted to do that. Sysadmins started to specialize. People who configured switches and routers and talked to telephone companies became “network engineers”. People who spent time working on firewalls and security policies and thinking about that became “security engineers”. Junior people who read scripts to end users became the helpdesk. And so forth. Then we noticed that a bunch of people were doing things manually when they should be automated. This was especially bad in places where there were no senior sysadmins or systems programmers. But we did have the internet, and senior sysadmins got together and started writing tools to make their lives easier: infrastructure automation. You probably know the story from there, but I’ll wrap up with one more important point: you know how when writing a business application you need to have a subject matter expert who actually knows what they are doing? Operations is exactly the same way. All the automation in the world won’t help you if you don’t have someone around who knows what they are doing. Some people can outsource this to “the cloud”, but not everyone.

lclarkmichalek 10y ago

Eh, I think they say something similar in the book. The other side of it was that they wanted a software engineering approach to systems administration (which completely depends on your opinions of the respective professions)

dj_jorjinho 10y ago
The opposite also occurs: a SysAdmin gets hired to do DevOps, but they're not qualified to be Software Engineers. The results is that, if you're not looking, your ops code ends up a series of "scripts" instead of a series of well structured tools. But I guess this depends on the company culture to begin with.
- deleted 10y ago
  [deleted]
SwellJoe 10y ago
It seems to be mostly working for them, though I think companies that haven't gone so far down that rabbit hole are also doing fine. One of the premises of SRE that I occasionally take issue with is "build your own tools". That's a fine idea, sometimes, but it also results in NIH syndrome, which has a real cost. Obviously, Google is operating at a scale that no one ever has before (and almost no one else does or ever will), and so, sometimes they have to invent the technologies for doing it. But, the somewhat dismissive tone of the SRE literature I've read that implies system administrators scale linearly while SREs scale drastically higher is kind of off-putting, and misrepresentative, I think. Sysadmins have been scaling systems for as long as there have been systems; sometimes just one person or a small team in the data center, running the whole show for quite large companies.

hashhar 10y ago

You may be saying it in jest but I've seen it to be true (sample size of 1). I am a recently graduating (this summer) CS undergrad from India and was offered the chance to interview for a customer service offering company (think Zendesk) for the role of a Site Reliability Engineer. The take home tests they gave me heavily reflected that they were looking for an engineer instead of a sysadmin as nowhere during the interview did they discuss about anything other than Chef, Terraform, Nginx and other things that mainly deal with deployment and devops and have very little to do with "reliability". They didn't have anything about networks or fault-tolerance. PS: I really liked the interview process though in that they had take home tests and there was a good depth of discussion relating to those tests the next day. I eventually won't be joining them because of some information about their office culture in their India office. Sad.

vertex-four 10y ago
I would suggest that what other companies call "SREs" is not likely to be what Google calls SREs, given the difficulty of implementing it correctly.
markdown 10y ago
Freshdesk?
- hashhar 10y ago
  No. Kayako.
StreamBright 10y ago
When you are applying for SRE outside Google and Amazon or maybe Facebook it mean something entirely different. Usually they just use SRE for describing sysadmins, or a little bit more, maybe adding some automation knowledge. Getting hired as an SRE at Google and Amazon is extremely challenging task, lots of areas where you need to be proficient.

acchow 10y ago

My impression is many sysadmin at the time didn't really understand the systems they were dealing with - that kind of problem solving, hacking, and mapping the territory comes with software engineering. So why not just hire software engineers?

tyingq 10y ago
I don't see either path as having an overwhelming advantage. What happened here is that the scale of sysadmin tasks escalated to the point where software engineering skills brought some advantage. It did not, though, negate the importance of traditional sysadmin skills. The best people in this space are those that can bridge both worlds. Not just "SRE" either. Devops is also a hybrid role, as is performance engineering. Look at Adrian Cockroft's background as an example.
- inopinatus 10y ago
  DevOps is a concept, not a role. It means breaking down the silos and allowing processes, ideas, capabilities to run across both SWE & operations teams. I never knew a really first-class *nix sysadmin who wasn't also a kernel hacker, so this isn't anything new, it's just got a label. Probably the most devops-y thing you can do in an applications environment is operate a continuous delivery pipeline. The most un-devops-y thing you can do is start a devops team.
  - tyingq 10y ago
    Whether it's correct or not, it's a named title people are hiring for. https://www.indeed.com/q-DevOps-jobs.html https://www.indeed.com/q-DevOps-jobs.html So I thought it worth mentioning as one of those hybrid roles.
    - inopinatus 10y ago
      That's a great example. There's nothing in those job listings that wasn't, conceptually speaking, in the job description of a system engineer or sysadmin thirty years ago. In this usage of "DevOps" it's a buzzword, and I recommend you should avoid anyone who puts buzzwords in your job title, because it speaks volumes up front about their culture and values and clarity of thinking. I really do like the descriptor Site Reliability Engineer, though, for exactly the same reasons.

jasode 10y ago

>they didn't know how to hire system administrators, but they knew how to hire software engineers Setting aside your partial jest, I think Google's approach of hiring "programmers" to fill the role of SRE makes perfect sense for their organization. The "sysadmin" can have a wide range of meanings and responsibilities but many will have experience managing COTS[1] software like Oracle RDBMS, Peoplesoft HR, MS Exchange email server, etc. A typical system admin can then use complimentary COTS tools like Microsoft SCOM System Center Operations Manager, HP OpenView, IBM Tivoli, CA Unicenter/Spectrum. On Linux, it would be similar tools like Nagios, Zabbix to monitor Apache webservers, Mysql, etc. In Google's case, they didn't cluster a bunch of COTS Oracle dbs together to serve web surfers. Instead, Google wrote a proprietary platform like BigTable or Spanner[2]. As a result, they also want additional software that monitors the health of Spanner. Since none of the monitoring tools from enterprise vendors like HP/MS/CA are adequate (because they have no out-of-box software-agents for Spanner), they need another layer of programmers to write custom management/monitoring tools. Those programmers are the SREs. All the above enterprise tools from HP/MS/CA/etc including the mother db server, the agents, the plugins are written in C/C++. If your hiring job description is "system admin", most candidates will not have the skills to develop those enterprise monitoring tools from scratch. Yes, many sysadmins have programming skills to write Bash/Powershell scripts. E.g. a bash script might have a polling loop that checks for an error file existence and then has if/else/fi to send an email. However, that's not the same programming skill as developing a proprietary version of HP OpenView and MS SCOM. Most traditional sys admins do not write low-level C code like Zabbix.[3] Basically, if you're a company that writes complex proprietary softare and none of the existing enterprise monitoring tools can manage it, you'll have to hire programmers instead of sysadmins as the baseline skillset for SREs. [1] https://en.wikipedia.org/wiki/Commercial_off-the-shelf https://en.wikipedia.org/wiki/Commercial_off-the-shelf [2] https://en.wikipedia.org/wiki/Spanner_(database) https://en.wikipedia.org/wiki/Spanner_(database) [3] e.g. https://github.com/zabbix/zabbix/blob/trunk/src/zabbix_server/events.c https://github.com/zabbix/zabbix/blob/trunk/src/zabbix_serve...

technofiend 10y ago
Makes me wonder what they pay and how they get over the usual SV prejudice against hiring people over 30. I read their SRE description a while ago and thought it sounded perfect for me, but if you end up there and for whatever reason it doesn't work out you'll end up losing a lot of money leaving SV again. It's not cheap to uproot yourself and move across country and it's not like anyone else in SV will hire folks over a certain age; at least that's what I'm left to conclude after lurking HN for a few years.

StreamBright 10y ago

This is not true at all. Most of the great first SRE were systems engineers (this is how Amazon used to call SREs) who understood the entire stack from the UI down the the network packet and OS kernel level. Most software engineers do not have serious networking or operating system knowledge. On the top of that, there is a reason why performance engineers are so rare, understanding the complex computing systems we have is not an easy task. System administrators are able to configure your services but not necessarily know all the details of the operating system and file systems that is required to be an SRE. I think you are trying to oversimplify this subject quite a bit.

jacquesm 10y ago
> Most of the great first SRE were systems engineers (this is how Amazon used to call SREs) who understood the entire stack from the UI down the the network packet and OS kernel level. Aka the 'full stack developer' as opposed to the 'full stack developer' that knows both JavaScript, PHP and how to install the base distribution of CentOS.
- metaphorm 10y ago
  you're commenting on sloppy use of language. you're right, we do tend to abbreviate and exclude relevant descriptive words. the "full stack developer" that knows both JavasScript and PHP (or whatever), and probably some database stuff, and how to config a HTTP server, and a few other bits of web odds and ends is really a "full stack web application developer". usually there is enough context to know if "web application" has been abbreviated out of the term though. as for the other kind of "full stack developer", the legendary creature that is equally comfortable doing UI scripting and Linux Kernel hacking...well personally I've never met one. mythical beast in my experience.
throwaway2016a 10y ago
> Most software engineers do not have serious networking or operating system knowledge. Anecdotally, this seems wrong but may actually be sadly right. I don't know how you can get through a decent Computer Science program without knowing these things. Although, I acknowledge quality in CS programs vary hugely. We had to write an Operating System (with basic memory and process management), know the OSI model up and down, and be able to deep dive into TCP/IP and UDP packets. In addition to coding and math. I'm actually pro software "engineer" but perhaps we should draw the line somewhere on a bare minimum of skills required. Probably about the line of an ABET accredited undergraduate BS in CS degree?

skywhopper 10y ago

Given the tone of the first answer, I would say this is correct, as they clearly didn't hire any good ones, anyway.

dsr_ 10y ago

[quoting myself from quite a while ago] Once upon a time there were programmers. Then there were systems programmers and application programmers. Systems programmers wrote operating systems and utilities for them. App programmers wrote apps. There was a lot of crossover. Then there were operators, systems programmers and application programmers. Operator was a junior position who did physical things (mount tapes, plug in cables) and ran commands to do things on the systems. They usually moved up to being… Systems administrators, who did some programming in service to the systems, but not too much. The more senior a sysadmin was, the more time they spent programming and the less time they spent doing physical things… unless they wanted to do that. Sysadmins started to specialize. People who configured switches and routers and talked to telephone companies became “network engineers”. People who spent time working on firewalls and security policies and thinking about that became “security engineers”. Junior people who read scripts to end users became the helpdesk. And so forth. Then we noticed that a bunch of people were doing things manually when they should be automated. This was especially bad in places where there were no senior sysadmins or systems programmers. But we did have the internet, and senior sysadmins got together and started writing tools to make their lives easier: infrastructure automation. You probably know the story from there, but I’ll wrap up with one more important point: you know how when writing a business application you need to have a subject matter expert who actually knows what they are doing? Operations is exactly the same way. All the automation in the world won’t help you if you don’t have someone around who knows what they are doing. Some people can outsource this to “the cloud”, but not everyone.