7 ms·

I don't know anything about Google and its SREs. Do they really just hire software devs and expect them to be good at Ops? The opposite, hiring a sysadmin and t

I don't know anything about Google and its SREs. Do they really just hire software devs and expect them to be good at Ops? The opposite, hiring a sysadmin and then assigning them to a software team to develop a product, seems equally problematic.

ipsum2 11y ago
> Candidates should be able to pass or nearly pass normal dev hiring bar, and may have some additional skills that are rare among devs (e.g., L1 - L3 networking or UNIX system internals). They hire devs with ops experience.
- toomuchtodo 11y ago
  Can't be good at everything. Only so many hours in the day.
- bogomipz 11y ago
  Can you or someone else possibly say what the "normal dev hiring bar" is? I've enjoyed what I've read so far in the book and I get it that there is an opportunity here for Google to market the SRE role b/c they are apparently hard to fill but why didn't they put a chapter in the book of what that dev skill set is? Or else what dev skill are expected of someone working in this capacity. Seems like lifting the shroud of mystery would do wonders for marketing a role that is hard to fill.
  - thrownaway2424 11y ago
    SRE-SWEs at Google are expected to be able to design and implement the same things that the developers they support are working on. If you are on Blah-SRE you would be expected to be capable of at the very least reading and understanding the code that Blah-Team writes, reviewing their code, etc. You might not _actually_ work on that code (it might be more important for you to be working on the automation, monitoring, or whatever) but you'd be qualified to do so. That's why the SRE-SWE interview is basically the SWE interview: graph coloring and complexity analysis and all that.
    - bogomipz 11y ago
      In C, Java, Python?
      - thrownaway2424 11y ago
        Good question :) SRE-written system are never or almost never written in Java. However many SRE teams support Java systems. Literally nothing at Google except third-party code (like Linux kernel) is written in straight C. C++ is very common, though. Python sucks and is useless. Most things that were written in python are now written with Go. I guess your question is about what happens if a C++ expert ends up on a Java team, but I don't really know. I happen to have landed in C++ world, which suited me.
        bluecmd 11y ago
        > Python sucks and is useless. Most things that were written in python are now written with Go. This is simply not true in any way.
        thrownaway2424 11y ago
        Well it's a big company. In my area people who propose writing new code in python are either gently corrected or heartily derided.
        packetslave 11y ago
        True in my area as well. The only new things being written in Python are things that need to interop with other systems that have giant existing Python libraries.
        morgosmaci 11y ago
        I was an SRE who was most comfortable in C++ who changed ladders to a SWE team who was primarily Java. To be honest it was harder to learn all the team specific stuff than pick up Java. The reason Python tends to suck is not all really nice libraries are ported to it in any reasonable amount of time. (If you need it, you end up doing the work). Go sucks in new and totally different ways.
sciurus 11y ago
Although I can't find the link now, I've read that there is a pretty intensive training process for SREs. This would make sense since Google's infrastructure is so unique.
- huangwei_chang 11y ago
  I think you are right. There is a lot of training after one join the team. The job does require very different skill sets.
etcet 11y ago
From my limited experience, they want devs. The initial phone screen is all sysadminy questions (e.g. what commands show you i/o usage?). The second screen is a coding interview. I'm a sysadmin who works in bash every day. The coding questions are mostly related to log parsing. My code interviewer didn't seem to even acknowledge that bash was a programming language and did not understand a simple grep and sort pipeline. Edit: Sorry everyone! I totally got my LinkedIn and Google interviews mixed up. What I described above is my LinkedIn experience.
- mason55 11y ago
  That's interesting, when I interviewed (admittedly it was like 8 years ago), the first interview was standard CS questions and the second interview focused on deeper C++ questions (things like analyzing how a vtable is constructed).
- mquander 11y ago
  I interviewed as SRE four months ago and I got a very typical programmer phone screen and interview, just exactly the kinds of whiteboard CS questions you expect Google might ask. (My bigger strength is on the programming side instead of the ops side, so I think perhaps they explicitly tailored the interview to measure that.)
- thesnider 11y ago
  There are actually two different SRE roles: the one people are describing above where you are 85-99% of the way to SWE (Software Engineer), and you have sysadminy experience, and another one where you are 100+% of the SWE bar and optionally have sysadminy experience. The former is called SRE-SE (Systems Engineer), and the latter SRE-SWE. SRE-SE interviews are super heavy on the sysadmin stuff usually, with less (but still significant) attention paid to SWE skills, whereas SRE-SWE interviews may not even have an SRE component (it's possible for candidates in the 'normal' SWE hiring pipeline to be shunted to SRE-SWE post-interview).
  - thrownaway2424 11y ago
    Yeah a lot of people don't understand this distinction. You have your pure SWEs who were hired that way who then were either picked for or switched to SRE-SWE. Then you have people who were recruited into SRE-SWE from the beginning. People in SWE and SRE-SWE job classes can freely move between them. Then finally you have people who were recruited as SRE-SysEng, or were recruited as SWEs and didn't quite make the cut. These folks have to do a transfer interview to jump to the SWE or SRE-SWE roles.
bobp127001 11y ago
It's notoriously hard to hire for. They cherrypick devs that are 85-99%[0] of the way to passing the Google SWE hiring bar, who in addition have ops experience. The Google SRE head honcho says: [1] >Fundamentally, it’s what happens when you ask a software engineer to design an operations function. So I think the short answer to your question is yes, they hire software devs and train them [2], but require them to already have some ops/Linux internals knowledge. [0]: SRE Book [1]: https://landing.google.com/sre/interview/ben-treynor.html https://landing.google.com/sre/interview/ben-treynor.html [2]: https://www.usenix.org/conference/srecon15/program/presentation/widdowson https://www.usenix.org/conference/srecon15/program/presentat...
- deleted 11y ago
  [deleted]
btilly 11y ago
They want a cross between a developer and a sysadmin. When I was there they hired people with one background and hoped they would be capable of the other. This didn't work out for about half of them. Making things worse, they had no procedure to say, "OK, we hired someone we think is good, but this is clearly the wrong role for them." Which lead to losing a lot of people who had potential. Making it personal, I interviewed for a pure dev position then was offered a job as an SRE. It turns out that I don't make a good sysadmin. After I left, I was amazed at how many people I met knew of someone else who had had the same experience. Ironically recruiters see a year as a Google SRE on my resume and think I might want to join SRE teams at other companies. Most of that stopped after I changed my Linked In profile to say that I am never interested in SRE jobs.
- deleted 11y ago
  [deleted]
- fishnchips 11y ago
  Yup. Exactly my experience.
- tayo42 11y ago
  Just curious, what made you think you weren't a good admin?
  - btilly 11y ago
    Explaining the concretes would be a lot of unpleasant detail. The why is far more interesting. I am very good at "tunnel vision". Taking a thread and following it in depth. Completely learning a system in depth so that anything that comes up, I know exactly where to go and what to do. I am weak on "peripheral vision". Keeping track of 15 balls in the air, and operating with limited information about each. Being effective with limited context about each individual system because there isn't time to learn any of them in depth. Tunnel vision is a virtue in a software developer. Peripheral vision is essential for a sysadmin. Now imagine a person with poor peripheral vision trying to learn a system as big and complex as Google's. And being responsible to support 15 different pieces of software written by 15 different teams so that when anything went belly up with any of them, you can trouble-shoot and get it running again. Nothing particularly bad happened, but I also wasn't accomplishing the job to the standards that Google wanted in that role. And I was not the only person on my team failing in that way. Ideally it would have been my manager's job to say, "I recognize that this person isn't working out here, is there a better role for them?" That didn't happen. Several months later my manager got fired. I was privately told that my situation was a trigger, but I don't know details. The fact that my situation is fairly common strongly suggests that the ultimate failure was organizational and systemic. I don't fault Google for hiring developers into their hybrid developer/sysadmin role. I do fault them for not having an explicit onramp/reconsideration process to mitigate the risk that they create by doing so.
bovermyer 11y ago
I don't work for Google, but I did follow that career path - started as a developer, then moved into operations. I didn't immediately become an expert as soon as I made the transition - it was a slow process, and I'm still not what I'd consider the equal of a veteran sysadmin in terms of low-level system knowledge. However, I can hold my own in either discipline.
mehta 11y ago
They kinda do. Over time, the skills of a dev and SRE diverge a bit but at the core they are the same. I am a dev at Google but went through a program called Mission Control[1]. (It was a great learning experience and helped me do my job as a dev much better.) As others have said, there is a pretty intensive training that you go through and obviously you learn a lot when you are handling things... [1] http://googleresearch.blogspot.com/2012/07/site-reliability-engineers-solving-most.html http://googleresearch.blogspot.com/2012/07/site-reliability-...
fishnchips 11y ago
You're quite right. I'm an ex-Googler and SRE was the last thing I did at Google. As a first-and-foremost programmer I was bored out of my wits and frustrated by the fact that while I was writing repetitive monitoring rules actual development was happening somewhere else and I had little impact on the design of the system. Granted, a lot of folks over in SRE loved it but I would venture to say that most of them were much closer to being a sysadmin than programmer. Folks like myself tended to 1) move to SWE, 2) quit and 3) some endured and became managers over time.
nunez 11y ago
They look for someone that is strong at operating systems and networking knowledge AND someone that's a decently strong programmer. IOW, someone that can pass a low-level ops interview and a Google SWE interview. This is a tough position to fill but I've worked with folks who are genuinely good at both, so they're out there.
- slyall 11y ago
  What I'm wondering is have they created a model that only works if you employee "top 5 percent" people. Sort of like if long-haul truckers needs the skills and training of Fighter pilots.
  - tamana 11y ago
    Yes. What you get is a lot of "grunge work" that never gets done because everyone is busy inventing the next gen cure-all.

ipsum2 11y ago

> Candidates should be able to pass or nearly pass normal dev hiring bar, and may have some additional skills that are rare among devs (e.g., L1 - L3 networking or UNIX system internals). They hire devs with ops experience.

toomuchtodo 11y ago
Can't be good at everything. Only so many hours in the day.
bogomipz 11y ago
Can you or someone else possibly say what the "normal dev hiring bar" is? I've enjoyed what I've read so far in the book and I get it that there is an opportunity here for Google to market the SRE role b/c they are apparently hard to fill but why didn't they put a chapter in the book of what that dev skill set is? Or else what dev skill are expected of someone working in this capacity. Seems like lifting the shroud of mystery would do wonders for marketing a role that is hard to fill.
- thrownaway2424 11y ago
  SRE-SWEs at Google are expected to be able to design and implement the same things that the developers they support are working on. If you are on Blah-SRE you would be expected to be capable of at the very least reading and understanding the code that Blah-Team writes, reviewing their code, etc. You might not _actually_ work on that code (it might be more important for you to be working on the automation, monitoring, or whatever) but you'd be qualified to do so. That's why the SRE-SWE interview is basically the SWE interview: graph coloring and complexity analysis and all that.
  - bogomipz 11y ago
    In C, Java, Python?
    - thrownaway2424 11y ago
      Good question :) SRE-written system are never or almost never written in Java. However many SRE teams support Java systems. Literally nothing at Google except third-party code (like Linux kernel) is written in straight C. C++ is very common, though. Python sucks and is useless. Most things that were written in python are now written with Go. I guess your question is about what happens if a C++ expert ends up on a Java team, but I don't really know. I happen to have landed in C++ world, which suited me.
      - bluecmd 11y ago
        > Python sucks and is useless. Most things that were written in python are now written with Go. This is simply not true in any way.
        thrownaway2424 11y ago
        Well it's a big company. In my area people who propose writing new code in python are either gently corrected or heartily derided.
        packetslave 11y ago
        True in my area as well. The only new things being written in Python are things that need to interop with other systems that have giant existing Python libraries.
      - morgosmaci 11y ago
        I was an SRE who was most comfortable in C++ who changed ladders to a SWE team who was primarily Java. To be honest it was harder to learn all the team specific stuff than pick up Java. The reason Python tends to suck is not all really nice libraries are ported to it in any reasonable amount of time. (If you need it, you end up doing the work). Go sucks in new and totally different ways.

sciurus 11y ago

Although I can't find the link now, I've read that there is a pretty intensive training process for SREs. This would make sense since Google's infrastructure is so unique.

huangwei_chang 11y ago
I think you are right. There is a lot of training after one join the team. The job does require very different skill sets.

etcet 11y ago

From my limited experience, they want devs. The initial phone screen is all sysadminy questions (e.g. what commands show you i/o usage?). The second screen is a coding interview. I'm a sysadmin who works in bash every day. The coding questions are mostly related to log parsing. My code interviewer didn't seem to even acknowledge that bash was a programming language and did not understand a simple grep and sort pipeline. Edit: Sorry everyone! I totally got my LinkedIn and Google interviews mixed up. What I described above is my LinkedIn experience.

mason55 11y ago
That's interesting, when I interviewed (admittedly it was like 8 years ago), the first interview was standard CS questions and the second interview focused on deeper C++ questions (things like analyzing how a vtable is constructed).
mquander 11y ago
I interviewed as SRE four months ago and I got a very typical programmer phone screen and interview, just exactly the kinds of whiteboard CS questions you expect Google might ask. (My bigger strength is on the programming side instead of the ops side, so I think perhaps they explicitly tailored the interview to measure that.)
thesnider 11y ago
There are actually two different SRE roles: the one people are describing above where you are 85-99% of the way to SWE (Software Engineer), and you have sysadminy experience, and another one where you are 100+% of the SWE bar and optionally have sysadminy experience. The former is called SRE-SE (Systems Engineer), and the latter SRE-SWE. SRE-SE interviews are super heavy on the sysadmin stuff usually, with less (but still significant) attention paid to SWE skills, whereas SRE-SWE interviews may not even have an SRE component (it's possible for candidates in the 'normal' SWE hiring pipeline to be shunted to SRE-SWE post-interview).
- thrownaway2424 11y ago
  Yeah a lot of people don't understand this distinction. You have your pure SWEs who were hired that way who then were either picked for or switched to SRE-SWE. Then you have people who were recruited into SRE-SWE from the beginning. People in SWE and SRE-SWE job classes can freely move between them. Then finally you have people who were recruited as SRE-SysEng, or were recruited as SWEs and didn't quite make the cut. These folks have to do a transfer interview to jump to the SWE or SRE-SWE roles.

bobp127001 11y ago

It's notoriously hard to hire for. They cherrypick devs that are 85-99%[0] of the way to passing the Google SWE hiring bar, who in addition have ops experience. The Google SRE head honcho says: [1] >Fundamentally, it’s what happens when you ask a software engineer to design an operations function. So I think the short answer to your question is yes, they hire software devs and train them [2], but require them to already have some ops/Linux internals knowledge. [0]: SRE Book [1]: https://landing.google.com/sre/interview/ben-treynor.html https://landing.google.com/sre/interview/ben-treynor.html [2]: https://www.usenix.org/conference/srecon15/program/presentation/widdowson https://www.usenix.org/conference/srecon15/program/presentat...

deleted 11y ago
[deleted]

btilly 11y ago

They want a cross between a developer and a sysadmin. When I was there they hired people with one background and hoped they would be capable of the other. This didn't work out for about half of them. Making things worse, they had no procedure to say, "OK, we hired someone we think is good, but this is clearly the wrong role for them." Which lead to losing a lot of people who had potential. Making it personal, I interviewed for a pure dev position then was offered a job as an SRE. It turns out that I don't make a good sysadmin. After I left, I was amazed at how many people I met knew of someone else who had had the same experience. Ironically recruiters see a year as a Google SRE on my resume and think I might want to join SRE teams at other companies. Most of that stopped after I changed my Linked In profile to say that I am never interested in SRE jobs.

deleted 11y ago
[deleted]
fishnchips 11y ago
Yup. Exactly my experience.
tayo42 11y ago
Just curious, what made you think you weren't a good admin?
- btilly 11y ago
  Explaining the concretes would be a lot of unpleasant detail. The why is far more interesting. I am very good at "tunnel vision". Taking a thread and following it in depth. Completely learning a system in depth so that anything that comes up, I know exactly where to go and what to do. I am weak on "peripheral vision". Keeping track of 15 balls in the air, and operating with limited information about each. Being effective with limited context about each individual system because there isn't time to learn any of them in depth. Tunnel vision is a virtue in a software developer. Peripheral vision is essential for a sysadmin. Now imagine a person with poor peripheral vision trying to learn a system as big and complex as Google's. And being responsible to support 15 different pieces of software written by 15 different teams so that when anything went belly up with any of them, you can trouble-shoot and get it running again. Nothing particularly bad happened, but I also wasn't accomplishing the job to the standards that Google wanted in that role. And I was not the only person on my team failing in that way. Ideally it would have been my manager's job to say, "I recognize that this person isn't working out here, is there a better role for them?" That didn't happen. Several months later my manager got fired. I was privately told that my situation was a trigger, but I don't know details. The fact that my situation is fairly common strongly suggests that the ultimate failure was organizational and systemic. I don't fault Google for hiring developers into their hybrid developer/sysadmin role. I do fault them for not having an explicit onramp/reconsideration process to mitigate the risk that they create by doing so.

bovermyer 11y ago

I don't work for Google, but I did follow that career path - started as a developer, then moved into operations. I didn't immediately become an expert as soon as I made the transition - it was a slow process, and I'm still not what I'd consider the equal of a veteran sysadmin in terms of low-level system knowledge. However, I can hold my own in either discipline.

mehta 11y ago

They kinda do. Over time, the skills of a dev and SRE diverge a bit but at the core they are the same. I am a dev at Google but went through a program called Mission Control[1]. (It was a great learning experience and helped me do my job as a dev much better.) As others have said, there is a pretty intensive training that you go through and obviously you learn a lot when you are handling things... [1] http://googleresearch.blogspot.com/2012/07/site-reliability-engineers-solving-most.html http://googleresearch.blogspot.com/2012/07/site-reliability-...

fishnchips 11y ago

You're quite right. I'm an ex-Googler and SRE was the last thing I did at Google. As a first-and-foremost programmer I was bored out of my wits and frustrated by the fact that while I was writing repetitive monitoring rules actual development was happening somewhere else and I had little impact on the design of the system. Granted, a lot of folks over in SRE loved it but I would venture to say that most of them were much closer to being a sysadmin than programmer. Folks like myself tended to 1) move to SWE, 2) quit and 3) some endured and became managers over time.

nunez 11y ago

They look for someone that is strong at operating systems and networking knowledge AND someone that's a decently strong programmer. IOW, someone that can pass a low-level ops interview and a Google SWE interview. This is a tough position to fill but I've worked with folks who are genuinely good at both, so they're out there.

slyall 11y ago
What I'm wondering is have they created a model that only works if you employee "top 5 percent" people. Sort of like if long-haul truckers needs the skills and training of Fighter pilots.
- tamana 11y ago
  Yes. What you get is a lot of "grunge work" that never gets done because everyone is busy inventing the next gen cure-all.