6 ms·

It very much is about the terms. 20-25 years ago a sysadmin was expected to be able to fix C bugs, and have skills not that far removed from programming in as

by tytso 15y ago

It very much is about the terms. 20-25 years ago a sysadmin was expected to be able to fix C bugs, and have skills not that far removed from programming in assembly (i.e., sendmail.cf files :-). About ten years ago I remember talking to some old timers who were surprised that a lot of the younger sysadmins no longer even did any perl progamming, but rather did things like program Cisco rounters and set up Oracle servers. At Google we don't have sysadmins at all, but rather "Site Reliability Engineers", and they are expected to be able to answer some fairly deep C/C++ data structure and algorithms questions. All of those skills are necessary to run a Google-sized infrastructure, and it's quite a far cry from the sysadmins that don't even write perl scripts any more -- and yet feel perfectly free to call themselves "sysadmins". When I first started, I both wore a pager and wrote C programs --- including the 'ninit' program that was described in the Unix haters handbook. The fact that I was one of the people who might be woken up at 3am when the mail server melted down due to named core dumping informed how I wrote my C programs (i.e., very defensively), and is a prime example of the dev/ops concept.

jvehent 15y ago
So we agree on the definition of what a sysadmin, systems engineers, devops or whatever people want to call it should be. A sysadmin must program, that's a given. When did we forget that exactly ? But why are so many devs shooting at the sysadmins these days ? I never read anything from sysadmins complaining about the astonishingly low level of knowledge of a lot of devs. And they are numerous, willingly ignoring even the most basic performance issues and vomiting thousands of lines of codes, hundreds of tons of libraries, to simple problems. It seems that devs are frustrated to not be able to push their random app to the web in 5 minutes, and throw their love to PaaS like an apple geek discovering the new iphone, blaming the sysadmins for not providing the same thing at their own company without even understanding (or trying to understand) how operating systems work, and how their own code should be built. The real issue here is that when devs and sysadmins DON'T work together, and ignore each others problems, constantly blaming each other, then the result sucks. When the sysadmins are 20% devs, and the devs 20% sysadmins, and they drink their coffees together in the morning, you don't see that kind of problem. They built good, reliable, scalable infrastructure, whether it's in the cloud or in the basement of their house.
- eropple 15y ago
  I never read anything from sysadmins complaining about the astonishingly low level of knowledge of a lot of devs. They should be. At my current employer we're pretty good about this, but in previous misadventures I've run into a lot of people who don't even understand HTTP or how a web server works (if I have to explain thread-per-connection to a guy who's been doing web development for five years, there's a problem!).
  - jrussbowman 15y ago
    I used to think like that. But then, doing web development myself on the side I've realized no, they don't. Languages and frameworks these days pretty well abstract the underlying technology. Developers end up writing some pretty amazing stuff. It's up to the sysadmin to weigh in on what changes are necessary to make it work well enough to succeed. A seasoned developer will know more, because they've been through the wringer a few times. The new guys, not so much. Remember, your developer spends their time making it work on a dev box, or even their own laptop. Performance testing and such is an art that if they had time to learn it... well I guess they'd be a sysadmin.
    - eropple 15y ago
      I am that new-guy web developer (or I was a year ago, anyway, I graduated college in 2010). I am the developer working on a dev box, and I do do performance testing. I knew it then, because it's important. Web frameworks do not meaningfully abstract the underlying workings of a web server; hell, the inability to do so is why C# 5 has the await keyword (to allow for better asynchronous IO) and why Servlet 3 has its own async stuff (not to mention Play Framework, which does all manner of shenanigans under-the-hood!). If you actually give a shit about web development beyond a paycheck, this is stuff you just gotta know. If you don't understand the system, your assumptions and assertions are so much more likely to be untenable that it becomes dangerous. The below questions wouldn't fly at my current employer due to a lot of legacy concerns, but my own list of interview questions would include a number of stack questions in addition to programming questions: -Explain asynchronous I/O to me. -How do you have to structure your code to leverage asynchronous I/O? (Alternately phrased: what happens if you write code that depends on synchronous I/O operations within an asynchronous I/O web framework?) -You've got 9 memcached servers with a cache hit rate of 80% resulting in a database load of 80%. What happens if you naively add another memcached server to your cluster? (a question our VP of engineering actually does ask, and I stole it 'cause it's great) You don't have to be a hardcore nginx source code diver to understand your stack. You just have to understand the principles behind it well enough to make good decisions. A sysadmin's job isn't to audit your code for your terrible understanding of the model on which your stack is predicated.
Goladus 15y ago
> It very much is about the terms. 20-25 years ago a sysadmin was expected to be able to fix C bugs, and have skills not that far removed from programming in assembly (i.e., sendmail.cf files :-). I would say, 25 years later, the need to fix C bugs is far less relevant. That said, I'd say around 50% of the system administrators I've worked since moving into the Unix world know C and can at least read the source code of the operating system they're supporting. Even if there's not much need to write anything or even submit bugfixes anymore. Also, the number of unix admins I know who don't know a scripting language well enough to write a reasonably-sized, useful tool is vanishingly small, and those few have been very solid with some other technology(network, storage, etc.) and valuable anyway. "System Administrator" is a broad term, just like "Developer." I'm not sure google has either, right? If we're going by fancy titles, Google has "Software Engineers." I'm not going to criticize Google's culture but the fact is if I say "Software Developer" you know what I'm talking about and from context should be able to tell the difference between Linus Torvalds and a guy who creates wordpress sites for his friends.
rachelbythebay 15y ago
Sorry, even there, you have sysadmins who just sit there and run through playbooks and never improve anything. This is how you have teams with 70 pages in a 9 hour daytime shift as a matter of course. There are pure SAs around, and I don't just mean the caste system. The entire sysops organization was just one example. They may have been rolled up into another organization but the people still exist.

by tytso 15y ago

jvehent 15y ago

So we agree on the definition of what a sysadmin, systems engineers, devops or whatever people want to call it should be. A sysadmin must program, that's a given. When did we forget that exactly ? But why are so many devs shooting at the sysadmins these days ? I never read anything from sysadmins complaining about the astonishingly low level of knowledge of a lot of devs. And they are numerous, willingly ignoring even the most basic performance issues and vomiting thousands of lines of codes, hundreds of tons of libraries, to simple problems. It seems that devs are frustrated to not be able to push their random app to the web in 5 minutes, and throw their love to PaaS like an apple geek discovering the new iphone, blaming the sysadmins for not providing the same thing at their own company without even understanding (or trying to understand) how operating systems work, and how their own code should be built. The real issue here is that when devs and sysadmins DON'T work together, and ignore each others problems, constantly blaming each other, then the result sucks. When the sysadmins are 20% devs, and the devs 20% sysadmins, and they drink their coffees together in the morning, you don't see that kind of problem. They built good, reliable, scalable infrastructure, whether it's in the cloud or in the basement of their house.

eropple 15y ago
I never read anything from sysadmins complaining about the astonishingly low level of knowledge of a lot of devs. They should be. At my current employer we're pretty good about this, but in previous misadventures I've run into a lot of people who don't even understand HTTP or how a web server works (if I have to explain thread-per-connection to a guy who's been doing web development for five years, there's a problem!).
- jrussbowman 15y ago
  I used to think like that. But then, doing web development myself on the side I've realized no, they don't. Languages and frameworks these days pretty well abstract the underlying technology. Developers end up writing some pretty amazing stuff. It's up to the sysadmin to weigh in on what changes are necessary to make it work well enough to succeed. A seasoned developer will know more, because they've been through the wringer a few times. The new guys, not so much. Remember, your developer spends their time making it work on a dev box, or even their own laptop. Performance testing and such is an art that if they had time to learn it... well I guess they'd be a sysadmin.
  - eropple 15y ago
    I am that new-guy web developer (or I was a year ago, anyway, I graduated college in 2010). I am the developer working on a dev box, and I do do performance testing. I knew it then, because it's important. Web frameworks do not meaningfully abstract the underlying workings of a web server; hell, the inability to do so is why C# 5 has the await keyword (to allow for better asynchronous IO) and why Servlet 3 has its own async stuff (not to mention Play Framework, which does all manner of shenanigans under-the-hood!). If you actually give a shit about web development beyond a paycheck, this is stuff you just gotta know. If you don't understand the system, your assumptions and assertions are so much more likely to be untenable that it becomes dangerous. The below questions wouldn't fly at my current employer due to a lot of legacy concerns, but my own list of interview questions would include a number of stack questions in addition to programming questions: -Explain asynchronous I/O to me. -How do you have to structure your code to leverage asynchronous I/O? (Alternately phrased: what happens if you write code that depends on synchronous I/O operations within an asynchronous I/O web framework?) -You've got 9 memcached servers with a cache hit rate of 80% resulting in a database load of 80%. What happens if you naively add another memcached server to your cluster? (a question our VP of engineering actually does ask, and I stole it 'cause it's great) You don't have to be a hardcore nginx source code diver to understand your stack. You just have to understand the principles behind it well enough to make good decisions. A sysadmin's job isn't to audit your code for your terrible understanding of the model on which your stack is predicated.

Goladus 15y ago

> It very much is about the terms. 20-25 years ago a sysadmin was expected to be able to fix C bugs, and have skills not that far removed from programming in assembly (i.e., sendmail.cf files :-). I would say, 25 years later, the need to fix C bugs is far less relevant. That said, I'd say around 50% of the system administrators I've worked since moving into the Unix world know C and can at least read the source code of the operating system they're supporting. Even if there's not much need to write anything or even submit bugfixes anymore. Also, the number of unix admins I know who don't know a scripting language well enough to write a reasonably-sized, useful tool is vanishingly small, and those few have been very solid with some other technology(network, storage, etc.) and valuable anyway. "System Administrator" is a broad term, just like "Developer." I'm not sure google has either, right? If we're going by fancy titles, Google has "Software Engineers." I'm not going to criticize Google's culture but the fact is if I say "Software Developer" you know what I'm talking about and from context should be able to tell the difference between Linus Torvalds and a guy who creates wordpress sites for his friends.

rachelbythebay 15y ago

Sorry, even there, you have sysadmins who just sit there and run through playbooks and never improve anything. This is how you have teams with 70 pages in a 9 hour daytime shift as a matter of course. There are pure SAs around, and I don't just mean the caste system. The entire sysops organization was just one example. They may have been rolled up into another organization but the people still exist.