7 ms·

I'm a sysadmin for 16 years, I have worked in big and small companies and I don't agree with a great part of this post. I'm not a developer, I have never neede

I'm a sysadmin for 16 years, I have worked in big and small companies and I don't agree with a great part of this post. I'm not a developer, I have never needed to disassemble or fix programs built by others, never needed to understand kernel subsystems or anything else kernel related (except may be how to replace a broken driver/module). I know nothing of C, I know just the basics of system calls and I've never heard of skb or select() and epoll(). I don't even like developing software, which is the reason I once joined the sysadmin team at the company I worked for and fell in love with this job. I see how these skills are a nice extra, but I understand that knowing too much also creates the problem of having to work harder and stay more often after hours. And saying no isn't an option because nobody else knows how to handle the problems that may arise. To me, a good sysadmin knows when to say no and spends the least effort in fixing errors in code. The developers should provide good programs or it doesn't get installed in production. Keep strict boundaries in what you do and don't to prevent being abused by your employer. If you are often the last one to leave because you're doing work that someone else should be doing, or asked to be stand by after hours while being the only sysadmin, step on the brakes before this will eventually burn you out or makes you hate your job (or even hobby). Also, knowing all of what parent describes + all the 'normal' sysadmin knowledge is (eventually) impossible to keep up and will also take too much of your free time. You will regret this later in life.

rdudek 11y ago
I agree. When we hire sysadmins, we look for folks with experience using various products and weigh in extra bonus points for folks who are also SecOps. We could care less if a person knows how to code. That's what we have the developers for.
- marcosdumay 11y ago
  Lots of people agree. Those end up creating environments where everything just sorta-works enough to run, for as long as you put a constant stream of sysadmin time into it. And then as see they doubting people that claim that a person can administer hundreds of servers, or that an admin can go away for a while and nothing will happen. (Also, exactly what do you look for in SecOps? Most people that use that term are selecting for exactly the wrong things.)
ocdtrekkie 11y ago
Absolutely agree. Sysadmins' interface with code is basically to be a compliance-checker. Software, be it from in-house developers or a third party solution need to meet that compliance bar, or it doesn't go to production. If your in-house developers are unable to meet those compliance requirements, that's a them issue and not a you issue. It's good for devs to know some ops, and ops to know some dev from a familiarity standpoint, particularly because mutual understanding fosters a good working relationship, but they're two very different roles.
- lamontcg 11y ago
  Continuous Delivery is going to eliminate that from the equation. The software developer will write software and tests and if they pass the tests throughout the pipeline then it will ship to production automatically and potentially multiple times per day. Much faster than the heavy ITIL based change control process that requires you to be inserted in the middle of it will manage. The companies that do this will have a velocity that outstrips the companies that do what you describe and will naturally have an advantage in the marketplace. There will still be room for security compliance and someone in the company to be responsible for that, but they'll be writing tests that check software for compliance automatically, they won't be some drone in an office looking at a form and stamping their approval on a change request. Your industry is changing. You may retire before it really hits you in the ass and kicks you out, but your advice for people getting into the industry is terrible.
  - exelius 11y ago
    Agreed, but for different reasons. You still need a sysadmin to set up your CI systems in a continuous delivery world. You still need people to debug performance issues -- most developers don't know enough about system I/O to do that effectively. You're right in that there's no longer some heavy-handed change control process, but wrong that sysadmins will go away. We just call them DevOps instead, but it's the same skill set, just embedded inside a dev team.
    - lamontcg 11y ago
jpgvm 11y ago
I guess I come from a different background, work for different companies and have dealt with different problems. I am aware there is plenty of room for the traditional sysadmin in enterprise still but I am not quite sure how long those days will be around. Google and facebook where first to discover the "Site Reliability Engineer" but they were far from the last. SRE is becoming the new sysadmin and you can bet your life on it they expect the skills I outlined.
- lamontcg 11y ago
  I was doing "SRE" level system engineering at Amazon back in 2001 long before facebook ever existed. Also when I grew up as a junior system engineer in the mid-90s I looked up to "system administrators" like Wietse Venema who coded in C and wrote tcp_wrappers and postfix, and Larry Wall who wrote his own programming language. The System Administration / Software Developer is not a new invention of Facebook/Google. What is "new" is the "System Administrator" that started getting hired in the late 90s who couldn't do anything other than install programs and configure them and maybe knew how to 'ps' and 'kill' and that's about it. Even the old school Enterprise-class system admins knew more about debugging crash dumps and using their tools even if they did call up IBM when they needed a patch.
  - mauricemir 11y ago
    at British Telecom ALL off the developers in some engineering centers where expected to be able to do basic admin on their sun work stations.
  - jpgvm 11y ago
    Very true, I should have been more specific when I say "created". Popularised is probably a better term. I worked with some old and crusty Perl and C hackers that called themselves sysadmins that definitely would have qualified as "SRE"s during the 90s, it's just they were taken for granted. All the term SRE has done is create a role that is more appreciated and better compensated with a better understood set of responsibilities and required skills.
lamontcg 11y ago
You are literally the problem with system administration. The attitude that you don't need to know how to program or understand the architecture of the systems that you run on is a highly privileged attitude that you are practitioner who can remain ignorant of their tools. The good news is that kids these days are being raised on SREs and there's a billion people in India who will take your job and won't have an attitude about learning how to program. Ultimately, you are going to be a curious dinosaur. You are a unique product of the late 90s Internet bubble where the need for system administrators expanded so much faster than the available supply and anyone halfway technically minded got hired for really good salaries and put in charge of servers. That is going to change and evolve. And I spend 5-10 years learning all that knowledge back in the 90s and its 15 years later and I can state with confidence that there's no regrets. If anything my only regret is that I didn't learn how to really practice "software development" as opposed to "programming" even earlier. Going forwards, I expect that more an more the lines between systems and software is going to blur and MY advice to kids these days would be that they will only be able to be ignorant of software development practices at their own peril. And I stated doing PC Tech work in the late 80s, became a Unix admin in the mid-90s, managed the configuration management system and base configuration of a site that grew to 30,000 linux servers in the 200Xs, and then after 15 years switched to being a Software Developer. I'm fully confident based on experience that you are offering absolutely terrible advice to someone who is just learning, and you are out of touch with the direction of your own field.
- dang 11y ago
  > You are literally the problem with system administration. No personal attacks on HN, please.
- redwood 11y ago
  Your post seems to boil down to a logical conclusion that everyone better be a software developer, as if there is no value in abstracting to a layer where things aren't pieced together with compiled software. Feels rather myopic to me. There will always be work to do at every layer
fsniper 11y ago
I'm sorry for you, my fellow sysadmin who has more years on his/her shoulders than me. I'm sorry because you lack the the ultimate knowledge needed to be a good system administator. You did not ever get hold of it. The sysadmin practice is not only installing software and OS, doing basic troubleshooting. It's more than that. It's knowing systems inside out, from kernel internals to application internals. It's knowing and handling more area than anyone else in the field. If you do not know about these, you do not automate, yo do not code, you do not debug, you become an IT helpdesk employee.

rdudek 11y ago

I agree. When we hire sysadmins, we look for folks with experience using various products and weigh in extra bonus points for folks who are also SecOps. We could care less if a person knows how to code. That's what we have the developers for.

marcosdumay 11y ago
Lots of people agree. Those end up creating environments where everything just sorta-works enough to run, for as long as you put a constant stream of sysadmin time into it. And then as see they doubting people that claim that a person can administer hundreds of servers, or that an admin can go away for a while and nothing will happen. (Also, exactly what do you look for in SecOps? Most people that use that term are selecting for exactly the wrong things.)

ocdtrekkie 11y ago

Absolutely agree. Sysadmins' interface with code is basically to be a compliance-checker. Software, be it from in-house developers or a third party solution need to meet that compliance bar, or it doesn't go to production. If your in-house developers are unable to meet those compliance requirements, that's a them issue and not a you issue. It's good for devs to know some ops, and ops to know some dev from a familiarity standpoint, particularly because mutual understanding fosters a good working relationship, but they're two very different roles.

lamontcg 11y ago
Continuous Delivery is going to eliminate that from the equation. The software developer will write software and tests and if they pass the tests throughout the pipeline then it will ship to production automatically and potentially multiple times per day. Much faster than the heavy ITIL based change control process that requires you to be inserted in the middle of it will manage. The companies that do this will have a velocity that outstrips the companies that do what you describe and will naturally have an advantage in the marketplace. There will still be room for security compliance and someone in the company to be responsible for that, but they'll be writing tests that check software for compliance automatically, they won't be some drone in an office looking at a form and stamping their approval on a change request. Your industry is changing. You may retire before it really hits you in the ass and kicks you out, but your advice for people getting into the industry is terrible.
- exelius 11y ago
  Agreed, but for different reasons. You still need a sysadmin to set up your CI systems in a continuous delivery world. You still need people to debug performance issues -- most developers don't know enough about system I/O to do that effectively. You're right in that there's no longer some heavy-handed change control process, but wrong that sysadmins will go away. We just call them DevOps instead, but it's the same skill set, just embedded inside a dev team.
  - lamontcg 11y ago

jpgvm 11y ago

I guess I come from a different background, work for different companies and have dealt with different problems. I am aware there is plenty of room for the traditional sysadmin in enterprise still but I am not quite sure how long those days will be around. Google and facebook where first to discover the "Site Reliability Engineer" but they were far from the last. SRE is becoming the new sysadmin and you can bet your life on it they expect the skills I outlined.

lamontcg 11y ago
I was doing "SRE" level system engineering at Amazon back in 2001 long before facebook ever existed. Also when I grew up as a junior system engineer in the mid-90s I looked up to "system administrators" like Wietse Venema who coded in C and wrote tcp_wrappers and postfix, and Larry Wall who wrote his own programming language. The System Administration / Software Developer is not a new invention of Facebook/Google. What is "new" is the "System Administrator" that started getting hired in the late 90s who couldn't do anything other than install programs and configure them and maybe knew how to 'ps' and 'kill' and that's about it. Even the old school Enterprise-class system admins knew more about debugging crash dumps and using their tools even if they did call up IBM when they needed a patch.
- mauricemir 11y ago
  at British Telecom ALL off the developers in some engineering centers where expected to be able to do basic admin on their sun work stations.
- jpgvm 11y ago
  Very true, I should have been more specific when I say "created". Popularised is probably a better term. I worked with some old and crusty Perl and C hackers that called themselves sysadmins that definitely would have qualified as "SRE"s during the 90s, it's just they were taken for granted. All the term SRE has done is create a role that is more appreciated and better compensated with a better understood set of responsibilities and required skills.

lamontcg 11y ago

You are literally the problem with system administration. The attitude that you don't need to know how to program or understand the architecture of the systems that you run on is a highly privileged attitude that you are practitioner who can remain ignorant of their tools. The good news is that kids these days are being raised on SREs and there's a billion people in India who will take your job and won't have an attitude about learning how to program. Ultimately, you are going to be a curious dinosaur. You are a unique product of the late 90s Internet bubble where the need for system administrators expanded so much faster than the available supply and anyone halfway technically minded got hired for really good salaries and put in charge of servers. That is going to change and evolve. And I spend 5-10 years learning all that knowledge back in the 90s and its 15 years later and I can state with confidence that there's no regrets. If anything my only regret is that I didn't learn how to really practice "software development" as opposed to "programming" even earlier. Going forwards, I expect that more an more the lines between systems and software is going to blur and MY advice to kids these days would be that they will only be able to be ignorant of software development practices at their own peril. And I stated doing PC Tech work in the late 80s, became a Unix admin in the mid-90s, managed the configuration management system and base configuration of a site that grew to 30,000 linux servers in the 200Xs, and then after 15 years switched to being a Software Developer. I'm fully confident based on experience that you are offering absolutely terrible advice to someone who is just learning, and you are out of touch with the direction of your own field.

dang 11y ago
> You are literally the problem with system administration. No personal attacks on HN, please.
redwood 11y ago
Your post seems to boil down to a logical conclusion that everyone better be a software developer, as if there is no value in abstracting to a layer where things aren't pieced together with compiled software. Feels rather myopic to me. There will always be work to do at every layer

fsniper 11y ago

I'm sorry for you, my fellow sysadmin who has more years on his/her shoulders than me. I'm sorry because you lack the the ultimate knowledge needed to be a good system administator. You did not ever get hold of it. The sysadmin practice is not only installing software and OS, doing basic troubleshooting. It's more than that. It's knowing systems inside out, from kernel internals to application internals. It's knowing and handling more area than anyone else in the field. If you do not know about these, you do not automate, yo do not code, you do not debug, you become an IT helpdesk employee.