8 ms·

I'll Probably Never Hire Another Pure SysAdmin

92 points by nemesisj 15y ago

a2tech 15y ago
That might work for him running a small business. How do you think those images you 'just spin up' at a cloud provider are made? Who makes sure the boxes that run those images you've just brought up have power and run smoothly?
- shuw 15y ago
  Yes, but those procedures will become increasingly automated and each sysadmin will manage orders of magnitude greater number of servers.
- jwatte 15y ago
  His point is that Amazon hires sysadmins, not him. However, that seems based on a very narrow definition or understanding of what a "sysadmin" is or does for the business. I identify "sysadmin" with "operations" but perhaps others see it differently.
imroot 15y ago
I'm not completely sure that I agree. While this might be great for small startups, once you get past the growth of needing more than a few dozen servers, your cloud and other computing costs are going to be exponentially higher than what they would if you had actually purchased the hardware from your favorite hardware vendor, hired a competent sysadmin, and managed the hardware (and storage/network) in-house. Currently, at $DAYJOB, our database is growing by about 1GB/day, and our storage is growing about 4GB/day -- in a downward housing market. If/when the housing market kicks up, I expect to see those double, if not triple. Having someone in house who can keep the right people informed of the risk (of running out of space, of buying the wrong equipment) who is focused on keeping of the needs of the company balanced with the needs of the software is critical -- do you honestly think your cloud provider cares if your customers can't access their data during an outage? If you're a small fish...probably not... EDIT: Spelling/grammar change.
- TomOfTTB 15y ago
  I agree with you but I'll go even further and say this guy has no idea what he's talking about. As someone whose been in charge of a fairly small cloud outlay (8 virtual servers, 2 of which go off at night) I can tell you the cloud introduces its own set of challenges. So there's less hardware replacement and more DNS configuration but none the less there's still a job for system administration. In fact, I'd say the role of administration becomes even more significant because the cloud industry is so new. Hardware has, in many ways, become pretty rock solid in the last decade. We're to the point where a hard drive in a RAID setup can go out and the users won't even notice. The same can't be true for things like SQL Server Federation and caching servers which used to be the exclusive domain of large web sites but are increasingly being used by smaller companies who don't have a whole staff to dedicate to their upkeep. (One Caveat: If cloud models like Microsoft's Azure take over this could change. By that I mean clouds that are literally frameworks in themselves and not virtual machines)
  - coenhyde 15y ago
    You have to make sure you use the right tools. If you re-invent the wheel a dozen times, managing cloud deployments will be a nightmare. Tools such as Rightscale or Scalr.net are well with the money. I manage 2 dozen Ec2 instances of various types as a side task to my regular development job. I script everything. When things are scripted they are consistently reproducible and fast. Humans make mistakes (including me). I would prefer there to be a dedicated SysAdmin but there just wouldn't be a full time job for him to fill.
amcintyre 15y ago
"Now, the watchword to the development team is: it's not done until I can one-click deploy it." I have apparently been spoiled somewhere along the line, because I really don't like having clumsy manual processes that involve software/computers. I was recently tasked with figuring out how some researcher's code worked (with an eye towards automating the use of their tools), and it took 8 pages to get most of the steps documented well enough that I could follow them. Steps like, "edit the path in line 977 to point to your input files, recompile, run, click button A..." -- i.e., things that should never have been done manually to start with. How on earth people whose primary job involves writing code to do things for them can fail to automate tedious (and frequently error-prone) parts of their job is beyond me. That said, many companies are still going to need to hire somebody to write the one-click deployment. Somebody will have to know how to interact with the cloud services, and how to get around all the little minefields and gaps left in the cloud service offerings, and odds are you aren't going to want to pay that person as much as you pay your developers that do the "real work." That's probably where the sysadmins will go.
- gridspy 15y ago
  It's simple : The people you are complaining about set these things up once on their computer and then forget about it. They don't realise how many things they have set up once along the years. It is not until you appear that it's time for the app to be deployed elsewhere.
  - amcintyre 15y ago
    Actually, no: in this case, the tools were used fairly frequently.
wccrawford 15y ago
With any non-trivial assortment of servers, you need a dedicated sysadmin. The security issues alone should dictate this, but your developers don't need to be worrying about sysadmin stuff. They need to be coding. I totally agree with proper deployment procedures. But having them doesn't eliminate the sysadmin at all. Sysadmin should be deploying standard packages that are necessary for the system, along with updates for them. Developers should be deploying in-house code, and updates for that. I worked at a company that hired a sysadmin too early. When he quit, I ended up doing his duties and mine, too. Of course, I automated 99% of it and everyone was happy. That was really early in the life of the company. Later, when it came necessary to hire one, they didn't. They'd expect a developer to deal with the issues, and were always dissatisfied with the time it took, and the lackluster results. Eventually it got so bad they hired a dedicated sysadmin again. Now, unlike the past, they actually needed one and he was busy all the time. Developers stopped being unproductive for weeks at a time, working on projects they didn't understand and couldn't complete satisfactorily. For a startup, yes, skip the sysadmin at first. But once you have a complex system, hire one. It'll save your company.
- illumin8 15y ago
  +1 - you can automate deployment all you want, but when something goes wrong with your production site (and it inevitably will), do you want to pull your top developers off of whatever new features they are working on and ask them to troubleshoot and fix production? Sysadmins will always be a necessary evil when you are dealing with hundreds of servers, physical, virtual, or cloud. The tools and process will change. We have gone from physical Unix boxes (Sun, HP, etc) to virtual Linux boxes, to cloud servers, but sysadmins are still doing the heavy lifting of designing and building highly scalable systems. I would say that sysadmins should look at moving into systems architecture roles, since one sysadmin will be able to manage hundreds of systems rather than a few dozen in the future. You really need architecture, networking, and performance tuning experience in order to succeed when you are managing more than a handful of systems.
poutine 15y ago
If he'll probably never hire another pure sysadmin then he'll probably never build anything of scale that people will pay for. Sysadmins do a lot of stuff, and this is only somewhat reduced by cloud services. Or is he expecting to get the developers to maintain OS versions, security patches, log monitoring, proxy servers, database replication, backups, firewall rules, failover testing and so on across dozens or hundreds of systems? Right. Absurd.
tptacek 15y ago
People on this thread seem to be objecting to the idea that agile companies should avoid hiring people who manage their systems. But that's not what he's saying! My read is that he's saying that the days of the "system administrator" with a job description that begins and ends with tar flags and Apache vhost configuration are numbered. He's not saying sysadmins are being replaced by Puppet and Chef; he's saying they're being replaced by a new kind of sysadmin that can be expected to extend Puppet and Chef. This seems closely related to a point Patrick McKenzie makes all the time, that being the difference between a "programmer" and "someone who provides business value by means of applying specialized skills with programming languages". Don't be a "programmer".
- jvehent 15y ago
  So, we're playing with the terms ? That's it ? As a sysadmin, I'm fine with the concept of continual improvement, there is nothing worse than a static industry. The job is changing every day and this.is.good. What I understand from this post is that startup people don't want to deal with sysadmin anymore. It makes a lot of sense, considering the amount of time/energy/money you need to run a stable infrastructure. But at the same time, heroku, ec2 & al. will never be able to cover everybody's needs. So while I'm convinced that a dev or devops will be fine deploying his webapp to some random cloud, I'm also convinced they will come crying when massive load hits them and they can't tune the system. The sysadmin is not only the guy who configure new servers, it's also the guy who explains the new dev dude that, no, he can't do 15 inner joins on large tables to create the home page because it will stress the IOs so bad that the whole website will collapse. Ease of use != flexibility.
  - tptacek 15y ago
    If you can explain SQL optimization to a developer, you're doing yourself a huge disservice by saying that you're a "sysadmin".
    - Goladus 15y ago
      No, the problem is arrogant programmers who imagine that sysadmins are people who only know how to configure an apache virtualhost. Although in the case of SQL, specifically, database administration has historically been a specialized sub-category of system administration.
ridruejo 15y ago
I think you need to distinguish here different kind of systems. a) Deployment of 'standard' applications like Drupal, Joomla!, Wordpress, etc. It makes sense to automate it once and let people "self service" In this case either the sysadmin is responsible for packaging those apps or they can get a third party "catalog" that you can deploy to your virtualized/cloud infrastructure (that's exactly what we are aiming for at BitNami, you can check it out http://bitnami.org/stacks/ http://bitnami.org/stacks/) b) Complex multi-tier, in-house applications. Those are much harder to automate because they tend to be "one off" systems and will require 'traditional' system administration for a while. c) Web companies that run a logical application across multiple servers (like Zynga, Netflix, though obviously most people will do so at a smaller scale). Here you need to use something like Chef or Puppet, there's simply no other way of doing so. In other words, before the advent of cloud it was 'optional' for system admins to automate their work. Now it is expected. In that regard, I think automation is the real benefit that cloud computing has brought us (rather than the capex/opex and autoscaling that people often mention)
darrikmazey 15y ago
As is typical with this line of thinking, it only scales so far and only applies as long as you are "on the map" of what is standard. Stray too far and your devops personnel will spend more and more time "administrating" and less time coding. Being primarily a coder with considerable sysadmin skills, this has happened to me in every job I've worked in the last 5 years. I start out devops while everything is standard and slowly do more and more admin until I eventually quit (because I'm primarily a coder). The job I'm quitting this month is because I'm now 95% admin. Like all resources, you need what you need. Devops is sufficient is specific circumstances, but if you stray from that scenario eventually you need an admin. They're the glue when things don't quite fit seamlessly together.
gersh 15y ago
Somebody has to write the deployment scripts, choose a cloud platform, and update the platform. If a cheaper cloud platform comes out, somebody has to figure out which platform is cheaper. You may have automated error logging, but someone has to go through the logging and see what went wrong. Somebody should also look over how reliable the platform is. If your business only requires a standard Rails site, there may standard tools. However, when you develop new technology, someone has to deal with deployment and scalability issues. This may be more of a developer job, but there are significant sysadmin issues.
yxhuvud 15y ago
This is just silly. You will need dedicated sysadmins when your organization gets large enough to support 5-10 nontechies.
pbreit 15y ago
I'd really like to see more activity around creating, publishing and refining Puppet, Chef, Fabric, etc scripts that are usable in various projects. For example, publicly available scripts that approximate what is provided by Heroku and DotCloud.
dfc 15y ago
Who runs the systems at the cloud provider / github or you outsourced DNS/SMTP+IMAP?
latchkey 15y ago
If you look at it from the point of view of creating a new company that is based online, chances are that you can design the system from the ground up to be entirely hosted in the cloud. Even better than AWS is if you can host your site on a PaaS solution such as Google AppEngine. GAE negates the need for a DBA, netadmin and a sysadmin. I'm building a web based company right now with a good friend of mine and we've gladly chosen GAE because of this single attribute. Both of us are 15+ year sysadmins as well as software engineers. That said, instead of worrying about how we are going to scale and run everything, we are able to focus entirely on the development of features and building the company. As a result, we hope to never have to hire any IT positions. In the end, I don't think the roles will go away since someone still needs to run Google. ;-) I just think that the roles will eventually morph a bit for the companies that can take advantage of PaaS solutions.
rythie 15y ago
What's a 'pure sysadmin'? If they DBA are they unpure? or code? or do biz dev? or support? These all seem pretty normal for a sysadmin nowadays. Reminds me of the 'No true Scotsman' thing: http://en.wikipedia.org/wiki/No_true_Scotsman http://en.wikipedia.org/wiki/No_true_Scotsman
jwatte 15y ago
It's not clear to me what you think sysadmins do. Who will make sure your provisioned configurations have proper redundancy? Who will move your domain when an availability zone has an outage? Who will work with upstream networks when you're getting DDoS'd? Who will wake up in the middle of the night when a service you're integrating with is not working? Maybe you've never run a large system at scale, or maybe ou just had bad experiences with "SysAdmins" that didn't, but the role of hand-holding a complex, integrated service is not going away. The fact that you may not need screwdrivers for certain systems is almost beside the point. Nevermind that, when you're at sufficient scale, you can run your own data centers cheaper than Amazon. Break-even happens well before a thousand nodes. (I'm not a sysadmin; i'm an engineer who is very glad we have good sysadmins around :-)
gcv 15y ago
I think even "pure" sysadmins will have jobs until the cloud providers figure out how to solve their IO bottlenecks. I keep hearing horror stories about database performance and how people are forced to go back to good ol' dedicated hardware to solve it.
samarudge 15y ago
I work at a company currently in the process of deploying an Oracle 7 cluster. Regardless of technological advances in startups, enterprise will always need sysadmins, if for nothing other than filling in the correct check boxes on the audit forms.
- chrisbolt 15y ago
  Today's startups are tomorrow's enterprises.
  - samarudge 15y ago
    It depends. If it's still around in 10 years, I'd be willing to bet not a single line of code currently used by Facebook will still be running. Some startups will become "enterprises", because of what they do, but I'm talking about financial services, banks, government departments, public services etc. Their technology doesn't move forwards, because it can't. I'm sure little to no startups are using Oracle IAS (Internet application server), yet it's one of the most used technologies in enterprise applications, and will be for some time. If you're a sysadmin with a deep understanding of IAS, or Oracle DB, or HP-UX, or any of the countless other "old" technologies used in enterprise, you'll have a job for life.
zobzu 15y ago
Who's going to setup puppet? Right. End of story.
mathattack 15y ago
I think it'll be both faster and slower than he says. Faster: Cloud adoption will be huge. Slower: Cloud technology will increase the amount of apps and data used. Low end jobs will exist helping firms get on the cloud. High end jobs (fewer though) will exist at the cloud providers.
dgabriel 15y ago
In 1999, everyone was telling me they'd never need to hire another Java programmer now that Rational Rose would soon be able to spit out complete code based on the input of a team of architects. Just sayin'.
- zalew 15y ago
  yeah, it's like saying you don't need programmers because there's django and rails
inopinatus 15y ago
A development team had a problem. They were in need of a manageable production environment. "I know," the Chief Developer said, "We'll use Chef!". Now they had two problems. However, I agree with the basic contention - that the role of a pure, old-school, risk-averse sysadmin is obsolete. In a world of Agile developers practising continuous delivery (or hoping to), the system engineer is now another cog in the development machine; not part of a separate machine. The resulting closeness does wonders for understanding and quality of the result. Unfortunately, at the time of writing, system engineers who speak Developer are relatively few & far between.
jmspring 15y ago
It is an interesting line of thinking. Way too many startups I've been involved with/talked to have over looked the role of admins/an ops team. The mentality seems to be that developers should also focus as admins/ops. Surely nothing will go wrong on a weekend after a developer has already put in an 80 hour week, right? From my experience, sys admins are important. If you are doing anything of scale, knowledge on how to setup, tweak, and debug systems is critical. It is helpful if the admins can stretch a bit into the app layer, just as it is helpful for developers to be able to stretch a bit into the systems layer, problems come up and knowledge of the parts helps. As an engineer that can also setup systems for scale, I know my limits and really appreciate help from and learning things from good sysadmins.
philwelch 15y ago
Summary: You don't need to hire sysadmins because you can outsource your systems to a cloud provider. This is true, but it's also a lot like saying you don't need to hire developers because you can outsource your development to Lithuania. Really what it means is that it's possible to outsource systems. If you're the little guy and your needs manage to mesh well with what you can get off the shelf from a cloud provider, this is true. Otherwise, this isn't true. If you're the little guy and your needs don't exactly fit what you can get off the shelf, it's going to be trickier. If you're the big guy, you can afford to cut out the middleman.
rythie 15y ago
People seem to think puppet and chef are some sort of revolution, as if we didn't automate all this stuff before with bash, rsync, kickstart and cfengine.
- gaius 15y ago
  That is my experience too. The people here are going crazy for func - and none of them have even heard of Expect!
migrantgeek 15y ago
Operations shouldn't develop and Development shouldn't do operations. That isn't to say that Ops can't program. I write a lot of Python to get things done but that's not Development. I'm a sysadmin in a 100% AWS shop. Before I showed up, Dev was running things. It was horrible. Versions differed greatly, they didn't use init scripts for anything, deployment was pretty manual, and their release cycle was really slow because they spent too much time in operations where they were clueless. My job is to keep production rock solid and provide Dev with the resources they need to write code. If Devs aren't writing code, I failed.

92 points by nemesisj 15y ago

That might work for him running a small business. How do you think those images you 'just spin up' at a cloud provider are made? Who makes sure the boxes that run those images you've just brought up have power and run smoothly?

shuw 15y ago
Yes, but those procedures will become increasingly automated and each sysadmin will manage orders of magnitude greater number of servers.
jwatte 15y ago
His point is that Amazon hires sysadmins, not him. However, that seems based on a very narrow definition or understanding of what a "sysadmin" is or does for the business. I identify "sysadmin" with "operations" but perhaps others see it differently.

imroot 15y ago

I'm not completely sure that I agree. While this might be great for small startups, once you get past the growth of needing more than a few dozen servers, your cloud and other computing costs are going to be exponentially higher than what they would if you had actually purchased the hardware from your favorite hardware vendor, hired a competent sysadmin, and managed the hardware (and storage/network) in-house. Currently, at $DAYJOB, our database is growing by about 1GB/day, and our storage is growing about 4GB/day -- in a downward housing market. If/when the housing market kicks up, I expect to see those double, if not triple. Having someone in house who can keep the right people informed of the risk (of running out of space, of buying the wrong equipment) who is focused on keeping of the needs of the company balanced with the needs of the software is critical -- do you honestly think your cloud provider cares if your customers can't access their data during an outage? If you're a small fish...probably not... EDIT: Spelling/grammar change.

TomOfTTB 15y ago
I agree with you but I'll go even further and say this guy has no idea what he's talking about. As someone whose been in charge of a fairly small cloud outlay (8 virtual servers, 2 of which go off at night) I can tell you the cloud introduces its own set of challenges. So there's less hardware replacement and more DNS configuration but none the less there's still a job for system administration. In fact, I'd say the role of administration becomes even more significant because the cloud industry is so new. Hardware has, in many ways, become pretty rock solid in the last decade. We're to the point where a hard drive in a RAID setup can go out and the users won't even notice. The same can't be true for things like SQL Server Federation and caching servers which used to be the exclusive domain of large web sites but are increasingly being used by smaller companies who don't have a whole staff to dedicate to their upkeep. (One Caveat: If cloud models like Microsoft's Azure take over this could change. By that I mean clouds that are literally frameworks in themselves and not virtual machines)
- coenhyde 15y ago
  You have to make sure you use the right tools. If you re-invent the wheel a dozen times, managing cloud deployments will be a nightmare. Tools such as Rightscale or Scalr.net are well with the money. I manage 2 dozen Ec2 instances of various types as a side task to my regular development job. I script everything. When things are scripted they are consistently reproducible and fast. Humans make mistakes (including me). I would prefer there to be a dedicated SysAdmin but there just wouldn't be a full time job for him to fill.

amcintyre 15y ago

"Now, the watchword to the development team is: it's not done until I can one-click deploy it." I have apparently been spoiled somewhere along the line, because I really don't like having clumsy manual processes that involve software/computers. I was recently tasked with figuring out how some researcher's code worked (with an eye towards automating the use of their tools), and it took 8 pages to get most of the steps documented well enough that I could follow them. Steps like, "edit the path in line 977 to point to your input files, recompile, run, click button A..." -- i.e., things that should never have been done manually to start with. How on earth people whose primary job involves writing code to do things for them can fail to automate tedious (and frequently error-prone) parts of their job is beyond me. That said, many companies are still going to need to hire somebody to write the one-click deployment. Somebody will have to know how to interact with the cloud services, and how to get around all the little minefields and gaps left in the cloud service offerings, and odds are you aren't going to want to pay that person as much as you pay your developers that do the "real work." That's probably where the sysadmins will go.

gridspy 15y ago
It's simple : The people you are complaining about set these things up once on their computer and then forget about it. They don't realise how many things they have set up once along the years. It is not until you appear that it's time for the app to be deployed elsewhere.
- amcintyre 15y ago
  Actually, no: in this case, the tools were used fairly frequently.

wccrawford 15y ago

With any non-trivial assortment of servers, you need a dedicated sysadmin. The security issues alone should dictate this, but your developers don't need to be worrying about sysadmin stuff. They need to be coding. I totally agree with proper deployment procedures. But having them doesn't eliminate the sysadmin at all. Sysadmin should be deploying standard packages that are necessary for the system, along with updates for them. Developers should be deploying in-house code, and updates for that. I worked at a company that hired a sysadmin too early. When he quit, I ended up doing his duties and mine, too. Of course, I automated 99% of it and everyone was happy. That was really early in the life of the company. Later, when it came necessary to hire one, they didn't. They'd expect a developer to deal with the issues, and were always dissatisfied with the time it took, and the lackluster results. Eventually it got so bad they hired a dedicated sysadmin again. Now, unlike the past, they actually needed one and he was busy all the time. Developers stopped being unproductive for weeks at a time, working on projects they didn't understand and couldn't complete satisfactorily. For a startup, yes, skip the sysadmin at first. But once you have a complex system, hire one. It'll save your company.

illumin8 15y ago
+1 - you can automate deployment all you want, but when something goes wrong with your production site (and it inevitably will), do you want to pull your top developers off of whatever new features they are working on and ask them to troubleshoot and fix production? Sysadmins will always be a necessary evil when you are dealing with hundreds of servers, physical, virtual, or cloud. The tools and process will change. We have gone from physical Unix boxes (Sun, HP, etc) to virtual Linux boxes, to cloud servers, but sysadmins are still doing the heavy lifting of designing and building highly scalable systems. I would say that sysadmins should look at moving into systems architecture roles, since one sysadmin will be able to manage hundreds of systems rather than a few dozen in the future. You really need architecture, networking, and performance tuning experience in order to succeed when you are managing more than a handful of systems.

poutine 15y ago

If he'll probably never hire another pure sysadmin then he'll probably never build anything of scale that people will pay for. Sysadmins do a lot of stuff, and this is only somewhat reduced by cloud services. Or is he expecting to get the developers to maintain OS versions, security patches, log monitoring, proxy servers, database replication, backups, firewall rules, failover testing and so on across dozens or hundreds of systems? Right. Absurd.

tptacek 15y ago

People on this thread seem to be objecting to the idea that agile companies should avoid hiring people who manage their systems. But that's not what he's saying! My read is that he's saying that the days of the "system administrator" with a job description that begins and ends with tar flags and Apache vhost configuration are numbered. He's not saying sysadmins are being replaced by Puppet and Chef; he's saying they're being replaced by a new kind of sysadmin that can be expected to extend Puppet and Chef. This seems closely related to a point Patrick McKenzie makes all the time, that being the difference between a "programmer" and "someone who provides business value by means of applying specialized skills with programming languages". Don't be a "programmer".

jvehent 15y ago
So, we're playing with the terms ? That's it ? As a sysadmin, I'm fine with the concept of continual improvement, there is nothing worse than a static industry. The job is changing every day and this.is.good. What I understand from this post is that startup people don't want to deal with sysadmin anymore. It makes a lot of sense, considering the amount of time/energy/money you need to run a stable infrastructure. But at the same time, heroku, ec2 & al. will never be able to cover everybody's needs. So while I'm convinced that a dev or devops will be fine deploying his webapp to some random cloud, I'm also convinced they will come crying when massive load hits them and they can't tune the system. The sysadmin is not only the guy who configure new servers, it's also the guy who explains the new dev dude that, no, he can't do 15 inner joins on large tables to create the home page because it will stress the IOs so bad that the whole website will collapse. Ease of use != flexibility.
- tptacek 15y ago
  If you can explain SQL optimization to a developer, you're doing yourself a huge disservice by saying that you're a "sysadmin".
  - Goladus 15y ago
    No, the problem is arrogant programmers who imagine that sysadmins are people who only know how to configure an apache virtualhost. Although in the case of SQL, specifically, database administration has historically been a specialized sub-category of system administration.

ridruejo 15y ago

I think you need to distinguish here different kind of systems. a) Deployment of 'standard' applications like Drupal, Joomla!, Wordpress, etc. It makes sense to automate it once and let people "self service" In this case either the sysadmin is responsible for packaging those apps or they can get a third party "catalog" that you can deploy to your virtualized/cloud infrastructure (that's exactly what we are aiming for at BitNami, you can check it out http://bitnami.org/stacks/ http://bitnami.org/stacks/) b) Complex multi-tier, in-house applications. Those are much harder to automate because they tend to be "one off" systems and will require 'traditional' system administration for a while. c) Web companies that run a logical application across multiple servers (like Zynga, Netflix, though obviously most people will do so at a smaller scale). Here you need to use something like Chef or Puppet, there's simply no other way of doing so. In other words, before the advent of cloud it was 'optional' for system admins to automate their work. Now it is expected. In that regard, I think automation is the real benefit that cloud computing has brought us (rather than the capex/opex and autoscaling that people often mention)

darrikmazey 15y ago

As is typical with this line of thinking, it only scales so far and only applies as long as you are "on the map" of what is standard. Stray too far and your devops personnel will spend more and more time "administrating" and less time coding. Being primarily a coder with considerable sysadmin skills, this has happened to me in every job I've worked in the last 5 years. I start out devops while everything is standard and slowly do more and more admin until I eventually quit (because I'm primarily a coder). The job I'm quitting this month is because I'm now 95% admin. Like all resources, you need what you need. Devops is sufficient is specific circumstances, but if you stray from that scenario eventually you need an admin. They're the glue when things don't quite fit seamlessly together.

gersh 15y ago

Somebody has to write the deployment scripts, choose a cloud platform, and update the platform. If a cheaper cloud platform comes out, somebody has to figure out which platform is cheaper. You may have automated error logging, but someone has to go through the logging and see what went wrong. Somebody should also look over how reliable the platform is. If your business only requires a standard Rails site, there may standard tools. However, when you develop new technology, someone has to deal with deployment and scalability issues. This may be more of a developer job, but there are significant sysadmin issues.

yxhuvud 15y ago

This is just silly. You will need dedicated sysadmins when your organization gets large enough to support 5-10 nontechies.

pbreit 15y ago

I'd really like to see more activity around creating, publishing and refining Puppet, Chef, Fabric, etc scripts that are usable in various projects. For example, publicly available scripts that approximate what is provided by Heroku and DotCloud.

dfc 15y ago

Who runs the systems at the cloud provider / github or you outsourced DNS/SMTP+IMAP?

latchkey 15y ago

If you look at it from the point of view of creating a new company that is based online, chances are that you can design the system from the ground up to be entirely hosted in the cloud. Even better than AWS is if you can host your site on a PaaS solution such as Google AppEngine. GAE negates the need for a DBA, netadmin and a sysadmin. I'm building a web based company right now with a good friend of mine and we've gladly chosen GAE because of this single attribute. Both of us are 15+ year sysadmins as well as software engineers. That said, instead of worrying about how we are going to scale and run everything, we are able to focus entirely on the development of features and building the company. As a result, we hope to never have to hire any IT positions. In the end, I don't think the roles will go away since someone still needs to run Google. ;-) I just think that the roles will eventually morph a bit for the companies that can take advantage of PaaS solutions.

rythie 15y ago

What's a 'pure sysadmin'? If they DBA are they unpure? or code? or do biz dev? or support? These all seem pretty normal for a sysadmin nowadays. Reminds me of the 'No true Scotsman' thing: http://en.wikipedia.org/wiki/No_true_Scotsman http://en.wikipedia.org/wiki/No_true_Scotsman

jwatte 15y ago

It's not clear to me what you think sysadmins do. Who will make sure your provisioned configurations have proper redundancy? Who will move your domain when an availability zone has an outage? Who will work with upstream networks when you're getting DDoS'd? Who will wake up in the middle of the night when a service you're integrating with is not working? Maybe you've never run a large system at scale, or maybe ou just had bad experiences with "SysAdmins" that didn't, but the role of hand-holding a complex, integrated service is not going away. The fact that you may not need screwdrivers for certain systems is almost beside the point. Nevermind that, when you're at sufficient scale, you can run your own data centers cheaper than Amazon. Break-even happens well before a thousand nodes. (I'm not a sysadmin; i'm an engineer who is very glad we have good sysadmins around :-)

gcv 15y ago

I think even "pure" sysadmins will have jobs until the cloud providers figure out how to solve their IO bottlenecks. I keep hearing horror stories about database performance and how people are forced to go back to good ol' dedicated hardware to solve it.

samarudge 15y ago

I work at a company currently in the process of deploying an Oracle 7 cluster. Regardless of technological advances in startups, enterprise will always need sysadmins, if for nothing other than filling in the correct check boxes on the audit forms.

chrisbolt 15y ago
Today's startups are tomorrow's enterprises.
- samarudge 15y ago
  It depends. If it's still around in 10 years, I'd be willing to bet not a single line of code currently used by Facebook will still be running. Some startups will become "enterprises", because of what they do, but I'm talking about financial services, banks, government departments, public services etc. Their technology doesn't move forwards, because it can't. I'm sure little to no startups are using Oracle IAS (Internet application server), yet it's one of the most used technologies in enterprise applications, and will be for some time. If you're a sysadmin with a deep understanding of IAS, or Oracle DB, or HP-UX, or any of the countless other "old" technologies used in enterprise, you'll have a job for life.

zobzu 15y ago

Who's going to setup puppet? Right. End of story.

mathattack 15y ago

I think it'll be both faster and slower than he says. Faster: Cloud adoption will be huge. Slower: Cloud technology will increase the amount of apps and data used. Low end jobs will exist helping firms get on the cloud. High end jobs (fewer though) will exist at the cloud providers.

dgabriel 15y ago

In 1999, everyone was telling me they'd never need to hire another Java programmer now that Rational Rose would soon be able to spit out complete code based on the input of a team of architects. Just sayin'.

zalew 15y ago
yeah, it's like saying you don't need programmers because there's django and rails

inopinatus 15y ago

A development team had a problem. They were in need of a manageable production environment. "I know," the Chief Developer said, "We'll use Chef!". Now they had two problems. However, I agree with the basic contention - that the role of a pure, old-school, risk-averse sysadmin is obsolete. In a world of Agile developers practising continuous delivery (or hoping to), the system engineer is now another cog in the development machine; not part of a separate machine. The resulting closeness does wonders for understanding and quality of the result. Unfortunately, at the time of writing, system engineers who speak Developer are relatively few & far between.

jmspring 15y ago

It is an interesting line of thinking. Way too many startups I've been involved with/talked to have over looked the role of admins/an ops team. The mentality seems to be that developers should also focus as admins/ops. Surely nothing will go wrong on a weekend after a developer has already put in an 80 hour week, right? From my experience, sys admins are important. If you are doing anything of scale, knowledge on how to setup, tweak, and debug systems is critical. It is helpful if the admins can stretch a bit into the app layer, just as it is helpful for developers to be able to stretch a bit into the systems layer, problems come up and knowledge of the parts helps. As an engineer that can also setup systems for scale, I know my limits and really appreciate help from and learning things from good sysadmins.

philwelch 15y ago

Summary: You don't need to hire sysadmins because you can outsource your systems to a cloud provider. This is true, but it's also a lot like saying you don't need to hire developers because you can outsource your development to Lithuania. Really what it means is that it's possible to outsource systems. If you're the little guy and your needs manage to mesh well with what you can get off the shelf from a cloud provider, this is true. Otherwise, this isn't true. If you're the little guy and your needs don't exactly fit what you can get off the shelf, it's going to be trickier. If you're the big guy, you can afford to cut out the middleman.

rythie 15y ago

People seem to think puppet and chef are some sort of revolution, as if we didn't automate all this stuff before with bash, rsync, kickstart and cfengine.

gaius 15y ago
That is my experience too. The people here are going crazy for func - and none of them have even heard of Expect!

migrantgeek 15y ago

Operations shouldn't develop and Development shouldn't do operations. That isn't to say that Ops can't program. I write a lot of Python to get things done but that's not Development. I'm a sysadmin in a 100% AWS shop. Before I showed up, Dev was running things. It was horrible. Versions differed greatly, they didn't use init scripts for anything, deployment was pretty manual, and their release cycle was really slow because they spent too much time in operations where they were clueless. My job is to keep production rock solid and provide Dev with the resources they need to write code. If Devs aren't writing code, I failed.