10 ms·

The Ops Identity Crisis

161 points by aberoham 10y ago

llama052 10y ago
I've always seen operations folks in a different light then most developers. Maybe I'm an exception but I've always viewed operations people as having in-depth knowledge of infrastructure on the lower layers (Session layer and lower). They are able to walk through the inner-workings of network concepts, operating systems, etc. They can implement and design systems that work in a highly available, scalable way. Not saying developers CANT do this, but a majority of them don't want to, and don't have the first idea on how to. Granted some "10x" developers who have full stack knowledge can do just as well, but let's be honest most people aren't 10x developers, they want to mess with the code stack and that's it. This reminds me of the all the job posting you see online that want you to have expert knowledge of C++ expert in SQL DBA expert in networking expert in web design expert in big data Yeah you might be able to find someone who can do all of the above, but odds are they won't do them all well. I don't think the expectation should be set that developers should have to manage the systems stack, as well as manage the code. Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production. Even with configuration toolsets, and infrastructure moving to "code-to-deploy" solutions. Things will still break, unusual things will still happen. Taking developers out of their zone to focus on problems will slow down the entire company. Ops can can always push to 'strive to automate themselves out of their jobs' but I'd argue that this is an endless job which always has outlets that you can continue to strive and build into. Of course, this is talking from my personal experience, I've never worked in a big company like Uber, so the environment might be entirely different from my own.
- morgante 10y ago
  > Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production. Really? As a dev, the vast majority of my projects/roles have required all 3 to some extent. I have been on call for the past 4 years. Honestly, people really need to admit that the age of the sysadmin (who is not also a strong programmer) is over. I might not have enough deep knowledge to answer network/OS trivia off the top of my mind, [0] but I'm perfectly capable of running a production build for my code (with monitoring, configuration management, etc.) and finding appropriate resources in the instances where problems are outside my domain. [0] https://news.ycombinator.com/item?id=12701272 https://news.ycombinator.com/item?id=12701272
  - llama052 10y ago
    > Really? As a dev, the vast majority of my projects/roles have required all 3 to some extent. I can run photoshop and mock some images at home and it's good enough, but does that make me a photoshop guru? I think not. edit: bad example.
    - morgante 10y ago
      > To some extent maybe, but running something on a heroku engine Talk about a straw man. I would agree with you that deploying to Heroku hardly constitutes ops. I'm talking about things like running a multi-AZ Kubernetes cluster.
      - llama052 10y ago
        Following a guide online and getting something online is different from running something in production that a company relies on. Learning a infrastructure piece thoroughly can take months alone in resilience testing, failover testing, and system design/architecture choices. It's a huge fallacy in the infrastructure side of things that because something "works" means it's setup correctly, and that couldn't be further from the truth. You'll get an environment built on technical debt. Outages will occur and you'll be playing sysadmin full time before long. You won't be a developer at this point anymore, you'll be the sysadmin/ops role of your company. I've seen this happen at multiple companies, and I doubt this is the last time I will see it happen.
        morgante 10y ago
        You continue to be extremely insulting and making significant assumptions about my experience. What shibboleth do I have to provide to prove that I know what I'm doing? This inherent assumption that other people don't know anything is incredibly damaging to the industry: I don't assume a SRE is an incompetent programmer, so you shouldn't assume I am barely competent at ops. > Following a guide online and getting something online is different from running something in production that a company relies on Where did I say that I had simply "followed a guide online" instead of "putting something in production." Like I said in my original comment, I have been the primary on call engineer for multiple projects with millions of users. Despite that, I still find time to develop new features and push the product forward. Yes, I occasionally deal with outages/failures and have to write some new automation to remediate them. But that doesn't make me a full-time sysadmin. I don't really feel like engaging with you further, since you're apparently more interested in arguing with a straw man of your own construction.
- toast0 10y ago
  > Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production. "sorry about the deadline, I had to make the fine network work" gets you out of making deadlines. The nice thing about being on call for your own code is that fixing your stuff at all hours is a natural consequence of writing fragile stuff. The not nice thing is that not all of the reasons your stuff breaks are your fault and some issues aren't realistically preventable.
  - llama052 10y ago
    I can understand being on call for your own code, but often that's just not the case. Infrastructure can go off the rails as well, clusters can go on the fritz, upstream might be having issues, a release update on your infrastructure side might introduce problems. I just don't see that falling on the developers who also must maintain the code-base for the product.
    - pjscott 10y ago
      That just means that being on-call for your own code results in good code, a love for automated testing, and a seething grudge against EC2 and its tendency to have multi-zone outages at 3 AM. It doesn't matter what time zone you're in, by the way; the outage will always, always page you at 3 AM.
    - toast0 10y ago
      If the issues for things that aren't your fault happen are infrequent, then it's not a huge burden, and developers may be able to help with mitigation: ex if your internet bandwidth is cut in half because of a fiber cut, how can you move or shed load to have a stable, degraded system. If the issues are frequent, then it provides incentive to build fault tolerance. (And also seething grudges for fault prone infrastructure)
- donavanm 10y ago
  > Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production. This seems to be the meme of "the supporting infrastructure always breaks and causes the pager engagements!" I have seen those orgs and times where networking or facility failures are the leading cause of outages. Long term those are symptoms of chronic underinvestment and tech debt accumulation. I also see, on a daily basis, a massive business where the absolute leading cause of "outage minutes" is software defect or deficiency. With investment in supporting infrastructure you're paged 10 times for a defect your team "developed" for every one unavoidable dependency failure. Big companies can make those (huge) investments themselves. Small companies can pay someone else for access to theirs. In any case its always a business decision, not a certainty.
- davidgerard 10y ago
  I explain my job to nongeeks as "computer roadie". My job is to make sure everything is in order, devs' job is to get up there and be Eric Clapton. People can do both (many roadies are really quite capable musicians). But they're fundamentally different mindsets, and expertise in one is not transferable to the other.
jzelinskie 10y ago
If you liked this post and want to know the full details to this change in ops, please read the SRE book[0]. It's a great read for both devs and ops and can immediately help you make changes to company policy for the better. [0]: http://shop.oreilly.com/product/0636920041528.do http://shop.oreilly.com/product/0636920041528.do
_qc3o 10y ago
I look forward to the day when every software developer has half a clue about monitoring, logging, high availability, configuration management, orchestration/scheduling, performance tuning, build/deployment pipelines, data management/archiving, security and exploit mitigation, etc. The full-stack developer like the serverless/ops-less future is a pipe dream. Most technology organizations barely even know how to build the software to begin with let alone figure out the right way to operate it.
- toomuchtodo 10y ago
  The startup scene is rife with overconfidence that you have enough waking hours to know everything you listed in one role. Regret I have but one upvote for you.
  - hibikir 10y ago
    The issue is that some people can spend just enough hours to make it LOOK like they really understand enough, when in practice, they have wide breadth, but not anywhere enough depth to do the job well. When the startup becomes mid size, suddenly you have a spaghetti codebase with unfortunate tradeoffs, a wide infrastructure full of hard coding that badly attempts to mimic the state of the art of 5 years ago, and one nine of reliability. All of this is of course undocumented, so when you bring in new people, they take forever to learn the existing system, as nothing resembles good practices. As those new people try to clean everything up, they look far less productive than those people that have made this gigantic mess in the first place, and then everyone wonders how they got so lucky with their early, uberproductive employees, which seem to be so much better than anyone else. If the startup has a good economic situation, they might survive while compounding the problem by overhiring (I am sure you all have heard the stories). If the finances aren't quite as good, then the startup folds. I see roads out of this, but they all pass by different early employee compensation: A good early employee is not a good employee in a midsized company, but stock options don't work all that well when early engineers should be leaving the company 3 years in, before exercising the options makes any sense.
    - 10y ago
advisedwang 10y ago
There is a trade off between development velocity and reliability. Not just because making more changes often breaks more things, but also in how much investment is made in infrastructure, monitoring. It even shows up in design choices - making a service multi-regional is more effort but results in a more stable system. A big part of SRE is motivation for reliability. Part of the reason to have SRE is to have engineers motivated by stability, not development velocity. Have somebody who knows the better choice for reliability, even if at the time a decision is made the other way. This necessitates both a separate profession and organisation separation (e.g. for product readiness reviews or for higher level stability decision making). (NB: I am a Googler that works closely with SRE, but not an SRE myself)
- dkarapetyan 10y ago
  I think this is a false dichotomy. If developers are incentivized to make stable software then they'll make stable software but that's not the case. Software engineers that work on products are promoted based on number of features they ship, not how many production outages they don't cause. It's like the senator that lobbies to put bolted doors on plane cockpits before 9/11. That senator will get zero credit for anything. Fundamentally it is harder to measure the effectiveness of preventive measures so most organizations don't and instead settle for number of features shipped.
  - pmoriarty 10y ago
    Don't forget all the death marches to meet unrealistic deadlines that some exec or sales rep pulled out of their ass. In far too many organizations devs are running around like the proverbial headless chickens to ship! ship! ship! while ops are endlessly fighting fire after fire. Many teams don't have the time or manpower to afford the luxury of being proactive, and when anyone suggests doing so, tries to put the breaks on, or asks for more resources they're treated like troublemakers.
sjg007 10y ago
developing on top of managed services will allow you to do more with less people over time. And adopt best practices by integration.
latch 10y ago
I agree. Quality is improved as developers start to understand, get involved in, and own the operation side of running their system. Ops ought to be an enabling force. But, I do wish that anytime anyone writes about ops or infrastructure, they put a big headline: There's a 99% chance you just just Scale Up and use Bash.
solipsism 10y ago
I don't know who writes/says dumb things like "ops is dead" but ideally we wouldn't waste time arguing against such reductive statements. Whatever the zeitgeist, there will always be homogeneous groups (all developers know/do roughly the same things), structurally heterogenous groups, and organically heterogenous groups.
jimjimjim 10y ago
back before the agile dark ages, things like performance, logging/instrumentation, stability guidelines, security guidelines and deployment abilities were able to be specified as non-functional requirements and the product was QA'd for these just as much as the features that were added.
- kasey_junk 10y ago
  Blaming agile for this seems odd. I've worked on agile teams where all of those were set as requirements.
deleted 10y ago
[deleted]
sigil 10y ago
It feels to me like title-centric corporate culture might be a part of the problem here. Are you an SRE? An SWE? Do you Build? Or do you Run? Are you a Software Developer or a Software Engineer? They're interchangeable to the author, but she includes both on the off chance that you the reader are sensitive to some fine splitting / quantum structure... I for one welcome the blurring boundaries. It feels weird to say, sure, I wrote this code with pathological runtime behavior, but it's your problem now, Ops Person. How do you learn to avoid that mistake if someone else absorbs the pain for you (ie learns your lesson)? Personally I feel a bit cheated when this happens. At some point in your career, you might get the nagging suspicion there are important lessons to be learned outside of your current role -- lessons that will make you N times better at your current role. If so, heed the call, take the extra responsibility, and level yourself up. Easier to do at small- to medium-sized companies, but not impossible at large companies either. Take the initiative!
- pmoriarty 10y ago
  "Nothing is impossible for the man who doesn't have to do it himself." -- A H Weiler
- TheSpiceIsLife 10y ago
  Reading your comment just now cause an amusing thought. Software Developer, it makes it sound like the software is already there, you just need to pour the right sequence of chemicals on it in the correct order, and rinse. In a dimly lit room.
- spc476 10y ago
  At work, I'm a developer (of backend call processing stuff). I work closely with QA (one guy right now) to get the call processing back end code tested (answering questions about the product, the SS7 stack (which nobody, not even me, wants to muck with---it's nasty) and the regression tests which I wrote back when I was in QA). When it comes time to deploy an update to call processin into production, I'm there, at 2:00 am along with the ops team and QA in the deployment (to answer questions and check to see if everything is running right). If anything is wrong, anyone can initiate a rollback to the previous version (I've initiated a rollback once). In our situation, our customers are the various Monopolistic Phone Companies, so there's quite a bit of work that goes into a deployment (some fairly nasty SLAs and what not), so I'm glad there's an ops team to deal with most of that red tape. Yes, it's annoying that I can deploy as often as I would like, but I understand the reasoning behind it (and there's more in production than just call processing, like billing, provisioning and updates to our smart phone application).
pmoriarty 10y ago
It's gotten a lot easier to learn and practice ops over the years. No longer do you need access to a university or government lab to get your hands on Unix. Nor do you have to scour obscure corners of university libraries to get your hands on some wizard manual that finally makes sense of some bit of it for you. Tons of free, quality tutorials are available online, and you can get help on forums and in chat groups. Online book stores are overflowing with books on just about everything you'd want to know. Unix (or Linux) has become a lot easier to use in many ways, you can practice on VMs, and anyone who wants it can have root on their own machine. The tools have gotten a lot better too (though both the tools and the OS's have increased in complexity, layers, and interaction with other systems). Cloud providers make spinning up machines, network infrastructure, and various services easier than ever. Computer literacy has become many orders of magnitude more common than it once was, and a lot of devs grow up being admins of their own Linux systems. In many ways, it's never been easier to learn ops, to some extent. The same could be said for development, with languages, tools and training being far more available than they once were. That does definitely reduce the need for a dedicated ops team or a dedicated dev team to some degree. But just as in medicine sometimes you need a specialist who's had the training and a lifetime of practice in that speciality, and when a generalist's knowledge is not enough, I think there'll always be roles for ops and roles for devs. All other things being equal, a dev who mostly does development and dabbles in ops just isn't going to get the level of professional skill in ops as someone who focuses mostly on ops does.. just as someone who focuses mostly on ops and dabbles in development is probably not going to be able to achieve the same level of development skill that someone who does a lot of development day in and day out will. It's like someone being both a brilliant brain surgeon and a brilliant hand surgeon. They do have something in common: they're both medical specialities that treat the human body and they both require going to medical school, but being great at both is still rare, and if I ever have hand surgery I'd usually prefer to be treated by someone who's done thousands of hand surgeries and specializes in that, not one who's mostly a brain surgeon who's occasionally operated on hands. Some people are able to straddle both specialities and do an excellent job at both, but those are relatively rare, because the amount of knowledge and experience you need to do really master both is still quite large, despite everything. This knowledge also changes quite rapidly, so you have to spend a lot of time keeping up with new languages, frameworks, tools, services, etc. That's a lot to ask for even for one speciality, never mind two.
hullsean 10y ago
Devs & ops seem to have different mandates. Devs to create change, new features & build product to meet customer needs. Ops to resist change. To provide a rate limit. And a sense of perspective. Bc they're tasked with stability & continuity.
drinchev 10y ago
I have network knowledge, linux knowledge, debugging knowledge, some best practices about logging knowledge and still ( as a developer ) I think dev-ops needs to be separated to a different role. Indeed, you don't need a dev-ops person for your MVP or small web-app, but once you can afford it and the users you have require you to do `no-downtime-deployment` mechanism or database replication or server cluster, I think it's too much to ask your developers to do that. Anyway they will have to spend months on reading about the latest and greatest practices and in the end will have something half-baked, compared to what a dev-ops engineer can do for the same amount of time.
skarap 10y ago
To rephrase the article: everyone should and can go ops-less, you just need the devs to take over the ops roles.
- gaius 10y ago
  So get the same work done with half the people, got it.
perlgeek 10y ago
This article seems to assume that everything runs in the cloud. If not, Ops still have a big part in operating the self-service infrastructure that Susan talks about. Another point I'd like to make is that if developers need to take on more operational responsibility, it creates another barrier on entry. As somebody with a family, I don't want to be regularly on-call. And on the technical side, developers already need to know about the problem domain, the programming language, algorithm and data structures, design patterns, testing, version control and so on. Piling more operational knowledge onto this heap seems hardly promising to on-board more developers, and finding good developers outside the big tech hubs is already a pain.
- sqldba 10y ago
  I also feel like the "on call" thing is oversold a little. I work in operations and I'm on call. Some months I hear nothing, some weeks everything goes wrong, but on average I get about one call a month. That's not so bad. The thing is I spend 40 hours a week working out how to do proactive maintenance, monitoring, and alerting on those 400 systems I look after, so that I don't get called outside of hours. I mean that's what my operations job is meant to be, right? Developers can't do that. They're focused on one system or a few systems at most and they're developing 40 hours a week. When the SAN dies and every server in the DC crashes there's no point calling them and asking why they're not on top of it and to try to develop some solution to improve the speed of recovery next time. Me? I built a Jenkins server so next time it happens I press a button and get an immediate dashboard of where to focus my attention. Developers aren't operations. Things may change a little when things go into the deep cloud though and you have services instead of servers running services, seeing as operations may not know what to do with those. On the other hand maybe there is meant to be a handover of some kind so that one person can look after everything instead of 10 developers each having a small stake in their little piece of world.
  - gtirloni 10y ago
    This article is myopic in that it fails to mention the premises only (mostly) work for web-only startup companies relying heavily on cloud computing for everything. And even then, only the ones that don't see operational/infrastructure efficiency as a differentiator. If you take those constraints into account, it makes a lot of sense: just ask your developers to learn the very basics of those 3rd-party services you signed up to and things will go fine. A developer will just be making sure he/she uses the logging provider API, that their 3rd-party monitoring and alerting system is updated, the automatic third-party load balancer has a little checkbox saying backend monitoring is configured, etc. Behind those magic services there will be your traditional Ops people (now better equipped with automation tools) making sure everything is running smoothly, properly sized, monitored and troubleshooted when it fails (it WILL fail, in unexpected ways, more often than you'd like). People parroting "Ops-less" and "Server-less" buzzwords seem to be so detached from the daily activities required to maintaining real-world computing systems that they forget it exists. It's really frustrating when the SV zeitgeist living in a distorted reality (or a very narrow one) proposes this is the way things are or will certainly be in the near future. You just need to skim over all the GitHub issues in these magical tools (or countless articles with workarounds for all sorts of issues) to learn things aren't that simple as they pretend to be (which is fine, things take a lot of time to mature). The work is not changing (except for increased automation everywhere), it's just shifting to third-parties or far away teams in your own company. If only the computing world was that dreamy. I'll think about that next time I spent a day analyzing packets with tcpdump, coming up with a patch for a basic flaw in some tool, profiling some code that is destroying the storage system, etc. Yeah, that won't be distracting at all for developers. Maybe they don't need to work on their team's core competencies. Everything will "just" work. </sarcasm, sorry>
donavanm 10y ago
I've been a bit perplexed by the industries obsession with "operations" for the past decade. Constantly striving to decide what is it, who does it, and whether it's shunned or exalted. My current workplace has convinced me that "ops" is simply a friction organizations pay, like "tech debt." How you prioritize and minimize it is a business decision, not a life calling. I suspect my employer is actually the largest (many many thousands of employees) and oldest (15-20 years) practitioner of the "you deploy and own what you write" method. There are nearly zero people in "operations" roles, compared to tens of thousands of Developers. The "Systems" folks who might be called SE or SRE or PE or DevOps somewhere else sieve in to three roles: 1) Specializing in development below the application & (slightly) above hardware/os 2) Developing tooling and systems around infrastructure & distributed systems management 3) Saying #2, but mostly driving manual or adhoc actions Groups that naively pursue #3 seem to implode or catch fire after 12-18 months. They can succeed if its an intentional choice; employees are a resource and not all problems require further investment. Investing in #2 drives down the "ops" cost on other Developers and/or improves returns on infrastructure investment. Category #1 improves medium & long term returns on the software & services that group develops. A while back the job role was changed to accentuate that the goal is Build value by Development. The specific flavor of development is less important. My job role is Systems Development Engineer and I work with Software Development Engineers. SDE collaborating with SDE. When I need to go beyond my domain knowledge I might consult a NDE. In short I agree with the article summary. There will never be a "post ops" world. But getting over the "ops" title obsession feels good to me.
- digi_owl 10y ago
  (Web) devs are user facing and sexy, admins stomping the server room aisles are not (or so the reasoning seems to go). This appears to have been a prevailing notion since the dot-com days, but has acquired new energy since "cloud computing" became a management buzzword.
jondubois 10y ago
I think that applications in the near future will be built on top of 'cloud native stacks' (made up of app servers, databases, memory stores, message queues, etc...) which are designed to run and autoscale on a Kubernetes cluster (or a similar ops/orchestration systems) and developers working for various companies will just focus on adding business logic on top of these stacks - They won't have to understand how all the components in the underlying stack interact with each other when scaling up or down (or recovering from failures) - That will be all encoded as part of the stack's configuration (E.g. as Kubernetes config .yaml files).
tristor 10y ago
The entire conversation around "ops-less" systems ignores that in most cases the Ops folks are stronger engineers than the SWEs. If a company wants to ditch Ops they'll need to pay far more money for SWEs, expect a higher competency in engineering tasks, and good luck ever outsourcing again.
- kod 10y ago
  Citation needed. I've personally never seen an org where ops were stronger enngineers than dev.
  - tristor 10y ago
    Engineering is not writing code. Writing code is what you do to implement something you've engineered. Ops may not be better at writing code, but in every organization in my career they've been stronger engineers. Of course that's just my anecdote.
pmyjavec 10y ago
The naming of these technologies and paradigms is unfortunate in my view, it's now made Ops sound like a bad thing. Noops, Serverless? As a long time DevOps / Sys Admin / Generalist I feel disappointed lately because there seemed to be a very brief golden age, which in my opinion was DevOps done right, and that was just getting polished and accepted, then it feels like for little good reason except for marketing or something, that was just thrown out the window ? It was really getting results in my last org, basically meeting half-way with devs felt like the sweet spot and now it's going to extremes. I was really into investing my time into the DevOps / SRE role, now it just feels demotivating, as any good Ops knows it's a tough job that requires dedication, but is it worth the effort anymore ? Will people still want to hire ops? Should I just move into Software Engineering (which I can do), full time ? I think she hit the nail on the head to be honest.
- digi_owl 10y ago
  I get the impression the problem is that it's with ops/admins as it is with safety inspectors. When they do their job right, nothing spectacularly fails, and thus management starts wondering why they have this salary expense on the quarterly spreadsheet. developers on the other hand go hand in hand with marketing, and thus is easily noticed when they do their thing right.
  - pmyjavec 10y ago
    This definitely happens too :)
throw2016 10y ago
These cycles keeps repeating themselves. Some marketing driven term gets traction and then people start believing the hype, repeating it as some sacred truth and dismissing experience as grey beards. A few years down the road when things don't go according to plan some other term gets traction and rinse repeat. HN especially is guilty of perpetuating hype when one would expert a far greater degree of scrutiny. When you get to the nitty gritty of scaling from networking, distributed storage, failover, high availability, security and managing state that's entire domains of expertise and experience that devops glosses over.
n72 10y ago
Anyone know the best place to learn more about kind of basic ops that a dev should know? I've done deployments, spun up AWS instances, configured load balancers etc., but my problem is that I don't know what I don't know. For example, when starting to debug, I get in there and muddle around, but there may be far more efficient way of doing it which I just don't know about. I'll watch ops guys use ps, netstat, etc., which are things I don't use, but presume are useful.
jonaf 10y ago
This was a good read. As a DevOps Engineer at a company that does not have a distinct Ops department, who's also a Tech Lead, I have some thoughts I'd like to share. First, while the ultimate goal of any engineer (even not among the Ops disciplines) should be to automate yourself out of a job, we have seen time and again that it is impossible to do so, as any good engineer will continue to advance the state of the art. Conclusively, there is no "finish line" for operations that will not be obsolete within 3 years. The concern that you'll just have to migrate across organizations, reaching the "finish line," rinsing and repeating is a non-issue. The notion of a finish line is really sugarcoated FUD. (The author's interesting thought experiment alludes to this and does refute the argument, so hopefully my statements simply complement the article in that regard. I call it a thought experiment because we will never arrive at this "you automated everything" goal.) I absolutely agree with the author that we do not really need ops engineers. We do, however, need specific disciplines of software engineering. Specifically, I recommend The Systems Engineering Side of Site Reliability Engineering[1] (as well as the book), Site Reliability Engineering[2] from Google. The usenix article in particular describes three distinct disciplines of software development: systems engineering, site reliability engineering and software engineering. The individuals behind the roles have little to do with the roles themselves (rather, the causal chain is the other way around); it's often misunderstood, for whatever reason, that software engineers and operations engineers have different skillsets because of who they are. This is true, but it does not mean that a software engineer cannot, in short order relative to individuals with no software background whatsoever, transition into an operations role, or vice-versa. Orthogonally, identifying individuals with the skills in any of these three disciplines is critical to placing them in work that is personally and professionally rewarding, as well as more valuable to the organization than if they were placed in some other discipline. And sometimes, individuals do not even know of these disciplines or, for whatever reason, think they are suited for a discipline that they are not actually best at. I was one of these people (a software engineer before moving into operations). In essence, what I'm trying to demonstrate here is that these disciplines of software development are permanent (or have generations-length longevity) and we should not be concerned with being replaced or becoming obsolete. Indeed, it is the specific tasks that will change over time. Consider, for example, electrical engineers. We do not anticipate that EE's will be replaced by robots. Despite robots automating the process of manufacturing circuits, EE's will always be invaluable and irreplaceable. However, their specific responsibilities will change over time. This is why I said before, advancing the state of the art results in new work (or even types of work) to exist. Finally -- and this is just a bonus -- any experience acquired by, say, an EE will be useful even if he or she transitions to a new discipline of engineering. In my experience, the best software engineers I ever known have understood in remarkable depth CPU architecture, memory models, networking protocols, configuration management, etc. > there is practically no difference between a software engineer and an operations engineer - they both build and run their own systems - other than the domain: the software engineer builds and runs the top-level applications, while the (ex-)operations engineer builds and runs the infrastructure and tooling underneath the applications. The above statement from the article's thought experiment vaguely describes two (of the three) software engineering disciplines that Hixson[1] talks about. Operations engineers and software engineers alike are, in this thought experiment, responsible for leveraging their expertise and talent (understand that I use the word talent according to the definition described by Hixson) at maximum efficiency. The manifestation of these disciplines is reflected in their domain, but the individual tasks themselves are only relevant today and will change tomorrow. The third discipline not described here (systems engineering) is very much relevant and deals specifically with the interactions among systems, which neither operations nor software engineers will focus on (or necessarily have significant talent in). Later in the article, the author sort of blends SRE (site reliability engineers) and SE (systems engineers) together. The distinction isn't important for the author to make her point, but I wanted to highlight it a little bit. Second, I think the author describes an environment that strongly reflects the ideals of the DevOps movement. From my reading, I'm inferring that the author is aligned with these ideals. I consider this a big selling point if I ever wanted to consider Uber as a place of employment. As some other comments here on HN have noted: it is extremely rare and difficult to find an organization that has embraced DevOps principles with such purity. I'm fortunate to be employed at one of them (not Uber), and it sounds like Uber has made some good decisions as an organization in this regard. (Hopefully this paragraph can be to the benefit of any employment-seeking operations engineers. The statements in the article reflect positively on Uber, particularly if you are trying to move from a traditional operations role to a DevOps/SE/SRE role.) Third, the article does a great job refuting the 3 identified arguments. In general, I can't agree more! The author takes the time to consider the merit of each argument and qualify the conditions under which they are true before refuting them, which makes it much easier to read coming from a more traditional organization. From my biased perspective, I don't even give these arguments the light of day and refute them without thinking twice about the qualifications that can alter their accuracy; so, one takeaway for me from the article has been to not make the assumption that these arguments are being made by like-minded individuals. It's quite likely that I'm too hard on people for bringing up concerns like these and, as a result, not open to new (old) ideas. My final thought on the article is that, while it's not really news in most of the social circles I spend my time with (as a byproduct of having learned much of what I know from a stellar colleagues in a great work environment -- not because of any personal accomplishment), I really appreciate that the author took the time to write out these thoughts and publish them so that the broader software community can grow and adopt ideals that move our industry forward in a very positive, very significant way. So thanks to the author, and to aberoham who posted the link here on HN! [1] https://www.usenix.org/publications/login/june15/hixson https://www.usenix.org/publications/login/june15/hixson [2] https://landing.google.com/sre/book.html https://landing.google.com/sre/book.html
dmourati 10y ago
I think the article gets this mostly right. I describe Dev and Ops as a continuum and DevOps as the concept that each side needs to get better at doing the others job. Noops and/or serverless are both newer terms that are early on the hype cycle. I wouldn't get too bent out of shape about them. My advice in this regard has been the following. Devs: start thinking more about running your code in production and solving user needs. Works on my laptop is over. There are a whole slew of things that you need to understand to build at scale. Talk to your Ops people. Especially those who have the good sense to show you mutual respect. Ops: Move up the stack. The days of providing just OS support and load balancing are gone. You have to learn to code. Spend time in IDEs, work in source control, do code testing. Learn from your Dev counterparts. Especially those who have the good sense to learn from you.
scurvy 10y ago
When it comes down to brass tacks, developers don't want to be on call. They want to work on the things that they want to work on, when they want to work on them. Look at the proliferation of 20% projects ("keep them happy and let them do what they want 1 day a week rather than what they get paid to do"), and the outright refusal to fix their own bugs. How many developers do you know who would turn their noses up at being placed on a sustained engineering team rather than creating new features? There's a stigma there. You think that stigma won't be there at 3AM Pacific when Europe starts hitting their new feature to the breaking point? I've been an ops engineer in a "lean startup" where developers were on-call for their services. It didn't work super well because people ignored their phones or put phones on silent at night. As a backstop, they put me (ops engineer) as the fallback secondary notification because they knew I would wake up. Ergo, everyone ignored everything and it all rolled to me. They'd wake up at 7AM, find everything ablaze (because they ignored all my phone calls), then would fix it and go about the rest of their day. Let's face it though. They probably don't want to be on call. Why force them to do this? There's the concept of ownership and closing the pain loop, but most people really don't understand what it truly means to be on call. "Sorry honey, can't go to the movies tonight I'm on call." "Sorry bro, can't get wasted tonight I'm on call." Only huge organizations with huge dev teams can go through a developer on-call rotation. Most leaner (smaller) companies have 1-2 devs per project, and it's unreasonable to expect that developer to be on-call 24x7. This stuff works at Uber, Facebook, and Google. But the vast majority of the world isn't Uber, Facebook, and Google. Also, I'd expect more pay as a developer if the job required on-call shifts. I don't think companies are willing to pay even more than they already are.

161 points by aberoham 10y ago

I've always seen operations folks in a different light then most developers. Maybe I'm an exception but I've always viewed operations people as having in-depth knowledge of infrastructure on the lower layers (Session layer and lower). They are able to walk through the inner-workings of network concepts, operating systems, etc. They can implement and design systems that work in a highly available, scalable way. Not saying developers CANT do this, but a majority of them don't want to, and don't have the first idea on how to. Granted some "10x" developers who have full stack knowledge can do just as well, but let's be honest most people aren't 10x developers, they want to mess with the code stack and that's it. This reminds me of the all the job posting you see online that want you to have expert knowledge of C++ expert in SQL DBA expert in networking expert in web design expert in big data Yeah you might be able to find someone who can do all of the above, but odds are they won't do them all well. I don't think the expectation should be set that developers should have to manage the systems stack, as well as manage the code. Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production. Even with configuration toolsets, and infrastructure moving to "code-to-deploy" solutions. Things will still break, unusual things will still happen. Taking developers out of their zone to focus on problems will slow down the entire company. Ops can can always push to 'strive to automate themselves out of their jobs' but I'd argue that this is an endless job which always has outlets that you can continue to strive and build into. Of course, this is talking from my personal experience, I've never worked in a big company like Uber, so the environment might be entirely different from my own.

morgante 10y ago
> Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production. Really? As a dev, the vast majority of my projects/roles have required all 3 to some extent. I have been on call for the past 4 years. Honestly, people really need to admit that the age of the sysadmin (who is not also a strong programmer) is over. I might not have enough deep knowledge to answer network/OS trivia off the top of my mind, [0] but I'm perfectly capable of running a production build for my code (with monitoring, configuration management, etc.) and finding appropriate resources in the instances where problems are outside my domain. [0] https://news.ycombinator.com/item?id=12701272 https://news.ycombinator.com/item?id=12701272
- llama052 10y ago
  > Really? As a dev, the vast majority of my projects/roles have required all 3 to some extent. I can run photoshop and mock some images at home and it's good enough, but does that make me a photoshop guru? I think not. edit: bad example.
  - morgante 10y ago
    > To some extent maybe, but running something on a heroku engine Talk about a straw man. I would agree with you that deploying to Heroku hardly constitutes ops. I'm talking about things like running a multi-AZ Kubernetes cluster.
    - llama052 10y ago
      Following a guide online and getting something online is different from running something in production that a company relies on. Learning a infrastructure piece thoroughly can take months alone in resilience testing, failover testing, and system design/architecture choices. It's a huge fallacy in the infrastructure side of things that because something "works" means it's setup correctly, and that couldn't be further from the truth. You'll get an environment built on technical debt. Outages will occur and you'll be playing sysadmin full time before long. You won't be a developer at this point anymore, you'll be the sysadmin/ops role of your company. I've seen this happen at multiple companies, and I doubt this is the last time I will see it happen.
      - morgante 10y ago
        You continue to be extremely insulting and making significant assumptions about my experience. What shibboleth do I have to provide to prove that I know what I'm doing? This inherent assumption that other people don't know anything is incredibly damaging to the industry: I don't assume a SRE is an incompetent programmer, so you shouldn't assume I am barely competent at ops. > Following a guide online and getting something online is different from running something in production that a company relies on Where did I say that I had simply "followed a guide online" instead of "putting something in production." Like I said in my original comment, I have been the primary on call engineer for multiple projects with millions of users. Despite that, I still find time to develop new features and push the product forward. Yes, I occasionally deal with outages/failures and have to write some new automation to remediate them. But that doesn't make me a full-time sysadmin. I don't really feel like engaging with you further, since you're apparently more interested in arguing with a straw man of your own construction.
toast0 10y ago
> Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production. "sorry about the deadline, I had to make the fine network work" gets you out of making deadlines. The nice thing about being on call for your own code is that fixing your stuff at all hours is a natural consequence of writing fragile stuff. The not nice thing is that not all of the reasons your stuff breaks are your fault and some issues aren't realistically preventable.
- llama052 10y ago
  I can understand being on call for your own code, but often that's just not the case. Infrastructure can go off the rails as well, clusters can go on the fritz, upstream might be having issues, a release update on your infrastructure side might introduce problems. I just don't see that falling on the developers who also must maintain the code-base for the product.
  - pjscott 10y ago
    That just means that being on-call for your own code results in good code, a love for automated testing, and a seething grudge against EC2 and its tendency to have multi-zone outages at 3 AM. It doesn't matter what time zone you're in, by the way; the outage will always, always page you at 3 AM.
  - toast0 10y ago
    If the issues for things that aren't your fault happen are infrequent, then it's not a huge burden, and developers may be able to help with mitigation: ex if your internet bandwidth is cut in half because of a fiber cut, how can you move or shed load to have a stable, degraded system. If the issues are frequent, then it provides incentive to build fault tolerance. (And also seething grudges for fault prone infrastructure)
donavanm 10y ago
> Personally, I can't imagine having the expectation set that I need to be on call, and work on code commits with deadlines, while debugging networking issues in production. This seems to be the meme of "the supporting infrastructure always breaks and causes the pager engagements!" I have seen those orgs and times where networking or facility failures are the leading cause of outages. Long term those are symptoms of chronic underinvestment and tech debt accumulation. I also see, on a daily basis, a massive business where the absolute leading cause of "outage minutes" is software defect or deficiency. With investment in supporting infrastructure you're paged 10 times for a defect your team "developed" for every one unavoidable dependency failure. Big companies can make those (huge) investments themselves. Small companies can pay someone else for access to theirs. In any case its always a business decision, not a certainty.
davidgerard 10y ago
I explain my job to nongeeks as "computer roadie". My job is to make sure everything is in order, devs' job is to get up there and be Eric Clapton. People can do both (many roadies are really quite capable musicians). But they're fundamentally different mindsets, and expertise in one is not transferable to the other.

jzelinskie 10y ago

If you liked this post and want to know the full details to this change in ops, please read the SRE book[0]. It's a great read for both devs and ops and can immediately help you make changes to company policy for the better. [0]: http://shop.oreilly.com/product/0636920041528.do http://shop.oreilly.com/product/0636920041528.do

_qc3o 10y ago

I look forward to the day when every software developer has half a clue about monitoring, logging, high availability, configuration management, orchestration/scheduling, performance tuning, build/deployment pipelines, data management/archiving, security and exploit mitigation, etc. The full-stack developer like the serverless/ops-less future is a pipe dream. Most technology organizations barely even know how to build the software to begin with let alone figure out the right way to operate it.

toomuchtodo 10y ago
The startup scene is rife with overconfidence that you have enough waking hours to know everything you listed in one role. Regret I have but one upvote for you.
- hibikir 10y ago
  The issue is that some people can spend just enough hours to make it LOOK like they really understand enough, when in practice, they have wide breadth, but not anywhere enough depth to do the job well. When the startup becomes mid size, suddenly you have a spaghetti codebase with unfortunate tradeoffs, a wide infrastructure full of hard coding that badly attempts to mimic the state of the art of 5 years ago, and one nine of reliability. All of this is of course undocumented, so when you bring in new people, they take forever to learn the existing system, as nothing resembles good practices. As those new people try to clean everything up, they look far less productive than those people that have made this gigantic mess in the first place, and then everyone wonders how they got so lucky with their early, uberproductive employees, which seem to be so much better than anyone else. If the startup has a good economic situation, they might survive while compounding the problem by overhiring (I am sure you all have heard the stories). If the finances aren't quite as good, then the startup folds. I see roads out of this, but they all pass by different early employee compensation: A good early employee is not a good employee in a midsized company, but stock options don't work all that well when early engineers should be leaving the company 3 years in, before exercising the options makes any sense.
  - 10y ago

advisedwang 10y ago

There is a trade off between development velocity and reliability. Not just because making more changes often breaks more things, but also in how much investment is made in infrastructure, monitoring. It even shows up in design choices - making a service multi-regional is more effort but results in a more stable system. A big part of SRE is motivation for reliability. Part of the reason to have SRE is to have engineers motivated by stability, not development velocity. Have somebody who knows the better choice for reliability, even if at the time a decision is made the other way. This necessitates both a separate profession and organisation separation (e.g. for product readiness reviews or for higher level stability decision making). (NB: I am a Googler that works closely with SRE, but not an SRE myself)

dkarapetyan 10y ago
I think this is a false dichotomy. If developers are incentivized to make stable software then they'll make stable software but that's not the case. Software engineers that work on products are promoted based on number of features they ship, not how many production outages they don't cause. It's like the senator that lobbies to put bolted doors on plane cockpits before 9/11. That senator will get zero credit for anything. Fundamentally it is harder to measure the effectiveness of preventive measures so most organizations don't and instead settle for number of features shipped.
- pmoriarty 10y ago
  Don't forget all the death marches to meet unrealistic deadlines that some exec or sales rep pulled out of their ass. In far too many organizations devs are running around like the proverbial headless chickens to ship! ship! ship! while ops are endlessly fighting fire after fire. Many teams don't have the time or manpower to afford the luxury of being proactive, and when anyone suggests doing so, tries to put the breaks on, or asks for more resources they're treated like troublemakers.

sjg007 10y ago

developing on top of managed services will allow you to do more with less people over time. And adopt best practices by integration.

latch 10y ago

I agree. Quality is improved as developers start to understand, get involved in, and own the operation side of running their system. Ops ought to be an enabling force. But, I do wish that anytime anyone writes about ops or infrastructure, they put a big headline: There's a 99% chance you just just Scale Up and use Bash.

solipsism 10y ago

I don't know who writes/says dumb things like "ops is dead" but ideally we wouldn't waste time arguing against such reductive statements. Whatever the zeitgeist, there will always be homogeneous groups (all developers know/do roughly the same things), structurally heterogenous groups, and organically heterogenous groups.

jimjimjim 10y ago

back before the agile dark ages, things like performance, logging/instrumentation, stability guidelines, security guidelines and deployment abilities were able to be specified as non-functional requirements and the product was QA'd for these just as much as the features that were added.

kasey_junk 10y ago
Blaming agile for this seems odd. I've worked on agile teams where all of those were set as requirements.

deleted 10y ago

[deleted]

sigil 10y ago

It feels to me like title-centric corporate culture might be a part of the problem here. Are you an SRE? An SWE? Do you Build? Or do you Run? Are you a Software Developer or a Software Engineer? They're interchangeable to the author, but she includes both on the off chance that you the reader are sensitive to some fine splitting / quantum structure... I for one welcome the blurring boundaries. It feels weird to say, sure, I wrote this code with pathological runtime behavior, but it's your problem now, Ops Person. How do you learn to avoid that mistake if someone else absorbs the pain for you (ie learns your lesson)? Personally I feel a bit cheated when this happens. At some point in your career, you might get the nagging suspicion there are important lessons to be learned outside of your current role -- lessons that will make you N times better at your current role. If so, heed the call, take the extra responsibility, and level yourself up. Easier to do at small- to medium-sized companies, but not impossible at large companies either. Take the initiative!

pmoriarty 10y ago
"Nothing is impossible for the man who doesn't have to do it himself." -- A H Weiler
TheSpiceIsLife 10y ago
Reading your comment just now cause an amusing thought. Software Developer, it makes it sound like the software is already there, you just need to pour the right sequence of chemicals on it in the correct order, and rinse. In a dimly lit room.
spc476 10y ago
At work, I'm a developer (of backend call processing stuff). I work closely with QA (one guy right now) to get the call processing back end code tested (answering questions about the product, the SS7 stack (which nobody, not even me, wants to muck with---it's nasty) and the regression tests which I wrote back when I was in QA). When it comes time to deploy an update to call processin into production, I'm there, at 2:00 am along with the ops team and QA in the deployment (to answer questions and check to see if everything is running right). If anything is wrong, anyone can initiate a rollback to the previous version (I've initiated a rollback once). In our situation, our customers are the various Monopolistic Phone Companies, so there's quite a bit of work that goes into a deployment (some fairly nasty SLAs and what not), so I'm glad there's an ops team to deal with most of that red tape. Yes, it's annoying that I can deploy as often as I would like, but I understand the reasoning behind it (and there's more in production than just call processing, like billing, provisioning and updates to our smart phone application).

pmoriarty 10y ago

It's gotten a lot easier to learn and practice ops over the years. No longer do you need access to a university or government lab to get your hands on Unix. Nor do you have to scour obscure corners of university libraries to get your hands on some wizard manual that finally makes sense of some bit of it for you. Tons of free, quality tutorials are available online, and you can get help on forums and in chat groups. Online book stores are overflowing with books on just about everything you'd want to know. Unix (or Linux) has become a lot easier to use in many ways, you can practice on VMs, and anyone who wants it can have root on their own machine. The tools have gotten a lot better too (though both the tools and the OS's have increased in complexity, layers, and interaction with other systems). Cloud providers make spinning up machines, network infrastructure, and various services easier than ever. Computer literacy has become many orders of magnitude more common than it once was, and a lot of devs grow up being admins of their own Linux systems. In many ways, it's never been easier to learn ops, to some extent. The same could be said for development, with languages, tools and training being far more available than they once were. That does definitely reduce the need for a dedicated ops team or a dedicated dev team to some degree. But just as in medicine sometimes you need a specialist who's had the training and a lifetime of practice in that speciality, and when a generalist's knowledge is not enough, I think there'll always be roles for ops and roles for devs. All other things being equal, a dev who mostly does development and dabbles in ops just isn't going to get the level of professional skill in ops as someone who focuses mostly on ops does.. just as someone who focuses mostly on ops and dabbles in development is probably not going to be able to achieve the same level of development skill that someone who does a lot of development day in and day out will. It's like someone being both a brilliant brain surgeon and a brilliant hand surgeon. They do have something in common: they're both medical specialities that treat the human body and they both require going to medical school, but being great at both is still rare, and if I ever have hand surgery I'd usually prefer to be treated by someone who's done thousands of hand surgeries and specializes in that, not one who's mostly a brain surgeon who's occasionally operated on hands. Some people are able to straddle both specialities and do an excellent job at both, but those are relatively rare, because the amount of knowledge and experience you need to do really master both is still quite large, despite everything. This knowledge also changes quite rapidly, so you have to spend a lot of time keeping up with new languages, frameworks, tools, services, etc. That's a lot to ask for even for one speciality, never mind two.

hullsean 10y ago

Devs & ops seem to have different mandates. Devs to create change, new features & build product to meet customer needs. Ops to resist change. To provide a rate limit. And a sense of perspective. Bc they're tasked with stability & continuity.

drinchev 10y ago

I have network knowledge, linux knowledge, debugging knowledge, some best practices about logging knowledge and still ( as a developer ) I think dev-ops needs to be separated to a different role. Indeed, you don't need a dev-ops person for your MVP or small web-app, but once you can afford it and the users you have require you to do `no-downtime-deployment` mechanism or database replication or server cluster, I think it's too much to ask your developers to do that. Anyway they will have to spend months on reading about the latest and greatest practices and in the end will have something half-baked, compared to what a dev-ops engineer can do for the same amount of time.

skarap 10y ago

To rephrase the article: everyone should and can go ops-less, you just need the devs to take over the ops roles.

gaius 10y ago
So get the same work done with half the people, got it.

perlgeek 10y ago

This article seems to assume that everything runs in the cloud. If not, Ops still have a big part in operating the self-service infrastructure that Susan talks about. Another point I'd like to make is that if developers need to take on more operational responsibility, it creates another barrier on entry. As somebody with a family, I don't want to be regularly on-call. And on the technical side, developers already need to know about the problem domain, the programming language, algorithm and data structures, design patterns, testing, version control and so on. Piling more operational knowledge onto this heap seems hardly promising to on-board more developers, and finding good developers outside the big tech hubs is already a pain.

sqldba 10y ago
I also feel like the "on call" thing is oversold a little. I work in operations and I'm on call. Some months I hear nothing, some weeks everything goes wrong, but on average I get about one call a month. That's not so bad. The thing is I spend 40 hours a week working out how to do proactive maintenance, monitoring, and alerting on those 400 systems I look after, so that I don't get called outside of hours. I mean that's what my operations job is meant to be, right? Developers can't do that. They're focused on one system or a few systems at most and they're developing 40 hours a week. When the SAN dies and every server in the DC crashes there's no point calling them and asking why they're not on top of it and to try to develop some solution to improve the speed of recovery next time. Me? I built a Jenkins server so next time it happens I press a button and get an immediate dashboard of where to focus my attention. Developers aren't operations. Things may change a little when things go into the deep cloud though and you have services instead of servers running services, seeing as operations may not know what to do with those. On the other hand maybe there is meant to be a handover of some kind so that one person can look after everything instead of 10 developers each having a small stake in their little piece of world.
- gtirloni 10y ago
  This article is myopic in that it fails to mention the premises only (mostly) work for web-only startup companies relying heavily on cloud computing for everything. And even then, only the ones that don't see operational/infrastructure efficiency as a differentiator. If you take those constraints into account, it makes a lot of sense: just ask your developers to learn the very basics of those 3rd-party services you signed up to and things will go fine. A developer will just be making sure he/she uses the logging provider API, that their 3rd-party monitoring and alerting system is updated, the automatic third-party load balancer has a little checkbox saying backend monitoring is configured, etc. Behind those magic services there will be your traditional Ops people (now better equipped with automation tools) making sure everything is running smoothly, properly sized, monitored and troubleshooted when it fails (it WILL fail, in unexpected ways, more often than you'd like). People parroting "Ops-less" and "Server-less" buzzwords seem to be so detached from the daily activities required to maintaining real-world computing systems that they forget it exists. It's really frustrating when the SV zeitgeist living in a distorted reality (or a very narrow one) proposes this is the way things are or will certainly be in the near future. You just need to skim over all the GitHub issues in these magical tools (or countless articles with workarounds for all sorts of issues) to learn things aren't that simple as they pretend to be (which is fine, things take a lot of time to mature). The work is not changing (except for increased automation everywhere), it's just shifting to third-parties or far away teams in your own company. If only the computing world was that dreamy. I'll think about that next time I spent a day analyzing packets with tcpdump, coming up with a patch for a basic flaw in some tool, profiling some code that is destroying the storage system, etc. Yeah, that won't be distracting at all for developers. Maybe they don't need to work on their team's core competencies. Everything will "just" work. </sarcasm, sorry>

donavanm 10y ago

I've been a bit perplexed by the industries obsession with "operations" for the past decade. Constantly striving to decide what is it, who does it, and whether it's shunned or exalted. My current workplace has convinced me that "ops" is simply a friction organizations pay, like "tech debt." How you prioritize and minimize it is a business decision, not a life calling. I suspect my employer is actually the largest (many many thousands of employees) and oldest (15-20 years) practitioner of the "you deploy and own what you write" method. There are nearly zero people in "operations" roles, compared to tens of thousands of Developers. The "Systems" folks who might be called SE or SRE or PE or DevOps somewhere else sieve in to three roles: 1) Specializing in development below the application & (slightly) above hardware/os 2) Developing tooling and systems around infrastructure & distributed systems management 3) Saying #2, but mostly driving manual or adhoc actions Groups that naively pursue #3 seem to implode or catch fire after 12-18 months. They can succeed if its an intentional choice; employees are a resource and not all problems require further investment. Investing in #2 drives down the "ops" cost on other Developers and/or improves returns on infrastructure investment. Category #1 improves medium & long term returns on the software & services that group develops. A while back the job role was changed to accentuate that the goal is Build value by Development. The specific flavor of development is less important. My job role is Systems Development Engineer and I work with Software Development Engineers. SDE collaborating with SDE. When I need to go beyond my domain knowledge I might consult a NDE. In short I agree with the article summary. There will never be a "post ops" world. But getting over the "ops" title obsession feels good to me.

digi_owl 10y ago
(Web) devs are user facing and sexy, admins stomping the server room aisles are not (or so the reasoning seems to go). This appears to have been a prevailing notion since the dot-com days, but has acquired new energy since "cloud computing" became a management buzzword.

jondubois 10y ago

I think that applications in the near future will be built on top of 'cloud native stacks' (made up of app servers, databases, memory stores, message queues, etc...) which are designed to run and autoscale on a Kubernetes cluster (or a similar ops/orchestration systems) and developers working for various companies will just focus on adding business logic on top of these stacks - They won't have to understand how all the components in the underlying stack interact with each other when scaling up or down (or recovering from failures) - That will be all encoded as part of the stack's configuration (E.g. as Kubernetes config .yaml files).

tristor 10y ago

The entire conversation around "ops-less" systems ignores that in most cases the Ops folks are stronger engineers than the SWEs. If a company wants to ditch Ops they'll need to pay far more money for SWEs, expect a higher competency in engineering tasks, and good luck ever outsourcing again.

kod 10y ago
Citation needed. I've personally never seen an org where ops were stronger enngineers than dev.
- tristor 10y ago
  Engineering is not writing code. Writing code is what you do to implement something you've engineered. Ops may not be better at writing code, but in every organization in my career they've been stronger engineers. Of course that's just my anecdote.

pmyjavec 10y ago

The naming of these technologies and paradigms is unfortunate in my view, it's now made Ops sound like a bad thing. Noops, Serverless? As a long time DevOps / Sys Admin / Generalist I feel disappointed lately because there seemed to be a very brief golden age, which in my opinion was DevOps done right, and that was just getting polished and accepted, then it feels like for little good reason except for marketing or something, that was just thrown out the window ? It was really getting results in my last org, basically meeting half-way with devs felt like the sweet spot and now it's going to extremes. I was really into investing my time into the DevOps / SRE role, now it just feels demotivating, as any good Ops knows it's a tough job that requires dedication, but is it worth the effort anymore ? Will people still want to hire ops? Should I just move into Software Engineering (which I can do), full time ? I think she hit the nail on the head to be honest.

digi_owl 10y ago
I get the impression the problem is that it's with ops/admins as it is with safety inspectors. When they do their job right, nothing spectacularly fails, and thus management starts wondering why they have this salary expense on the quarterly spreadsheet. developers on the other hand go hand in hand with marketing, and thus is easily noticed when they do their thing right.
- pmyjavec 10y ago
  This definitely happens too :)

throw2016 10y ago

These cycles keeps repeating themselves. Some marketing driven term gets traction and then people start believing the hype, repeating it as some sacred truth and dismissing experience as grey beards. A few years down the road when things don't go according to plan some other term gets traction and rinse repeat. HN especially is guilty of perpetuating hype when one would expert a far greater degree of scrutiny. When you get to the nitty gritty of scaling from networking, distributed storage, failover, high availability, security and managing state that's entire domains of expertise and experience that devops glosses over.

n72 10y ago

Anyone know the best place to learn more about kind of basic ops that a dev should know? I've done deployments, spun up AWS instances, configured load balancers etc., but my problem is that I don't know what I don't know. For example, when starting to debug, I get in there and muddle around, but there may be far more efficient way of doing it which I just don't know about. I'll watch ops guys use ps, netstat, etc., which are things I don't use, but presume are useful.

jonaf 10y ago

This was a good read. As a DevOps Engineer at a company that does not have a distinct Ops department, who's also a Tech Lead, I have some thoughts I'd like to share. First, while the ultimate goal of any engineer (even not among the Ops disciplines) should be to automate yourself out of a job, we have seen time and again that it is impossible to do so, as any good engineer will continue to advance the state of the art. Conclusively, there is no "finish line" for operations that will not be obsolete within 3 years. The concern that you'll just have to migrate across organizations, reaching the "finish line," rinsing and repeating is a non-issue. The notion of a finish line is really sugarcoated FUD. (The author's interesting thought experiment alludes to this and does refute the argument, so hopefully my statements simply complement the article in that regard. I call it a thought experiment because we will never arrive at this "you automated everything" goal.) I absolutely agree with the author that we do not really need ops engineers. We do, however, need specific disciplines of software engineering. Specifically, I recommend The Systems Engineering Side of Site Reliability Engineering[1] (as well as the book), Site Reliability Engineering[2] from Google. The usenix article in particular describes three distinct disciplines of software development: systems engineering, site reliability engineering and software engineering. The individuals behind the roles have little to do with the roles themselves (rather, the causal chain is the other way around); it's often misunderstood, for whatever reason, that software engineers and operations engineers have different skillsets because of who they are. This is true, but it does not mean that a software engineer cannot, in short order relative to individuals with no software background whatsoever, transition into an operations role, or vice-versa. Orthogonally, identifying individuals with the skills in any of these three disciplines is critical to placing them in work that is personally and professionally rewarding, as well as more valuable to the organization than if they were placed in some other discipline. And sometimes, individuals do not even know of these disciplines or, for whatever reason, think they are suited for a discipline that they are not actually best at. I was one of these people (a software engineer before moving into operations). In essence, what I'm trying to demonstrate here is that these disciplines of software development are permanent (or have generations-length longevity) and we should not be concerned with being replaced or becoming obsolete. Indeed, it is the specific tasks that will change over time. Consider, for example, electrical engineers. We do not anticipate that EE's will be replaced by robots. Despite robots automating the process of manufacturing circuits, EE's will always be invaluable and irreplaceable. However, their specific responsibilities will change over time. This is why I said before, advancing the state of the art results in new work (or even types of work) to exist. Finally -- and this is just a bonus -- any experience acquired by, say, an EE will be useful even if he or she transitions to a new discipline of engineering. In my experience, the best software engineers I ever known have understood in remarkable depth CPU architecture, memory models, networking protocols, configuration management, etc. > there is practically no difference between a software engineer and an operations engineer - they both build and run their own systems - other than the domain: the software engineer builds and runs the top-level applications, while the (ex-)operations engineer builds and runs the infrastructure and tooling underneath the applications. The above statement from the article's thought experiment vaguely describes two (of the three) software engineering disciplines that Hixson[1] talks about. Operations engineers and software engineers alike are, in this thought experiment, responsible for leveraging their expertise and talent (understand that I use the word talent according to the definition described by Hixson) at maximum efficiency. The manifestation of these disciplines is reflected in their domain, but the individual tasks themselves are only relevant today and will change tomorrow. The third discipline not described here (systems engineering) is very much relevant and deals specifically with the interactions among systems, which neither operations nor software engineers will focus on (or necessarily have significant talent in). Later in the article, the author sort of blends SRE (site reliability engineers) and SE (systems engineers) together. The distinction isn't important for the author to make her point, but I wanted to highlight it a little bit. Second, I think the author describes an environment that strongly reflects the ideals of the DevOps movement. From my reading, I'm inferring that the author is aligned with these ideals. I consider this a big selling point if I ever wanted to consider Uber as a place of employment. As some other comments here on HN have noted: it is extremely rare and difficult to find an organization that has embraced DevOps principles with such purity. I'm fortunate to be employed at one of them (not Uber), and it sounds like Uber has made some good decisions as an organization in this regard. (Hopefully this paragraph can be to the benefit of any employment-seeking operations engineers. The statements in the article reflect positively on Uber, particularly if you are trying to move from a traditional operations role to a DevOps/SE/SRE role.) Third, the article does a great job refuting the 3 identified arguments. In general, I can't agree more! The author takes the time to consider the merit of each argument and qualify the conditions under which they are true before refuting them, which makes it much easier to read coming from a more traditional organization. From my biased perspective, I don't even give these arguments the light of day and refute them without thinking twice about the qualifications that can alter their accuracy; so, one takeaway for me from the article has been to not make the assumption that these arguments are being made by like-minded individuals. It's quite likely that I'm too hard on people for bringing up concerns like these and, as a result, not open to new (old) ideas. My final thought on the article is that, while it's not really news in most of the social circles I spend my time with (as a byproduct of having learned much of what I know from a stellar colleagues in a great work environment -- not because of any personal accomplishment), I really appreciate that the author took the time to write out these thoughts and publish them so that the broader software community can grow and adopt ideals that move our industry forward in a very positive, very significant way. So thanks to the author, and to aberoham who posted the link here on HN! [1] https://www.usenix.org/publications/login/june15/hixson https://www.usenix.org/publications/login/june15/hixson [2] https://landing.google.com/sre/book.html https://landing.google.com/sre/book.html

dmourati 10y ago

I think the article gets this mostly right. I describe Dev and Ops as a continuum and DevOps as the concept that each side needs to get better at doing the others job. Noops and/or serverless are both newer terms that are early on the hype cycle. I wouldn't get too bent out of shape about them. My advice in this regard has been the following. Devs: start thinking more about running your code in production and solving user needs. Works on my laptop is over. There are a whole slew of things that you need to understand to build at scale. Talk to your Ops people. Especially those who have the good sense to show you mutual respect. Ops: Move up the stack. The days of providing just OS support and load balancing are gone. You have to learn to code. Spend time in IDEs, work in source control, do code testing. Learn from your Dev counterparts. Especially those who have the good sense to learn from you.

scurvy 10y ago

When it comes down to brass tacks, developers don't want to be on call. They want to work on the things that they want to work on, when they want to work on them. Look at the proliferation of 20% projects ("keep them happy and let them do what they want 1 day a week rather than what they get paid to do"), and the outright refusal to fix their own bugs. How many developers do you know who would turn their noses up at being placed on a sustained engineering team rather than creating new features? There's a stigma there. You think that stigma won't be there at 3AM Pacific when Europe starts hitting their new feature to the breaking point? I've been an ops engineer in a "lean startup" where developers were on-call for their services. It didn't work super well because people ignored their phones or put phones on silent at night. As a backstop, they put me (ops engineer) as the fallback secondary notification because they knew I would wake up. Ergo, everyone ignored everything and it all rolled to me. They'd wake up at 7AM, find everything ablaze (because they ignored all my phone calls), then would fix it and go about the rest of their day. Let's face it though. They probably don't want to be on call. Why force them to do this? There's the concept of ownership and closing the pain loop, but most people really don't understand what it truly means to be on call. "Sorry honey, can't go to the movies tonight I'm on call." "Sorry bro, can't get wasted tonight I'm on call." Only huge organizations with huge dev teams can go through a developer on-call rotation. Most leaner (smaller) companies have 1-2 devs per project, and it's unreasonable to expect that developer to be on-call 24x7. This stuff works at Uber, Facebook, and Google. But the vast majority of the world isn't Uber, Facebook, and Google. Also, I'd expect more pay as a developer if the job required on-call shifts. I don't think companies are willing to pay even more than they already are.